Loading Accelerated Logic…
GA Release

The Multi-Agent WebGPU Engine & Interactions API are now generally available.

Part 4: Browser-Local Models — WebLLM & Gemini Nano

Set up supported on-device model runtimes, understand their download requirements, and check what stays local during inference.

Category: Getting Started • Read Time: 27 min read • Updated: 2026-11-10

1. Why You Would Want Offline At All

Cloud models are fast and capable, but there are three critical scenarios where offline local execution wins:

🔒 Local inference after setup

With a browser-local model selected and loaded, inference runs in the browser instead of a hosted model provider. Model/runtime downloads and other app features can still use network services. Avoid entering highly sensitive data unless the full data path is appropriate for it.

✈️ Some work can continue offline

After the app, runtime, and model files are available in the browser, local generation may continue without a connection. Browser cache eviction, app reloads, provider calls, and other features may require network access.

No hosted-model token charge

Local inference does not use a hosted provider's token quota, but storage, device memory, speed, electricity, model license terms, and browser storage limits still apply. Browser caches can be cleared or evicted.

⚖️ The Trade-off:

Browser-local models use your device resources and may have smaller capacity than some hosted models. They can be useful for short tasks when local inference fits your hardware. If you need a larger model or more context, select a connected provider and review its data and pricing terms.

2. WebLLM — Your GPU Runs the Model

What it is: WebLLM uses WebGPU where the browser and hardware support it to run model computations on the device. The selected model files are downloaded, cached by the browser, and loaded into available memory. A browser may later clear cached files, so do not assume the download is permanent.

Hardware & Browser Requirements:
  • • Supported Browsers: Chrome 113+, Edge 113+, Brave with WebGPU enabled. Safari does not support WebGPU yet.
  • • Apple Silicon Macs: M1/M2/M3/M4 Macs run WebLLM exceptionally well due to high-bandwidth Unified Memory architecture.
  • • Windows PCs: Dedicated GPU with 4GB+ VRAM (NVIDIA GTX 1060 or better) and 8GB–16GB system RAM.

3. The WebLLM Model Catalog — What to Pick

In Model Organizer -> WebLLM provider, Accelerated Logic AI displays curated prebuilt models:

🚀 Llama-3.2-1B (~800MB Download)

Ultra-fast download, runs on low-end hardware. Great for quick text transformations, routing, and short responses.

⭐ Llama-3.2-3B Instruct (~2GB Download) [Recommended Starter]

Optimal balance between memory footprint, inference speed, and coding accuracy. Fits comfortably on 8GB RAM systems.

🧠 Llama-3.1-8B or Qwen2.5-Coder-7B (~4GB Download)

Highest quality offline code generation. Requires 12GB–16GB RAM and dedicated GPU VRAM.

👁️ Phi-3.5-Vision-Instruct

Supports image input offline. Upload UI screenshots and prompt "Clone this layout in React".

4. Download, Cache, and Load — Understanding Model States

Each WebLLM model card transitions through 4 distinct visual lifecycle states:

1. Download

Fetches model shards into browser storage. Real-time percentage progress displays in bottom bar.

2. Cached

Weights stored on local disk (Cache API). Click Load to initialize weights into GPU memory.

3. Loaded

Model is active in RAM/VRAM and ready for instant streaming generation.

4. Delete Cache

Removes local weight shards to immediately free up browser disk space.

5. Check the local inference data path

  1. Equip and Load a WebLLM model (e.g. Llama-3.2-3B).
  2. Send a test prompt in chat: Hello in 5 languages.
  3. For a controlled check, keep the same loaded page and model selected, then inspect the browser Network panel during a prompt.
  4. Confirm the prompt is handled by the local runtime and that no hosted model request is made for this generation. Other background requests may still occur.

6. Storage Management & Deleting Weights

WebLLM model shards are cached in browser Cache Storage. To inspect or manage disk usage:

Checking Storage:

Open Chrome DevTools -> Application tab -> Storage. It shows exact storage used (e.g., 3.8 GB).

Freeing Storage:

In Model Organizer -> WebLLM, click Delete Cache on any cached model card to delete its downloaded weights instantly.

7. Performance Tips for Local WebLLM

Memory Headroom

Close memory-heavy browser tabs (Figma, YouTube) when loading 7B/8B models to prevent WebGL context loss.

Context Management

Keep context length under 4096 tokens for local execution by unpinning unused attachments.

8. Gemini Nano — Chrome on-device model where available

Some Chrome builds expose a built-in on-device language model API. Availability, prerequisites, and names of experimental flags can change by Chrome version and device; the workspace shows this provider only when its capability check succeeds.

How to Enable Chrome Gemini Nano:
  1. Use a supported Chrome build on compatible hardware and check Chrome's current built-in AI documentation for the required setup.
  2. Open the Model Organizer. The Gemini Nano entry is available only when the browser reports support.
  3. If the provider is unavailable, use a supported WebLLM, Transformers.js, or local Ollama model instead.

9. WebLLM vs Gemini Nano vs Ollama Comparison

Feature Gemini Nano WebLLM (WebGPU) Ollama Local
Setup Needed Supported Chrome build and device 1-Click Browser Download Desktop Install + CORS
Download Size Managed by Chrome 1GB - 4GB in Cache 2GB - 10GB Local Disk
Context Window ~4096 tokens 4096 tokens 8k - 128k tokens
Vision Support No Yes (Phi-3.5 Vision) Yes (LLaVA / MiniCPM)

10. Offline Troubleshooting

Q:
"WebGPU not supported"

Ensure you are using Chrome 113+ or Edge 113+ on desktop. Check chrome://gpu to verify hardware acceleration.

Q:
Tab crash during model loading

Out of GPU memory. Switch from 8B model to a lighter 1B or 3B model (e.g. Llama-3.2-3B).

Q:
Gemini Nano provider card not showing Active

Verify both flags in chrome://flags are Enabled and check chrome://components status.

11. Transformers.js — A Second In-Browser Local Path

WebLLM is not the only browser-local engine. The app also ships Transformers.js models (ONNX runtime, WebGPU with WASM fallback). Current catalog entries center on compact Qwen models — vision-language and text variants — which are far smaller than 3–8B WebLLM weights and run on modest hardware.

When Transformers.js is the better pick
  • Weak GPU or no WebGPU — small ONNX models run on WASM CPU paths
  • Fast startup matters — sub-1B models download and load quickly
  • Vision on-device — the VL variant accepts images where WebLLM vision models are not cached
What you give up
  • Capability — tiny models are fine at extraction, classification and short Q&A, not deep reasoning
  • Context — 2048–32k windows depending on the entry, far below cloud 128k+
  • Consistency — switching models unloads the previous engine to free memory

Treat Transformers.js as the "always-available floor" for local inference and WebLLM as the "quality ceiling" when your GPU can load bigger weights.

12. Bringing Your Own GGUF — Importing Local Files into Ollama

Already own a .gguf file (fine-tune, quantized community release, private model)? The app can stream it straight into the Ollama instance on this computer and register it as a usable chat model — no manual ollama create step.

How the import works (verified against the shipped import path):

  1. The target is loopback-only — the importer requires an http(s):// Ollama URL on localhost / 127.0.0.1 / ::1 with no credentials, path, or query string. It will not upload weights to a remote or LAN host.
  2. Each file is validated and hashed (SHA-256) before transfer; the app checks whether each blob already exists in Ollama and skips re-sending duplicates.
  3. Every import gets its own unique tag under acceleratedlogic/<stem>:<random>, so importing the same file twice (or importing a file named like an existing model) never overwrites anything you already have.
  4. You pick a context size (whole number, at least 32 tokens); it is applied as Ollama's num_ctx parameter for the created model.
  5. Progress reports per-file percentages, and a failed import never leaves a half-registered model behind without a clear error.

After import, the model appears in the local model catalog like any pulled Ollama model and can be equipped in the Model Organizer. Vision and tool capabilities are detected from Ollama's own metadata for the created model.

13. Will a Model Fit? Doing the Memory Math Before You Click Download

Model cards show download size; this is how to translate that into "will my machine survive it":

Rough rule: resident memory ≈ weights × bytes-per-parameter + KV-cache overhead for your context. A 3B Q4 model (~2 GB weights) typically needs 3–4 GB free; a 7–8B Q4 model (~4–5 GB) wants 6–8 GB free plus browser baseline.

  • Quantization notation: Q4 variants trade a little quality for ~half the memory of Q8/F16. WebLLM builds are pre-quantized; that is what the download size you see already reflects.
  • Context is not free: 4k vs 128k context differs by gigabytes of KV cache on larger models. Local builds cap around 4k tokens deliberately — long-context local work belongs on Ollama with a big-RAM machine.
  • Shared GPU memory: WebGPU allocations compete with everything else on the GPU. A 4 GB download on a 4 GB card leaves no headroom for the browser itself — prefer a smaller model over an out-of-memory crash mid-load.
  • Check before you commit: close memory-heavy tabs, watch Activity Monitor / Task Manager during first load, and if the tab stutters during the progress bar, switch to the next smaller model rather than forcing it.

14. Long-Session Stability — Keeping a Local Model Alive for Hours

Symptoms you are near the edge
  • First tokens get slower as the session grows (KV cache pressure)
  • Occasional WebGPU "device lost" warnings or stutters
  • Browser tab reload kills the loaded engine — reload means re-Load from cache (not re-download, but re-initialize RAM/VRAM)
Habits that keep it stable
  • Unpin files you no longer need — pinned context is re-sent every turn
  • Start a fresh chat when the Context % badge climbs past ~90%
  • If the GPU resets, re-Load the model from cache rather than re-downloading it
  • Keep the tab group small while a local engine is resident

When a local model crashes the tab repeatedly even on short prompts, the hardware budget is simply too small for that model — drop one size class rather than debugging the prompt.