1. Why You Would Want Offline At All
Cloud models are fast and capable, but there are three critical scenarios where offline local execution wins:
Working on confidential client code, medical notes, legal contracts, or startup pitch decks? WebLLM and Gemini Nano never transmit prompts over the network after model download.
On an airplane, commuter train, or spotty hotel WiFi? WebLLM and Gemini Nano run smoothly with WiFi turned completely off.
No monthly subscriptions, no 429 rate limit errors, no API quota warnings, and no sign-in popups. Download weights once (1–4 GB) and chat indefinitely.
Offline models are smaller in parameter scale (1B to 8B) compared to cloud frontier models (hundreds of billions). They excel at code editing, classification, summarization, and concise function generation. For deep reasoning across massive repositories, combine offline models with cloud fallback chains (covered in Part 3).
2. WebLLM — Your GPU Runs the Model
What it is: WebLLM leverages WebGPU — a modern browser API that allows web applications to run computations directly on your graphics hardware. Model weights are fetched as 4-bit compressed shards (q4f16), cached locally in browser storage, and executed directly in VRAM/RAM.
- • Supported Browsers: Chrome 113+, Edge 113+, Brave with WebGPU enabled. Safari does not support WebGPU yet.
- • Apple Silicon Macs: M1/M2/M3/M4 Macs run WebLLM exceptionally well due to high-bandwidth Unified Memory architecture.
- • Windows PCs: Dedicated GPU with 4GB+ VRAM (NVIDIA GTX 1060 or better) and 8GB–16GB system RAM.
3. The WebLLM Model Catalog — What to Pick
In Model Organizer -> WebLLM provider, Accelerated Logic AI displays curated prebuilt models:
Ultra-fast download, runs on low-end hardware. Great for quick text transformations, routing, and short responses.
Optimal balance between memory footprint, inference speed, and coding accuracy. Fits comfortably on 8GB RAM systems.
Highest quality offline code generation. Requires 12GB–16GB RAM and dedicated GPU VRAM.
Supports image input offline. Upload UI screenshots and prompt "Clone this layout in React".
4. Download, Cache, and Load — Understanding Model States
Each WebLLM model card transitions through 4 distinct visual lifecycle states:
Fetches model shards into browser storage. Real-time percentage progress displays in bottom bar.
Weights stored on local disk (Cache API). Click Load to initialize weights into GPU memory.
Model is active in RAM/VRAM and ready for instant streaming generation.
Removes local weight shards to immediately free up browser disk space.
5. How to Verify It Is Truly Offline
- Equip and Load a WebLLM model (e.g. Llama-3.2-3B).
- Send a test prompt in chat:
Hello in 5 languages. - Turn OFF WiFi / disconnect ethernet.
- Send another prompt:
Write a python function to reverse a string. - The model continues streaming answers smoothly. Open browser DevTools Network tab — zero HTTP requests are transmitted during response generation.
6. Storage Management & Deleting Weights
WebLLM model shards are cached in browser Cache Storage. To inspect or manage disk usage:
Checking Storage:
Open Chrome DevTools -> Application tab -> Storage. It shows exact storage used (e.g., 3.8 GB).
Freeing Storage:
In Model Organizer -> WebLLM, click Delete Cache on any cached model card to delete its downloaded weights instantly.
7. Performance Tips for Local WebLLM
Close memory-heavy browser tabs (Figma, YouTube) when loading 7B/8B models to prevent WebGL context loss.
Keep context length under 4096 tokens for local execution by unpinning unused attachments.
8. Gemini Nano — Built-in Chrome On-Device Model (Zero Download)
Google Chrome includes built-in on-device AI capabilities via the experimental Prompt API.
- Open Chrome flags by visiting
chrome://flags. - Set
#prompt-api-for-gemini-nanoto Enabled. - Set
#optimization-guide-on-device-modelto Enabled BypassPerfRequirement. - Relaunch Chrome. Go to
chrome://components-> find Optimization Guide On Device Model -> click Check for update. - Open Accelerated Logic AI -> Model Organizer -> select Gemini Nano (Built-in). The green Active badge will light up.
9. WebLLM vs Gemini Nano vs Ollama Comparison
| Feature | Gemini Nano | WebLLM (WebGPU) | Ollama Local |
|---|---|---|---|
| Setup Needed | Chrome Flags | 1-Click Browser Download | Desktop Install + CORS |
| Download Size | Managed by Chrome | 1GB - 4GB in Cache | 2GB - 10GB Local Disk |
| Context Window | ~4096 tokens | 4096 tokens | 8k - 128k tokens |
| Vision Support | No | Yes (Phi-3.5 Vision) | Yes (LLaVA / MiniCPM) |
10. Offline Troubleshooting
Ensure you are using Chrome 113+ or Edge 113+ on desktop. Check chrome://gpu to verify hardware acceleration.
Out of GPU memory. Switch from 8B model to a lighter 1B or 3B model (e.g. Llama-3.2-3B).
Verify both flags in chrome://flags are Enabled and check chrome://components status.