PrismML’s Bonsai 27B Shows Why Local AI Is Getting Serious

If you’ve ever wanted to run a powerful AI model locally without buying top-tier hardware, PrismML’s new Bonsai 27B is a glimpse of where AI may be heading next. PrismML, a company with research roots at Caltech, released Bonsai 27B on July 14, 2026 and describes it as its largest and most capable Bonsai model so far. It is based on Qwen3.6-27B, and PrismML says it is the first 27B-class model that can run on a phone.

That claim stands out because the original model is large by local-AI standards. Qwen3.6-27B is a multimodal model with a 262,144-token default context length, and PrismML says a 16-bit version of a 27B model takes up roughly 54 GB of memory. Even a conventional 4-bit build is still around 18 GB, which is too large for phones and still inconvenient for many laptops.

Model weights are stored using a certain number of bits per weight, and quantization reduces that precision so the model needs less memory and less bandwidth during inference. Instead of using 16-bit floating point values for every weight, low-bit models restrict weights to a much smaller set of possible values. PrismML’s ternary version uses weights from {-1, 0, +1}, while the 1-bit version uses {-1, +1}, both with group-wise scaling to preserve quality.

There is a subtlety here that matters more than it looks: how a model gets low-bit. The conventional route is post-training quantization — take an existing FP16 checkpoint and compress it afterwards — which is what the 4-bit GGUF builds you have seen on Hugging Face do. PrismML went the other way: the Bonsai models are trained natively at low precision rather than squeezed after the fact, so there is no full-precision fallback hiding anywhere in the weights. That is the same bet Microsoft Research made with BitNet b1.58, the research line Bonsai descends from.

The lineage is worth knowing. The original “Era of 1-bit LLMs” paper (Ma et al., 2024) introduced BitNet b1.58, where every weight is one of three values — {-1, 0, +1} — hence “1.58 bits,” because log₂(3) ≈ 1.58. In April 2025, Microsoft shipped BitNet b1.58 2B4T, a 2-billion-parameter proof of concept that matched full-precision open-weight models of similar size, beat conventionally quantized INT4 builds of Qwen2.5-1.5B, and shipped with open weights plus custom CPU and GPU inference kernels. Bonsai 27B is the first credible demonstration that this idea scales out of the lab and onto a 27B multimodal model you can carry in your pocket.

Bonsai 27B arrived on July 14, 2026 under an Apache 2.0 license, with weights on Hugging Face in formats for basically every local runtime: native MLX builds for Apple Silicon and GGUF for llama.cpp everywhere else. The roughly 27.32B language weights get the extreme quantization treatment, while a 461M-parameter vision tower rides along in 4-bit so the model keeps reading screenshots and documents. This is a genuinely multimodal model, not a text-only compression demo.

In the official release, the 1-bit Bonsai 27B variant has an effective 1.125 bits per weight and a 3.9 GB footprint, while Ternary Bonsai 27B has an effective 1.71 bits per weight and a 5.9 GB footprint. Both versions are multimodal, running end-to-end on device.

The memory arithmetic checks out when you do it yourself: 27.32 billion weights at 1.125 effective bits is about 3.8 GB, at 1.71 bits it is about 5.8 GB, and at 16-bit floats it is 54+ GB — a compression factor of 9.4–14.2× over the original. That is what turns “27B on a phone” from a slogan into a file that fits inside the roughly 6 GB per-app memory budget of a 12 GB iPhone, and it is why PrismML can claim support all the way down to an iPhone 15.

Weights are only half the memory story. Every token you feed a transformer also writes entries into the KV cache, and that cache grows with your context — quietly, until it dwarfs the model. Qwen3.6-27B’s hybrid architecture helps here: about 75% of its layers use linear attention with a fixed-size recurrent state, so only 16 of 64 layers hold a cache that grows with context at all.

Even so, at FP16 the cache costs roughly 64 KiB per token. At the full 262,144-token window that adds up to about 17.2 GB — more than the model’s own weights. That is why the release also ships a 4-bit KV cache quantizer that shrinks the cache roughly fourfold, and why PrismML’s measurements matter as much as the weight compression: using forward-KL divergence against the FP16 cache of the same model, they found Bonsai tolerates cache quantization almost losslessly.

According to PrismML’s published benchmark results, Ternary Bonsai 27B retains about 94.6% of the full-precision baseline across a 15-benchmark suite, while 1-bit Bonsai 27B retains about 89.5%. The memory savings allow 11 tokens per second on an iPhone 17 Pro Max, unlocking true offline, private AI on consumer mobile devices.

Independent reporting adds useful texture around those numbers. The FP16 Qwen3.6-27B baseline scores 85.07 across that suite, so “94.6% retained” means roughly 80.5 in absolute terms — a real but survivable tax for most everyday tasks. On desktop-class Apple Silicon, the same builds reach up to 87 tokens per second on an M5 Max using DSpark, a lossless speculative-decoding drafter that does part of the work. And the launch hit the front page of Hacker News with over 500 points, where the top comments did the healthy thing and poked holes in the benchmark chart — PrismML’s figures are self-reported and have not yet been independently verified by third parties.

According to CNBC and the Wall Street Journal, Apple is actively evaluating PrismML’s compression technology as a path to cutting the cloud AI spending that currently backs Siri. Multiple reports indicate Apple’s own internal attempts to compress models for on-device use ran into significant performance degradation — which is exactly the failure mode native low-bit training is designed to avoid.

The contrast in engineering bets is the interesting part. Apple Intelligence leans on sparse activation, routing requests to a subset of parameters and paying a flash-to-DRAM bandwidth cost on the way. Bonsai keeps every one of the 27 billion parameters resident in DRAM at under 4 GB, but reduces each weight to a handful of discrete values. Opposite answers to the same question: how do you fit intelligence into a phone?

PrismML itself is a Caltech spinout led by Caltech computer scientist and mathematician Babak Hassibi, and it has raised $16.25 million from Khosla Ventures, Cerberus Capital, and Caltech itself. That is the kind of investor list that suggests a platform bet rather than a demo repo.

The headline result is bigger than one model: the qualitative breakdown below 4 bits, long assumed to be a property of the models themselves, turned out to be a property of the quantization methods. If a 27B multimodal model can run offline at reading speed on hardware people already own, the baseline definition of “local AI” just moved. For the local-first crowd — which is the crowd we build AcceleratedLogic for — Bonsai is the strongest evidence yet that private, offline, capable AI is a 2026 reality rather than a roadmap item.