What's Actually Inside Kimi K3

Moonshot AI just released a technical report for Kimi K3, and it's dense. If you don't work on LLM internals every day, the paper reads like alphabet soup: KDA, AttnRes, LatentMoE, SiTU-GLU, QB. Here's what's actually going on, translated into normal language.

The Basic Numbers

Kimi K3 is a 2.8 trillion parameter model, but it only "activates" 104 billion of those parameters for any given token. This is the mixture of experts idea: instead of running every parameter for every word, the model routes each token to a small subset of specialized sub networks (experts), and only those get used. It's how you get a model with trillions of parameters that doesn't cost trillions of parameters worth of compute to run.

It also supports a context window of up to one million tokens, meaning it can hold roughly a very long book's worth of text in memory at once, and it has native vision, so it can read and reason over images and video in the same stream as text, no separate pipeline required.

The Attention Problem It's Solving

Normal transformer attention (the mechanism that lets a model relate any word to any other word) gets expensive as sequences get longer, because every new token has to be compared against every previous token. That's fine for a paragraph. It's brutal for a million tokens.

Kimi K3's answer is a hybrid: three layers of something called Kimi Delta Attention (KDA) for every one layer of traditional global attention. KDA is a linear attention variant, meaning it processes the sequence with a fixed size running memory state rather than an ever growing cache. Think of it like a running summary that updates as it reads, rather than keeping every sentence it's ever seen laid out side by side. This is much cheaper for long sequences, but it's less precise at pulling up an exact detail from far back.

That's why the model doesn't use only KDA. Every fourth layer switches to Gated MLA (Multi-head Latent Attention), which is the more expensive, more precise, full attention mechanism, compressed to keep memory costs down. The idea is that the cheap layers handle general flow and local patterns, while the occasional expensive layer gives the model a chance to reach back and grab something specific from anywhere in the context.

One clever detail here: the researchers found that the standard math for the "decay" in KDA's memory (how fast old information fades) could technically blow up to infinity during training, which causes numerical problems. They fixed this by bounding the decay so it can never go below a certain floor, which let them use faster, simpler math on the chip (dense matrix multiplication) instead of slower per position calculations.

Letting Layers Talk to Each Other

Normally in a deep neural network, each layer only sees the output of the layer right before it, information gets passed along like a game of telephone. Kimi K3 adds something called Attention Residuals, which lets every layer selectively pull information from any earlier layer, not just the previous one, using an attention mechanism applied across depth instead of across the sequence. To keep this affordable at nearly 100 layers deep, they group layers into blocks and only let blocks attend to each other rather than every single layer to every other layer.

The Mixture of Experts Architecture

The "896 routed experts, 16 active per token" language means the model has a huge library of specialized sub networks, and a router decides which 16 of them are relevant to each individual token, plus two "shared" experts that always run. This is what lets total parameters balloon into the trillions while keeping the actual computation manageable.

Cramming in that many experts creates two problems. First, the internal activations (the numbers flowing through the network) can spiral out of control at that scale, so they built a new activation function called SiTU-GLU that behaves like the standard modern activation near zero but gets capped smoothly at large values, so it can't explode.

Second, load balancing. If the router always sends tokens to the same few popular experts, the rest sit idle and undertrained, and the busy ones become a bottleneck. Kimi K3 introduces Quantile Balancing, a method that looks at a batch of tokens, figures out exactly what bias adjustment each expert needs so it ends up with a fair, even share of the workload, and applies that. Because computing this exactly across millions of tokens is too slow, they approximate it using histograms, essentially quick tallies that get combined across all the machines training the model at once.

Vision and Training

Instead of bolting a pretrained image model onto a finished language model (the common approach), Kimi K3 trains its vision encoder, called MoonViT-V2, from scratch alongside the language model, predicting the next token the whole way through. The researchers found this was more stable during joint training than starting from an image model pretrained a different way.

For actually running the finished model efficiently, there's a lot of engineering around caching. Because KDA carries state instead of a growing list of past tokens, and because MLA layers still need the traditional cache, the serving system has to manage two very different kinds of memory together, letting the model reuse work from earlier in a conversation instead of recomputing everything from scratch.

Where This Leaves Things

According to the report's own benchmarks, Kimi K3 sits just behind the very best proprietary models like Claude and GPT's newest releases, but ahead of other open weight models, at a noticeably lower cost per task. Whether that holds up outside the company's own test suite is something outside evaluators will need to confirm, but the architectural ideas here, especially the hybrid attention design and the load balancing method, are genuinely interesting engineering, not just bigger numbers for their own sake.