Skip to content

Kimi Linear

How channel wise forgetting, corrective writes, and occasional global attention make long context memory more efficient.

Updated View as Markdown

Kimi Linear replaces most token history lookups with a small memory that updates as tokens arrive. It retains occasional global attention for information that compressed memory may lose. The central question is not whether attention disappears, but how much exact history a language model needs at each layer.1

Highlights

  • Forget at channel resolution. Kimi Delta Attention (KDA) gives different memory channels different retention rates, rather than one decay value for an entire head.1
  • Correct existing associations. The delta rule writes the difference between a new value and what memory currently predicts for its key.1
  • Preserve a global path. A roughly three to one mixture of KDA and MLA balances efficient recurrent memory against direct access to earlier tokens.1

The release includes Base and Instruct weights, custom model code, and public KDA kernel implementations. Its controlled comparisons make it valuable for studying attention alternatives, but the results do not prove that recurrent memory dominates full attention for every task or scale.12

Study scope: October 2025 report, recurrence derivation and final checkpoint appendix, pinned Instruct configuration, cache and attention code, and license. No checkpoint or kernel executed. Evidence checked October 10, 2026.

Model in numbers

Representative checkpoint: Kimi-Linear-48B-A3B-Instruct. Distinguish this release from the report’s matched 1.4T token experimental models.

Dimension Value and scope
Parameters About 48B total and 3B active per token. Stored parameters remain much larger than the active computation.1
Pretraining 5.7T processed tokens for the released model; 1.4T for the controlled MLA and GDN hybrid comparisons. Neither is a unique data volume.1
Compute Total accelerator hours and full run FLOPs are not disclosed. Scaling experiments report relative efficiency, not a complete training cost.1
Context Pretraining begins at 4,096 tokens. The final release supports 1,048,576 positions and Appendix D evaluates RULER at 1M.13
Layer composition Released config: 27 layers, 20 KDA and 7 full MLA, with global attention at the final layer.3

Architecture

Pan on smaller screensOpen full size ↗

Figure 1. The hybrid decoder and KDA module. Reproduced from the release documentation’s Figure 3, under MIT terms.4

Figure 1 is a layerwise hybrid, not two parallel networks. Three normalized KDA and expert block pairs alternate with one normalized MLA and expert block pair. The lower right inset shows local convolutions on queries and keys, distinct retention and write gates, and a gated normalized readout. Each global MLA layer can still access earlier token history directly.41

Memory that forgets and corrects

A single KDA head carries a matrix St∈Rdk×dvS_t\in\mathbb{R}^{d_k\times d_v}. Think of it as a learned associative memory: a query reads a value, while a key identifies the association to update. KDA first decays the old state, then corrects the predicted value at the current key.1

S~t−1=Diag⁡(αt)St−1,St=S~t−1+βtkt(vt−S~t−1⊤kt)⊤,ot=St⊤qt(1)\widetilde S_{t-1}=\operatorname{Diag}(\alpha_t)S_{t-1},\qquad S_t=\widetilde S_{t-1}+\beta_t k_t\left(v_t-\widetilde S_{t-1}^{\top}k_t\right)^{\top},\qquad o_t=S_t^{\top}q_t \tag{1}

Equation (1) is an algebraic rearrangement of the report’s recurrence. The vector αt\alpha_t controls retention for each key channel; scalar βt\beta_t controls write strength. The term inside parentheses is the prediction error. A plain additive memory would keep stacking key value associations; this corrective write can revise an association already represented in memory.1

Two complementary ways to retain history
KDA layer
  1. Prior matrix state128 × 128 per head
  2. Channel decayChoose which channels persist
  3. Corrective writeUpdate the current key's value
  4. Query readRead the updated state

The updated state becomes the prior state for the next token. Its shape does not grow with history.

Periodic MLA layer
  1. Previous token historyRetained keys and values
  2. Global attentionDirect content lookup
  3. Output representationMix retrieved values

This cache still grows with sequence length. The hybrid is not an entirely fixed memory model.

Figure 2. Recurrent memory and direct history lookup. Original explanatory diagram informed by the recurrence and hybrid design.13

Queries and keys pass through short convolutions, Swish, and L2 normalization. Normalized keys help keep the corrective update stable. Low rank projections produce the channel decay and output gate; a headwise RMSNorm and sigmoid output gate control the readout. This memory is input dependent, not a fixed average over the past.1

From sequential recurrence to efficient kernels

The recurrence is sequential across tokens, but training need not execute one tiny operation at a time. KDA groups tokens into chunks: a recurrent state carries earlier chunk information, while a causal triangular computation handles interactions within a chunk in parallel. A specialized triangular transform incorporates the corrective writes.1

The transition matrix has a diagonal plus low rank form, but its two low rank factors are tied to the write key and decay: one is proportional to the key, the other to the key multiplied by retention. In the chunk formulation, a WY representation packs the corrective transforms; a lower triangular UT transform produces pseudo values that incorporate earlier writes. This is not ordinary softmax attention on the chunk.5

Pan on smaller screensOpen full size ↗

Figure 3. Parallel inside a causal chunk, recurrent across chunks. Independent mechanism diagram of Equations (8)–(9).5

Figure 3 has two contributions: the decayed incoming state carries earlier chunks; a causal triangular computation handles the current chunk. Updating the boundary state is still necessary before the next chunk. Log space relative decay and secondary tiles protect small cumulative retention ratios. Tying the low rank factors removes two of the general DPLR secondary matrices and approximately three matrix multiplications in the reported comparison.5

This is why “linear attention” describes scaling with sequence length, not absence of a nonlinear model or zero matrix work. With fixed head dimensions, the recurrent state is bounded; projections, experts, convolution states, and global attention layers still cost computation and memory.1

Complexity and the memory learning interpretation

For one head with width dhd_h, fixed chunk size C=64C=64, and sequence length TT, the report’s kernel estimate is:6

FLOPs⁡KDA=6Tdh2+3TCdh+TC2,FLOPs⁡full≈2T2dh(2)\operatorname{FLOPs}_{\mathrm{KDA}}=6Td_h^2+3TCd_h+TC^2,\qquad \operatorname{FLOPs}_{\mathrm{full}}\approx2T^2d_h \tag{2}

This count excludes projections, experts, output gating, local convolutions and the hybrid’s global layers. Fixed C and head width make the KDA kernel linear in T; they do not make the whole hybrid decoder entirely linear in T.6

The corrective write also has a local learning view. On the already decayed state, descend the squared prediction error βt2∥vt−S~t−1⊤kt∥2\tfrac{\beta_t}{2}\lVert v_t-\widetilde S_{t-1}^{\top}k_t\rVert^2 by one unit step to recover Equation (1)’s rank one correction. The state is an input dependent fast memory, not a new optimizer update to the model’s learned parameters during every decode token.6

The released decoder and its actual cache

The residual width is 2,304. KDA has 32 heads with key and value dimensions of 128. The first FFN is dense; the artifact selects eight routed experts from 256 and adds one shared expert, each with intermediate width 1,024. The report’s phrase “including one shared expert” is ambiguous about that count; configuration and forward code explicitly execute the shared branch in addition to the eight selected routes.371

Global MLA layers use NoPE: there is no explicit rotary position transformation. KDA’s ordered recurrence, decay, and local convolutions provide position sensitive computation. Removing RoPE from the global path does not make the whole model insensitive to token order.17

The custom cache stores recurrent matrices and convolution state for KDA, alongside token growing keys and values for global layers. The inspected Transformers implementation expands MLA content into head specific keys and values before cache update. Therefore its memory use must not be equated with an ideal latent only MLA cache without inspecting the serving backend.7

The code dispatches short inputs to a fused recurrent KDA kernel and longer inputs to the chunk kernel. Training is restricted to the chunk path. These are implementation observations, not measured performance on this machine.7

Training and post training

The matched experiments use identical 1.4T token data from the K2 corpus, 4K contexts, MuonClip, a WSD schedule, peak learning rate 0.0011, and 32M token global batches. The final model expands token exposure to 5.7T and extends context to 1M. Full data mixture proportions and total run cost are not supplied.1

SFT combines broad instruction learning with targeted reasoning data. RL concentrates on math, code, and STEM, selecting tasks of moderate starting difficulty. An auxiliary pretraining loss helps preserve general behavior. The report follows the K1.5 policy algorithm, with truncated importance sampling for training versus rollout precision mismatch, adaptive KL penalty, and adjusted numbers of updates per iteration.1

What the comparisons establish

The 1.4T comparison controls architecture scale and training recipe, varying the token mixer. Kimi Linear improves MMLU-Pro from MLA’s 47.2 to 51.0; EvalPlus is 60.2 versus GDN hybrid’s 63.1. It performs better across many tested tasks, not every column. Temperature is 1.0, with repeated averages for high variance evaluations.1

Small model ablations use the same FLOP budget and hyperparameters. The 7:1 ratio has similar training but worse validation loss than 3:1; the 1:1 ratio has similar validation loss but higher inference overhead. This is a capacity/generalization/runtime tradeoff, not a universal rule that 25% global layers is optimal. Removing the sigmoid gate or local convolution also harms that experiment. The claim that NoPE provides a better distributed positional bias is an author interpretation of comparisons, not a separately proven explanation.1

The final 5.7T Instruct model reports RULER 94.8 at 1M. Its comparison with 16B Moonlight changes total capacity as well as attention, so it is less clean as an isolated attention ablation than the matched 1.4T study.1

Read the speed and cache headline with its conditions

Figure 7 measures batch size one. At 1M context it shows roughly 2.9 times faster prefill and about 2.3 times faster decoding than the full MLA comparison. The roughly six times headline also uses increased batching made possible by memory savings; Section 6 describes the 6.3 times case as a theoretical decoding speedup. It is not a six times latency promise for every request.1

“Up to 75%” cache reduction describes replacing most full history layers. The released 27 layer pattern has a final extra global layer, and recurrent and convolution state remain. Actual bytes depend on precision, cache layout, batch size, and backend. The paper does not fully specify a reproducible hardware and kernel environment for every plotted efficiency comparison.137

Limits and reusable lessons

Finite memory may lose exact details or competing associations, which motivates retaining global lookup. A 1M context score does not guarantee reliable recall of arbitrary information at that length. Efficient recurrent kernels also complicate prefix reuse, branching, and speculative decoding because state must correspond to the correct token boundary.17

The repository uses MIT and the official model card labels the weights MIT. Public weights and kernel code do not make the full dataset or training stack available.89

For builders, the useful pattern is compress where possible, retain exact access where necessary, and validate the boundary. Continue to Kimi K3 to see how this memory design interacts with depth mixing, expert width, native vision and deployment precision.

References

  1. Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture, October 30, 2025. Sections 3 to 7, Tables 1 to 5, Figures 1 and 7, and Appendices B to D. Original version inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22 ↩23 ↩24 ↩25 ↩26

  2. Moonshot AI. Kimi Linear release, inspected October 10, 2026. ↩

  3. Moonshot AI. Kimi-Linear-48B-A3B-Instruct configuration, inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5

  4. Kimi Team. Kimi Linear release documentation, Figure 3, page 6. Repository MIT grant explicitly includes associated documentation; the report and matching architecture asset reside there. Artwork unchanged; surrounding prose and caption removed. The arXiv edition’s CC BY-NC-ND grant is not the reproduction basis. ↩ ↩2

  5. Kimi Team. Kimi Linear report, Sections 3.1–3.2 and 6.2, Equations (3)–(9), Listing 8 and Appendix B: WY/UT chunk transforms and constrained DPLR simplification. The native figure omits normalization and local projection operators, shown in Figure 1. ↩ ↩2 ↩3

  6. Kimi Team. Kimi Linear report, Section 6.3, Equations (13)–(14), and Table 7: kernel FLOP accounting and decayed-state SGD interpretation. These are not measured complete model costs. ↩ ↩2 ↩3

  7. Moonshot AI. Custom model and cache implementation, classes KimiDynamicCache, KimiMLAAttention, KimiDeltaAttention, and KimiMoE, inspected October 10, 2026. Static code inspection only. ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  8. Moonshot AI. MIT license, inspected October 10, 2026. ↩

  9. Moonshot AI. Official model card, release inventory and license metadata, inspected October 10, 2026. ↩

Navigation

Type to search…

↑↓ navigate↵ selectEsc close