Kimi K2 asks how to train a much larger pool of experts without making every token equally expensive. Its answer combines a familiar sparse decoder with an unusual optimizer safeguard, more varied presentations of training material, and post training that teaches the model to act through tools.1
Highlights
- Stabilize the optimizer, not just the forward pass. MuonClip constrains growing attention logits by rescaling selected query and key weights after an optimizer step.1
- Expand capacity through sparse experts. The model stores about a trillion parameters but activates roughly 33B for a token, with MLA controlling attention cache size.1
- Train decisions in environments. Synthetic tool trajectories, executable verifiers, and a learned self critic provide different forms of feedback. They are complementary, not equally reliable ground truth.1
The public weights and technical report make these choices inspectable. This is an open weight release with a modified MIT license, not a complete open training dataset or unrestricted standard MIT grant.23
Study scope: original July 2025 report, relevant optimizer and tool formatting appendices, original K2 Instruct artifact and licenses. This page does not cover K2 Thinking or K2-Instruct-0905. No checkpoint executed. Evidence checked October 10, 2026.
Model in numbers
Representative checkpoint: Kimi-K2-Instruct, the original nonthinking instruction model.
| Dimension | Value and scope |
|---|---|
| Parameters | Table 2 reports 1.04T total and 32.6B active. The abstract rounds these to 1T and 32B.1 |
| Pretraining | Headline 15.5T processed tokens. The recipe separately describes 400B annealing tokens and an additional 60B long context tokens; do not treat the headline as a unique corpus size.1 |
| Tool synthesis | More than 3,000 real MCP tools plus about 20,000 synthetic tools are described. These counts are tool interfaces, not training trajectories or verified tasks.1 |
| Compute | Full training FLOPs and accelerator hours are not disclosed in the report. A model parallel group of 256 GPUs describes placement, not the total run budget.1 |
| Context | Main pretraining at 4,096 tokens; long context training at 32K with YaRN extension to 128K. Released maximum position setting is 131,072.14 |
Architecture: more experts, similar active work
The decoder has 61 layers and a residual width of 7,168. Its first FFN is dense. Other layers combine one shared expert with eight selected experts from 384 routed experts, each with intermediate width 2,048. MLA uses 64 attention heads, a 512 dimensional KV latent, and a decoupled 64 dimensional RoPE key. Compared with DeepSeek-V3, the design expands the expert pool while reducing the attention head count.14
The important distinction is stored capacity versus per token computation. A larger pool permits more specialist weights without executing every expert, but all those weights still need storage and suitable placement. Dispatch, communication, and uneven expert loads remain real costs. Unlike V3, K2 does not group its expert selection into node limited routing. Compressed MLA cache also grows with context; it is not recurrent fixed memory.1
Why increase sparsity but reduce attention heads?
The small scale sparsity experiments hold eight active experts and one shared expert fixed while varying total experts. At validation loss 1.5, the fitted optimum for sparsity 48 needs 1.15 times less FLOPs than sparsity 32. This is a scaling fit, not a measured full K2 versus V3 training cost.5
A separate head count study finds only 0.5–1.2% lower validation loss from doubling heads under its iso token comparisons. The report estimates that increasing 64 to 128 heads at 128K context, keeping 384 experts, adds 83% inference FLOPs. K2 therefore spends more capacity on stored experts and less arithmetic on attention heads; communication and weight storage still constrain the choice.5
MuonClip: control a failure mode after the update
Muon orthogonalizes momentum based matrix updates. At larger scale, the authors observed exploding pre softmax query key scores with ordinary Muon. RMS normalization of the input does not directly bound the spectral norms of the learned query and key projections. MuonClip therefore adds QK-Clip after the optimizer’s weight update.1
- Forward and backwardRecord each head's largest attention score
- Muon updateMomentum, orthogonalization, scale and weight decay
- QK-ClipRescale only heads above threshold
- Next forward passApply the rescaled projections
The clipping changes weights after the update. It is not a soft cap inserted into the current attention softmax.
Figure 1. MuonClip’s post update safeguard. Original explanatory diagram informed by the report’s Algorithm 1 and Appendix D.1
For a head with recorded maximum score , define:1
The content query and key projection factors are each multiplied by . Their dot product therefore scales by . The head specific rotary query factor receives ; the shared rotary key projection is left unchanged. This respects MLA’s shared positional path rather than blindly multiplying every Q and K weight by the same factor.1
The safeguard is selective and can become inactive. It controls weight growth using the observed batch; it does not promise that a different next batch can never exceed the threshold. Appendix D reports that 12.7% of heads triggered clipping at least once during the initial 70,000 steps, then all heads eventually fell below the threshold. A separate 3B total, 0.5B active experiment with a more aggressive threshold of 30 found negligible loss differences. That bounded experiment supports the safeguard’s feasibility; it does not isolate every contributor to the final trillion parameter model.1
Data and training recipe
K2 rephrases selected knowledge and mathematics material to expose the same information through different styles and perspectives. Generation proceeds over chunks, with checks for fidelity to the source. The goal is to reduce repetitive presentations, not license arbitrary hallucinated facts as pretraining data.1
Figure 2. Rephrase chunks without losing the wider context. Reproduced intact from report Figure 4, under CC BY-NC-ND 4.0.6
Figure 2 retains the full 4,096 token excerpt as context while rewriting 256 token chunks sequentially. Earlier rewritten chunks condition later ones. Fidelity checking remains a separate quality control step; the diagram is not a guarantee that generation preserves every fact.6
One early knowledge experiment contrasts repeating raw documents, repeating one rewrite, and using multiple distinct rewrites. SimpleQA rises from 23.76 to 27.39 to 28.94 in that setup. The production knowledge corpora are rephrased at most twice, rather than applying the ten rewrite experimental condition to every corpus. Those are recipe comparisons on an internal experimental model; they do not establish a universal advantage for synthetic data or attribute the full K2 gain to rephrasing.1
The main recipe uses 4K sequences, a 67M token global batch, weight decay 0.1, and MuonClip. After 500 warmup steps, learning rate stays at 0.0002 for the first 10T tokens, then decays across 5.5T tokens. Annealing and long context activation lower it further to 0.000007.1
Precision and parallelism are distinct from the optimizer
The reported training system combines 16 way pipeline parallelism, 16 way expert parallelism, and ZeRO-1 data parallelism. It overlaps communication with an interleaved pipeline schedule and offloads activations to CPU memory. Selective recomputation protects scarce device memory.1
FP8 is used to store selected activations, not for the report’s training matrix computation. The authors explicitly avoid FP8 computation after observing quality risks in preliminary studies. The released Instruct artifact’s quantization metadata is an inference storage choice, not evidence that the pretraining run used the same arithmetic.14
Post training: grounded practice and a critic
Figure 3. Tool interfaces become filtered trajectories. Reproduced intact from the report’s Figure 8, under CC BY-NC-ND 4.0.1
Figure 3 shows two different creation steps. On the left, real MCP tools and generated interfaces define the available action space, which supports task and rubric generation. On the right, a user agent, task agent, and tool simulator produce an interaction history; a separate rubric judge filters it. A simulated observation is not an independently executed real world outcome.1
Tool synthesis first builds interfaces and state transitions, then agent tasks with rubrics, then multi step interactions between user, agent and tool simulator. A judge filters trajectories. Real executable environments provide additional coding and software engineering experience. Simulated tool state is useful coverage, but its fidelity must be tested before transferring a policy to real systems.1
RL mixes tasks with verifiable answers and less directly verifiable tasks. Code can use execution checks; instruction tasks can combine programmatic and judge checks. A learned self critic produces pairwise rubric based judgments for broader behavior and is refined using signals from verifiable tasks. Its judgments remain model generated preferences, not independent truth.1
K2 uses the policy optimization method from Kimi K1.5, rather than simply labeling all group sampled reasoning training “GRPO.” A simplified form of the report’s objective is:1
Here is the group reward baseline and controls regularization, distinct from the clipping threshold in Equation (1). The report also describes task dependent token budgets, truncation penalties, auxiliary pretraining loss to resist forgetting, and temperature scheduling. These choices affect exploration and response cost.1
RL systems: synchronized learning, partial trajectories
K2’s training and inference engines are colocated and synchronized, unlike GLM-5’s continuously decoupled actors. A controller alternates rollout and learning on the same workers. Training parameters are offloaded during generation; a checkpoint engine streams and reshards updated parameters for the inference replicas. The authors report a full refresh in less than 30 seconds in their system, not a universal bandwidth requirement or latency promise.7
Agent tasks can wait on environment services for a long time. Independent environment services and many concurrent rollouts reduce idle GPU time. Partial rollout pauses unfinished long trajectories and resumes them in a later RL iteration rather than waiting for every straggler. That scheduling optimization does not mean the entire learner and actor system is fully asynchronous.7
Inference, evidence and limits
Tool calls are serialized model outputs. An application still needs to parse them, authorize operations, execute tools, and return results using the release’s conversation format. A strong tool benchmark is not permission to expose unrestricted shell or network access.12
The main evaluation is nonthinking, generally with an 8K output limit; the SWE-bench Agentless setup uses 16K. Repeated run averages and best of multiple attempts must not be mixed. For example, the report’s SWE-bench agent result rises from 65.8 in a single attempt to 71.6 with an internal verifier selecting among attempts. That gain includes extra test time search and selection.1
The modified MIT license requires prominent Kimi K2 attribution for qualifying commercial products above specified monthly user or revenue thresholds. Read the exact grant before deployment. The release does not disclose all data, complete training code, or total project cost.32
The reusable lesson is to connect capacity, training stability, and feedback quality. More experts require a robust optimizer and communication system; more synthetic experience requires fidelity controls; more agentic behavior requires grounded execution and careful evaluation. Continue to Kimi Linear for a different strategy: changing the memory carried across long sequences.
References
-
Kimi Team. Kimi K2 Technical Report, July 28, 2025. Sections 2 to 4, Algorithm 1, Table 2, and Appendices A, D and E. Original version inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22 ↩23 ↩24 ↩25 ↩26 ↩27 ↩28 ↩29
-
Moonshot AI. Kimi K2 release and tool usage, pinned repository, inspected October 10, 2026. Current documentation also includes later variants; this study uses the original model and report. ↩ ↩2 ↩3
-
Moonshot AI. Modified MIT license, inspected October 10, 2026. ↩ ↩2
-
Moonshot AI. Original Kimi-K2-Instruct configuration, inspected October 10, 2026. ↩ ↩2 ↩3
-
Kimi Team. Kimi K2 report, Section 2.3, Table 2, Figures 5 and 6: sparsity fits, head count comparisons and 128K inference FLOP estimate. ↩ ↩2
-
Kimi Team. Kimi K2 report, Section 2.2, Figure 4, PDF page 6, and Table 1. Figure reproduced without adaptations under exact v1 CC BY-NC-ND 4.0; notice. Reuse is noncommercial; future monetization requires review. ↩ ↩2
-
Kimi Team. Kimi K2 report, Sections 3.3.1–3.3.4: synchronized colocated engines, checkpoint refresh and partial agent rollouts. ↩ ↩2

