Skip to content

GLM-5

How content selected attention and asynchronous reinforcement learning support long agent workflows.

Updated View as Markdown

GLM-5 is designed for work that unfolds over many actions: inspect a codebase, try a change, run a tool, observe failure, and revise. It combines a large expert model with attention that selects relevant history and a training system that does not wait for the slowest agent rollout before doing useful work.1

Highlights

  • Select where attention spends computation. DeepSeek Sparse Attention, or DSA, adds a content indexer that selects a subset of past positions for the main attention operation.12
  • Decouple rollout generation from learning. Agent trajectories can be collected while the learner updates, with token identity, policy lag and probability mismatch handled explicitly.1
  • Keep the model and agent system separate. Preserved thinking, tools, context management and environments affect long horizon results; weights alone are not the complete agent.1

This makes GLM-5 worth studying as a bridge between efficient architectures and agent training systems. It is not evidence that larger weights alone cause reliable software engineering, or that every published agent result follows from a default chat call.1

Study scope: original February 2026 GLM-5 report and appendix, BF16 GLM-5 artifact, tokenizer metadata, official repository and license. Later GLM-5.1 to 5.3 variants are excluded. No checkpoint executed. Evidence checked October 10, 2026.

Model in numbers

Representative checkpoint: zai-org/GLM-5, not GLM-5-FP8 or a later generation.

Dimension Value and scope
Parameters 744B total and 40B active in the report’s convention: includes MTP parameters but excludes embeddings and output layer.1
Base training 28.5T processed tokens across pretraining and mid training; not 28.5T unique source tokens.1
Agent data More than 10K real SWE environments in the disclosed agent training expansion; a full count of all SFT examples and RL trajectories is undisclosed.1
Compute Full GLM-5 accelerator hours, FLOPs and project cost undisclosed. Reported attention reductions are stage and workload specific, not a complete training cost ledger.1
Context Mid training progresses through 32K, 128K and 200K; SFT and the released config support 202,752 positions. Selected long context comparisons run at 128K.12

Architecture and selected attention

The released config has 78 decoder layers, comprising three initial dense FFNs and 75 expert FFNs, a 6,144 wide stream, 64 attention heads, and eight selected experts from 256 plus one shared expert. Appendix A agrees on three dense plus 75 MoE layers, while the narrative body says 80 layers. This study follows the explicit artifact and appendix contract rather than silently equating the counts.21

Select useful history; keep agent learning moving
Network path
  1. Token streamText, reasoning and serialized tool results
  2. DSA indexerSelect up to 2,048 prior positions
  3. Sparse MLAAttend over selected content latents
  4. Expert FFN1 shared + 8 routed experts

The indexer chooses positions, not experts. The expert router chooses computation, not history.

Asynchronous training path
  1. Rollout servicesAgents interact with environments
  2. TITO gatewayExact token IDs and rollout probabilities
  3. LearnerCorrect mismatch and filter stale samples
  4. Weight refreshUpdate actors periodically

Weight refresh feeds back to rollout services; it is a loop, not the terminal output of training.

Figure 1. Two separate selection and feedback systems. Original explanatory diagram informed by the report and configuration.12

MLA compresses the state kept for old tokens. DSA changes which old tokens participate in the expensive attention operation. The artifact’s indexer has 32 heads, width 128, and a top k of 2,048. These selection scores are distinct from the main attention’s final normalized scores.21

For a sequence of length nn and at most kk selected keys per query, the main sparse attention work is proportional to nknk rather than n2n^2, holding dimensions fixed. This describes that selected operation, not the whole model: indexer work, cache storage, expert layers and communication remain.1

The report adapts a dense attention base after mid training: first train the indexer with the base frozen, then continue sparse adaptation for 20B tokens. Table 3’s 128K comparison shows SQuAD improving from 79.7 to 86.0, while HotpotQA falls from 66.3 to 63.0. This is close capability under those tests, not literal losslessness on every task.1

Optimization choices

Muon Split applies matrix orthogonalization independently to head specific projection pieces, allowing different heads to update at different scales. The report explores this because its first MLA/Muon combination did not match the GQA baseline on several tasks. It also changes attention head dimensions to reduce decoding arithmetic under the target hardware constraints. An optimizer and representation can interact; a cache efficient design is not automatically optimization neutral.1

MTP reuses parameters across three training depths rather than storing three independent proposal modules. On a private prompt set with four speculative steps, the reported average acceptance length is 2.76 versus 2.55 for V3.2. Shared proposal weights do not alone prove that every backend can share all depth specific cache state. This is a proposal mechanism with verification, not unrestricted parallel token generation.1

Training across long tasks

Pretraining includes filtered web, code, and math/science data. Mid training adds longer documents, repository files, diffs, issues and pull requests. The reported issue and PR subset contains approximately 160B unique tokens, which is a different unit and scope from the complete 28.5T training exposure.1

Pan on smaller screensOpen full size ↗

Figure 2. Base training and post training. Reproduced from the report’s Figure 5, under CC BY 4.0.1

Figure 2 separates base model exposure and context extension from the later alignment stages. Its final distillation path receives logits from different stage specialists, rather than treating the last RL checkpoint as uniformly best at every task. The token labels are training exposure within the depicted stages, not counts of unique source material.1

Read the stage labels in Figure 2

The pretraining column labels 18T general tokens followed by 9T code and reasoning tokens, both at 4K context. Mid training labels 1T at 32K, 500B at 128K, and 50B at 200K. A separate sparse attention adaptation step is 20B tokens at 200K. These are processed stage exposures, not unique data counts; the report summarizes base training as 28.5T rather than treating every rounded diagram label as an exact accounting ledger. On the right, SFT, reasoning RL, and general RL supply teacher logits, while the general RL checkpoint also supplies the distillation policy’s starting weights.1

SFT covers general chat, reasoning, coding and agents, with erroneous trajectory segments sometimes retained as context but masked from the loss. It teaches interleaved thinking, preservation of prior thinking during coding, and turn level control of reasoning. Reasoning RL uses a GRPO based objective with training/inference mismatch filtering; agent RL uses a separately described asynchronous objective. Those are not one interchangeable algorithm.1

Synchronous reasoning RL has a different probability correction

This stage samples 32 responses per question in batches of 32 questions. Even with synchronized weights, the inference and training engines can assign different probabilities because of precision and implementation differences. The report separates that mismatch from the learner’s update:3

ρt=πold,train(at∣st)πold,infer(at∣st),rt=πθ,train(at∣st)πold,train(at∣st)(1)\rho_t=\frac{\pi_{\mathrm{old,train}}(a_t\mid s_t)}{\pi_{\mathrm{old,infer}}(a_t\mid s_t)},\qquad r_t=\frac{\pi_{\theta,\mathrm{train}}(a_t\mid s_t)}{\pi_{\mathrm{old,train}}(a_t\mid s_t)} \tag{1}

The mismatch multiplier is kept only when it falls between 1/2 and 2. The separate PPO style ratio is clipped with lower/upper parameters 0.2 and 0.28. The response loss averages over its tokens, and the KL penalty is removed. Those details are not the asynchronous objective below.3

Why asynchronous RL needs more than a queue

Agents finish at different times and may observe multiple actor versions along one trajectory. The learner therefore uses recorded rollout probabilities rather than pretending all samples came from its newest policy. The disclosed importance ratio is:1

rt=exp⁡ ⁣(log⁡πθ(at∣st)−log⁡πrollout(at∣st))(2)r_t=\exp\!\left(\log\pi_\theta(a_t\mid s_t)-\log\pi_{\mathrm{rollout}}(a_t\mid s_t)\right) \tag{2}

Tokens outside a bounded ratio interval are masked from the gradient, not merely assigned the PPO clipped objective. The resulting gradient surrogate uses a masked group advantage times the token log probability, not the PPO minimum of clipped and unclipped terms. The optimizer is reset after actor weight refreshes. Very stale samples and environmental failures are additionally filtered. Token-in-Token-out, or TITO, preserves the actual sampled token IDs and action boundaries; retokenizing final text could assign probabilities to a different sequence.1

In DSA RL, the authors report that inconsistent top k selection can destabilize updates. They use a deterministic operator in their setup and freeze the indexer by default. Determinism should be verified for the actual operator and environment; it is not a blanket guarantee for every implementation named topk.1

Cross stage distillation and systems details

Different stage specialists recover capabilities lost during later RL. The student starts from general RL and receives teacher supervision from SFT, reasoning RL and general RL policies. For each student generated token, the report uses the detached teacher/student log probability ratio as its advantage:4

At=stopgrad⁡ ⁣[log⁡πteacher,infer(at∣st)πθ,train(at∣st)](3)A_t=\operatorname{stopgrad}\!\left[\log\frac{\pi_{\mathrm{teacher,infer}}(a_t\mid s_t)}{\pi_{\theta,\mathrm{train}}(a_t\mid s_t)}\right] \tag{3}

It applies the same GRPO based training machinery with group size 1 and batch size 1,024. This advantage is not a group standardized answer reward. The teacher’s useful judgment and train/inference precision still matter.4

The slime rollout stack uses FP8 inference and separates prefill from decoding to reduce long prompt head of line blocking. Request affinity keeps an agent’s repeated prefix on the same data parallel rank to reuse cache. Pipeline ZeRO-2 gradient sharding, activation offload and chunked output loss reduce training memory. Separately, INT4 QAT adapts an SFT model to a matched quantization kernel; the representative BF16 artifact is not itself an INT4 checkpoint.5

Inference and evaluating the whole agent

Pan on smaller screensOpen full size ↗

Figure 3. Interleaved and preserved thinking are different history policies. Reproduced intact from report Figure 7, under CC BY 4.0.6

Figure 3 distinguishes reasoning between tool steps from retaining that reasoning across user turns. Preserving earlier thinking changes the next input context; it is not an extra attention block. A wrapper that strips those spans cannot be assumed to reproduce the preserved thinking evaluations.6

The decoder emits tokens, tool requests and reasoning. A runtime executes tools and returns observations. Context management decides what history remains. None of these tool effects arise from network logits alone. The report’s BrowseComp context strategy combines retaining recent turns with discarding history at a threshold, so its score includes this orchestration.1

Benchmark wrappers matter. SWE-bench uses OpenHands with a tailored instruction, temperature 0.7, top p 0.95, a 16,384 output cap and 200K context. Terminal-Bench uses both Terminus and Claude Code settings, with different timeouts and output limits. A single comparison bar can conceal those differences unless the harness is retained.1

The official weight card labels the BF16 release MIT, while the pinned code repository’s license text is Apache 2.0. These are different artifact scopes; do not transfer one grant to the other or infer complete training code and data availability from public weights. A current repository includes several later GLM-5 variants; its newest release claims do not establish the behavior of this February checkpoint.789

Lessons

The strongest lesson is the distinction between capacity, history selection, and environment interaction. DSA selects attention history; experts select computation; async RL learns from distributed interactions. Keeping those boundaries explicit makes both reproduction gaps and system tradeoffs easier to see. Compare DeepSeek-V3 for dense latent attention and Kimi K2 for another agentic data and optimization recipe.

References

  1. GLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering, February 17, 2026. Sections 2 to 4, Table 3, Appendix A Table 10 and Appendix B.2; inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22 ↩23 ↩24 ↩25 ↩26 ↩27

  2. Z.ai. GLM-5 BF16 configuration, inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5

  3. GLM-5 Team. GLM-5 report, Section 3.2 and Section 4.1.2: synchronous GRPO precision correction versus asynchronous direct rollout probability masking. ↩ ↩2

  4. GLM-5 Team. GLM-5 report, Section 3.5: detached teacher log probability advantage, group size 1 and cross stage teachers. ↩ ↩2

  5. GLM-5 Team. GLM-5 report, Sections 2.4, 3.6 and 4.1.1–4.1.2: ZeRO-2, activation offload, INT4 QAT, FP8 rollout, prefill/decode disaggregation and prefix affinity. ↩

  6. GLM-5 Team. GLM-5 report, Figure 7, PDF page 10, and Section 3.1. Reproduced unchanged under the exact v1 CC BY 4.0 grant; surrounding prose and original caption removed. Attribution and notice: source notice. ↩ ↩2

  7. Z.ai. Official GLM-5 BF16 weight card, license: mit metadata and release instructions, inspected October 10, 2026. ↩

  8. Z.ai. GLM-5 code repository Apache 2.0 license, inspected October 10, 2026. This repository text is not treated as the weight card’s license. ↩

  9. Z.ai. GLM-5 repository; original GLM-5 artifact links and later variant boundaries. ↩

Navigation

Type to search…

↑↓ navigate↵ selectEsc close