Original GLM asks a useful question: can one network understand visible context in both directions, yet generate missing text one token at a time? Its answer is autoregressive blank infilling. GLM-130B carries that objective into a bilingual 130B model, where numerical stability and practical inference become central engineering problems.12
This page studies two related releases, not one checkpoint. The Hugging Face resource above is original GLM-10B, not GLM-130B. The separate GLM-130B technical report and official release repository document the 130B model and its download instructions.34
Study scope: original GLM report, March 2022 revision; GLM-130B report, October 2023 revision; selected released configurations and licenses. No checkpoint executed. Evidence checked October 10, 2026.
Highlights
- Bidirectional context, causal answers. Observed text can interact freely, while missing spans are generated without seeing future target tokens.1
- An objective is not a permanent architecture. Original GLM uses two learned position coordinates; GLM-130B switches to one-dimensional RoPE. Later causal GLM models should not inherit the original mask by name alone.12
- Scaling exposes numerical bottlenecks. GLM-130B reports embedding-gradient spikes, failed normalization trials and a simple gradient shrink intervention.2
- Quantization depends on the trained model. Its weight distributions permit useful INT4 results, but the report does not establish that all models quantize equally well.2
Model in numbers
| Dimension | Original GLM / GLM-10B | GLM-130B |
|---|---|---|
| Parameters | Report experiments span 110M to 515M; the linked release is a separate 10B checkpoint.13 | 130B dense bilingual model, not an expert model.2 |
| Architecture | Released 10B config: 48 layers, width 4,096, 64 attention heads.3 | 70 layers, width 12,288, 96 heads, FFN inner width 32,768.4 |
| Training data | Original controlled experiments use BooksCorpus/Wikipedia; released 10B card names The Pile. A verified processed-token total for that artifact is not supplied here.13 | Approximately 400B processed tokens, roughly balanced between English and Chinese.2 |
| Training compute | Complete accelerator hours and FLOPs for the selected 10B artifact undisclosed in the inspected card. | Reported 60-day training window on 96 DGX-A100 nodes, eight 40GB GPUs each: 768 GPUs. This is not an exact useful GPU-hour ledger.2 |
| Sequence contract | Linked 10B config: 1,024 positions, with task-dependent mask choices. Longer spans in the paper do not imply an unlimited released context.13 | 2,048-token training sequence length; do not confuse weight quantization with longer context.24 |
Blank infilling, step by step
Imagine a sentence with two missing spans. A conventional causal model cannot look to the right of a missing span. A masked encoder can look both ways, but its usual token-classification objective does not naturally generate an unknown-length answer. GLM keeps the useful parts of both contracts.1
Figure 1. Bidirectional context with autoregressive missing spans. Du et al., original GLM report Figure 2, March 2022 revision; figure-only crop under CC BY 4.0.1
- Sample spans and replace each with a single mask token. The remaining observed sequence is Part A.
- Move the missing spans into Part B, shuffle their order, and mark each span’s beginning and end.
- Let Part A attend to all of Part A, but never to the hidden targets in Part B.
- Let each Part B token attend to all of Part A and previously generated Part B tokens. It cannot see future target tokens, even inside its own span.
The report writes the span-order objective as:1
Here permutes the missing spans, and each span probability factorizes autoregressively over its tokens. The mask prevents label leakage; shuffling changes which recovered spans can provide context for other spans.1
Two positions solve two different questions. The first coordinate identifies the span’s mask location in the corrupted sequence. The second identifies progress within that span, with zero for observed Part A tokens. The original implementation uses learned position embeddings; this is not the same mechanism as modern rotary positions.15
Short spans are sampled with Poisson mean 3, covering about 15% of the input. Longer document blanks cover 50% to 100% of the input, teaching continuation; sentence-level blanks teach conditional generation. Changing the mask distribution changes the tasks the model practices.1
What the original ablation actually establishes
In Table 6, the 340M GLM-Large experiment averages 77.0 on SuperGLUE; removing span shuffling gives 68.5, and replacing cloze-style fine-tuning with classification gives 70.0. These are controlled experiments at that size and setting, not measurements of the linked 10B artifact.1
There is a tradeoff, not universal dominance. Adding the document objective slightly reduces GLM-Large’s SuperGLUE average from 77.0 to 75.7, while enabling useful language modeling. At equal model size, that multitask model does not match the causal GPT baseline’s language modeling performance; the report’s larger models narrow or reverse the gap. The objective’s value depends on the downstream task and budget.1
Scaling the objective to GLM-130B
GLM-130B preserves short blank recovery and long continuation, but changes its positional and numerical design. It uses one-dimensional RoPE and GeGLU, not the original learned two-coordinate scheme. For short spans it discards the intra-span coordinate; for long end spans it continues ordinary positions beyond the visible prefix. The causal target mask already provides token-order constraints.2
Its training mixtures have two different denominators. About 95% of training tokens are self-supervised and 5% come from 74 instruction-style datasets. Within the self-supervised recipe, 30% of sequences use short-span masks and 70% use a long end-span mask. Calling this entirely instruction-free pretraining would be incorrect.2
Normalization, precision and distributed training
The report describes more than 30 unsuccessful large-scale normalization trials before its adopted DeepNorm-style post-LN design. This rescales the residual path and initializes selected projections accordingly. It is a report-specific stability intervention, not evidence that ordinary pre-LN is always unstable.2
Training uses FP16 forward/backward operations with FP32 master weights and optimizer states. Four-way tensor parallelism, eight pipeline stages and data parallelism distribute the workload. The reported hardware/model FLOP utilization values are 43.3% and 32.5%, under the authors’ definitions; they are not wall-clock percentages that can be transferred to a different kernel stack.2
The architecture ablation in Appendix B.3 is a 110M test: perplexity falls from 24.58 to 22.95 with RoPE, and then to 22.31 with GeGLU. This supports a design choice at small scale, not a full 130B factorial ablation.2
A small intervention for a large instability
The team noticed embedding-gradient spikes preceding loss spikes and NaNs. Rather than change the forward embeddings, it shrinks the gradient entering the embedding layer:2
The forward value remains , but the derivative with respect to it is multiplied by 0.1. This is neither freezing embeddings nor shrinking their forward activations.2
Figure 2. Shrink the gradient, not the embedding value. GLM-130B report Figure 4, October 2023 revision; both panels retained, under CC BY 4.0.2
The lower plot in Figure 2 is explicitly a 40B-scale test. It strengthens the stability argument without isolating every cause of the final 130B run’s success. The transferable lesson is to locate the numerical bottleneck before applying a global intervention.2
Inference and INT4: what becomes smaller
Figure 3. Weight distributions affect quantization loss. GLM-130B report Figure 5, October 2023 revision; complete histogram and accuracy panels under CC BY 4.0.2
The disclosed path stores weights in INT4 and retains FP16 activations, dequantizing weights for FP16 computation. It is weight-only quantization, not an all-INT4 execution claim. The authors relate its small quality loss to relatively narrow, symmetric weight distributions; that is evidence about their model, not a universal architectural guarantee.2
Table 2 reports five-shot MMLU at 44.75 for FP16 and 44.80 for INT4, but zero-shot LAMBADA drops from 80.21 to 79.47. The latter is a 0.74 percentage-point drop. “Little loss” is more accurate than “lossless.”2
Reported weight memory is approximately 70GB, including overhead. Full runtime memory also includes KV cache, activations, buffers and parallel communication. The demonstrated four-RTX-3090 setup is tied to the authors’ inference stack, not a hardware guarantee for any present-day backend.2
Evidence limits and release boundaries
The 130B attribution study that compares bidirectional context and instruction-style pretraining uses an English-only 10B model, not retrained 130B variants. Some instruction-mixture gains are concentrated in similarity/coreference tasks and some tasks regress. Treat that as a mechanistic lead with a scale limitation.2
The official 130B weight license restricts use to noncommercial research, while its code is Apache 2.0. The inspected original GLM-10B card does not establish the same license grant. Public weights, code rights and report-figure rights are separate questions.43
Open reproduction questions include the exact training exposure for the selected original 10B artifact, complete compute accounting, and independent verification of INT4 behavior under a modern serving stack. Neither this study nor the inspected artifact check runs the model.
Lessons
GLM is a useful reminder that attention visibility, target order and positional representation are separate design decisions. GLM-130B then shows that scaling an objective requires numerical and systems work, not just more parameters. Continue to GLM-4 to see a later causal, grouped-query checkpoint rather than assuming the original blank-infilling contract persists.
References
-
Zhengxiao Du, Yujie Qian, Xiao Liu et al. GLM: General Language Model Pretraining with Autoregressive Blank Infilling, March 17, 2022 revision, ACL 2022. Sections 2 and 3, Figures 2 and 4, Tables 1 and 6. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15
-
Aohan Zeng, Xiao Liu, Zhengxiao Du et al. GLM-130B: An Open Bilingual Pre-Trained Model, October 25, 2023 revision, ICLR 2023. Sections 2 to 5; Figures 4 and 5; Table 2; Appendices B.3, B.8 and B.9. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22
-
THUDM / Z.ai. Original GLM-10B card and released configuration, inspected October 10, 2026. The resource is GLM-10B, not GLM-130B. ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
Z.ai / THUDM. GLM-130B official repository, model configuration, weight license and code license, inspected October 10, 2026. ↩ ↩2 ↩3 ↩4
-
THUDM. Original GLM implementation, inspected October 10, 2026. Targeted input, position and mask interfaces; not executed. ↩


