Skip to content

gpt-oss

Open weight reasoning through sparse experts, mixed attention, MXFP4 and a structured agent interface.

Updated View as Markdown

gpt-oss makes a reasoning and tool use model available as downloadable weights, rather than only as a hosted service. Its interesting engineering question is how to combine substantial stored capacity with a smaller active computation and a practical deployment footprint.1

Highlights

  • Sparse capacity, compact active path. The two models select four experts per token. The 120b release stores about 116.8B parameters but activates about 5.1B per token per forward pass.1
  • Memory efficiency is designed into the release. Alternating local and full attention, GQA and MXFP4 expert weights address different parts of inference cost.1
  • Reasoning is an interface, not just more tokens. Low, medium and high effort, Harmony channels and tool handoff delimiters form a structured model contract.12
  • Useful disclosure, not a complete recipe. The model card describes architecture, compute, evaluation and safety, but not a fully reproducible dataset or RL implementation.1

Study scope: the August 2025 model card, public 120b configuration, model card and license. The linked document is a substantive model card, not a separately named technical report. Both sizes are covered; exact implementation dimensions below use 120b. Evidence checked October 10, 2026. No weights or inference code executed.

Model in numbers

Dimension 120b 20b
Total / active parameters 116.83B / 5.13B 20.91B / 3.61B
Layers / experts 36 / 128, top four 24 / 32, top four
Published checkpoint size 60.8 GiB 12.8 GiB
Context 131,072 positions 131,072 positions
Pretraining data Trillions of text tokens; exact count and mixture undisclosed Same disclosure boundary
Training compute 2.1 million H100 hours Almost ten times fewer; no exact number stated

Parameter counts and checkpoint sizes come from Table 1, not the rounded model names. Active counts include the output projection, but exclude input embeddings. Accelerator hours are not dollars or hardware independent FLOPs. The model card associates the compute with the training run without giving a stage by stage cost split.1

Architecture: three separate efficiency levers

Stored capacity, active work, retained history
Decoder block
  1. Residual streamWidth 2,880
  2. RMSNorm + attentionLocal / full layers alternate
  3. Residual additionRetain the input path
  4. RMSNorm + MoEFour selected experts
  5. Residual additionContinue to next layer

36 blocks in 120b; 24 in 20b. All attention remains causal.

Efficiency levers
  1. MoE routingReduce active FFN work
  2. GQA + local layersReduce attention/cache costs
  3. MXFP4 expertsReduce stored weight bytes

These are different savings. None removes all weights, all full-history cache, or the cost of long generation.

Figure 1. Three complementary efficiency mechanisms. Independent explanation of the model card and released configuration; the report has no corresponding architecture figure.13

Pre-normalization RMSNorm precedes attention and MoE. Attention projects the 2,880 dimensional residual stream into 64 query heads of width 64, with eight KV heads. The concatenated query width is therefore 4,096, not the residual width; the output projection maps it back. Each full layer attends to the causal prefix, while alternating local layers use a 128 token window.13

The router selects the four highest scoring experts and applies softmax only over those selected scores. The 120b network has 128 experts per MoE layer; 20b has 32. The card notes that its SwiGLU differs from the usual textbook form through clamping and a skip term within the activation. The reference code caps the gate above seven, clamps the linear branch to minus seven through seven, and computes the gated product with a plus one on the linear branch. Its sigmoid coefficient is 1.702. This is not another outer residual stream addition; ordinary SwiGLU is not checkpoint equivalent.14

RoPE with YaRN extends full layer positions to 131,072. Each attention head also has a learned denominator bias: it provides an option to allocate probability mass to no value token. This is an attention sink mechanism, not an extra token whose value must be appended to the input.13

Quantization and the real memory boundary

Quantization is included in post training, and the official card states that the reported evaluations use the same MXFP4 format. MXFP4 stores expert weights at about 4.25 bits per parameter, including block scale overhead. Experts contain more than 90% of parameters, so quantizing this path greatly reduces the checkpoint footprint. The released configuration excludes attention, routers, embeddings and the output head from that quantization.13

OpenAI describes 120b fitting on an 80 GB GPU and 20b operating with as little as 16 GB memory. Those are deployment footprint claims, not a guarantee of full context, arbitrary batch size or every backend. KV cache, activations, runtime buffers and unsupported low precision kernels can change the requirement. Full attention layers still retain token growing history; local layers can use bounded history in a suitable implementation.12

Training and what remains undisclosed

Pretraining uses text only, with emphasis on STEM, coding and general knowledge. The knowledge cutoff is June 2024. Training uses H100 GPUs, PyTorch, expert optimized Triton kernels and FlashAttention. The exact corpus, token count, optimizer schedule and data mixture are not supplied.1

Post training teaches chain of thought reasoning and tool use with RL techniques described as similar to OpenAI o3. The arXiv abstract also mentions distillation; the PDF body does not give a corresponding recipe.5 It does not identify GRPO, PPO or a complete reward recipe, so this study does not fill those gaps with a familiar algorithm. Reasoning levels are trained behaviors selected through the system prompt, rather than three different parameter checkpoints.1

Harmony: the agent boundary matters

The o200k_harmony BPE vocabulary contains 201,088 tokens. Harmony distinguishes roles, message channels, recipients, turn endings and control handoffs. Analysis, final answers and tool invocations are separate structured messages, not arbitrary prose tags. The model learns the priority order system, developer, user, assistant and tool.12

The model can emit a browser, Python or developer function request. An external harness must execute it and return the tool observation. Downloading weights does not grant network access or provide a secure sandbox. Remove reasoning from completed prior assistant turns, but preserve it while continuing an active tool interaction. Message end, completed assistant turn and tool handoff are distinct delimiters; treating all three as one end token breaks the loop.6 A generic chat template or incorrect handoff handling can break the trained behavior.1

Structured output capability similarly needs the appropriate format and serving support. It is not evidence that every locally generated answer will satisfy a schema without validation.2

Evidence for adjustable reasoning

Pan on smaller screensOpen full size ↗

Figure 2. Reasoning effort trades tokens for accuracy on two benchmarks. Original model card Figure 3, cropped to the complete plots; CC BY 4.0.1

Figure 2 measures average chain of thought plus answer length. It supports a test time accuracy versus generation budget tradeoff on these tasks, not a universal latency curve or a guarantee that high effort helps every request. The broader benchmark comparisons use pass@1 and specified reasoning levels; some math results allow tools while a comparison model’s result does not. Compare matched tool conditions, not headline bars alone.1

The card’s safety evaluations include jailbreak resistance, instruction hierarchy, hallucinations and adversarial fine tuning. Open weights allow later modification, so published default safety behavior is not a system level guarantee after fine tuning or arbitrary tool integration.1

Limits and reusable lessons

This is a text model, not a VLM. Its released weights and inference components are Apache 2.0, with a separate usage policy referenced by the model card; the report artwork uses CC BY 4.0. Public weights do not make the training corpus or complete RL stack public.71

The transferable lesson is to study the network, representation format and execution harness together. Sparse FFNs make stored capacity affordable to activate; quantization makes it cheaper to hold; a correct agent protocol makes its trained skills usable. Continue to MiniMax M1 for a more explicit policy optimization recipe.

References

  1. OpenAI. gpt-oss-120b & gpt-oss-20b Model Card, August 2025. Sections 2 to 5, Tables 1 and 3, Figures 1 to 3 and Appendix 1. Exact report inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20

  2. OpenAI. Official 120b model card and official repository. Interface, deployment and release scope inspected October 10, 2026. ↩ ↩2 ↩3 ↩4

  3. OpenAI. Released 120b configuration. Shapes, layer patterns, positions and quantization inspected; no runtime executed. ↩ ↩2 ↩3 ↩4

  4. OpenAI. Reference attention and activation implementation, sdpa and swiglu functions inspected October 10, 2026; not executed. ↩

  5. OpenAI. arXiv model card abstract, distillation statement in the metadata, distinct from the PDF training disclosure. ↩

  6. OpenAI. Harmony response format, message endings and handling reasoning output in subsequent sampling. Official guide inspected October 10, 2026. ↩

  7. OpenAI. Released checkpoint license, Apache 2.0. Report figure attribution and extraction record. ↩

Navigation

Type to search…

↑↓ navigate↵ selectEsc close