DeepSeek-R1 is primarily a study in how to train reasoning behavior, not a new attention architecture. It starts from DeepSeek-V3-Base and uses rewards, selected reasoning traces, and additional alignment to make the model spend useful computation before answering.1
Highlights
- R1-Zero isolates an important question. Can reinforcement learning improve reasoning without a preliminary supervised reasoning stage? Its reported experiment says yes, starting from an already pretrained base model, not from no prior learning.1
- R1 makes that behavior more usable. Cold start examples and later curated SFT address readability, mixed languages, and general tasks alongside reasoning rewards.1
- Distillation transfers behavior to smaller networks. The six released dense students learn from selected R1 generated examples through SFT; they are not miniature copies of its expert architecture.1
The useful lesson for researchers is that the reward, training sequence, base model, and inference budget are all part of the reasoning system. Public weights allow these resulting policies to be studied, while incomplete rollout and optimization disclosures limit exact reproduction.12
Study scope: original January 2025 R1 report, original full R1 artifact, tokenizer guidance and license; targeted training and evaluation analysis. The training illustration comes from the September 2025 Nature article; numerical claims remain scoped to the January report unless stated otherwise. No checkpoint executed. Evidence checked October 10, 2026.
Model in numbers
Representative checkpoint: DeepSeek-R1, not R1-Zero, R1-0528, or a distilled student.
| Dimension | Value and scope |
|---|---|
| Parameters | 671B total and 37B active per token in the reported main MoE network.1 |
| Base training | Inherits V3-Base; its 14.8T token pretraining is not a new 14.8T R1 run.13 |
| Post training | Thousands of cold start examples; later approximately 600K reasoning and 200K other SFT examples. RL rollout count undisclosed.1 |
| Compute | R1 specific total FLOPs, accelerator hours and complete financial cost undisclosed in this report.1 |
| Context and output | Release advertises 128K context; config holds 163,840 positions. Reported benchmark output cap is 32,768 tokens, a different quantity.421 |
Network and training are different layers
The full R1 checkpoint uses the V3 decoder architecture: a 7,168 wide stream, 61 layers, latent attention, and expert FFNs after the first three dense layers. Each expert layer selects eight of 256 routed experts plus one shared expert. These dimensions are artifact facts; reasoning behavior is shaped by the subsequent training, not a special “reasoning attention” block.4
Figure 1. The R1 training pipeline. Reproduced from the later Nature article’s Figure 2, under CC BY 4.0.5
Figure 1 distinguishes model weights from training data transfers. R1-Zero generates and filters examples for cold start SFT on V3-Base. The reasoning policy then supplies curated examples, but the later combined SFT stage starts again from V3-Base. Broad RL follows that supervised checkpoint. The arrows carrying sampled data do not mean all stages continuously update one actor.15
GRPO and verifiable rewards
For each question, the old policy samples several complete responses. A verifier or reward model scores them. GRPO compares each score with the group’s mean and standard deviation to obtain an advantage:1
The policy update increases probability for relatively better responses, with clipped importance ratios and a KL penalty to a reference policy. Eliminating the separate critic reduces one source of memory and training complexity. It does not eliminate rollout cost, the reference policy, reward computation, or the need for groups that provide a useful contrast.1
Figure 2. Three policy roles, one group reward signal. Independent explanatory diagram of the January report’s GRPO objective.6
Figure 2 separates sampling from updating. The old policy supplied the responses; the current policy is optimized; the reference policy supplies a separate KL anchor. A verifier or preference model supplies rewards, not the critic baseline that GRPO omits.6
The reported objective and its boundaries
Let . The report presents the following response level surrogate:6
Equation (2) retains the original report’s response level presentation. It is not a claim that every later library uses the same token normalization, importance ratio, zero variance handling, or KL estimator. Equation (1) needs a nondegenerate group; the January report does not provide a full implementation specification for edge cases.6
For R1-Zero, the report uses correctness and format rewards, including mathematical answer checks and executable code tests. It avoids a learned process or outcome reward model in that experiment because of reward hacking and added training complexity. An answer verifier is not a proof that every intermediate reasoning step is faithful or correct.1
R1 later adds a language consistency reward to improve readable reasoning, explicitly accepting a small performance tradeoff in the authors’ ablation. The final broad RL stage uses rule rewards for reasoning and preference models for general tasks. Helpfulness assesses the summary; harmlessness assesses the whole response. “R1 only uses rule rewards” is therefore too broad.1
Curated reasoning and distillation
After reasoning RL, multiple candidate responses are sampled and correct examples retained. Filtering additionally removes mixed language or poorly formatted traces. Some expanded reasoning tasks use a generative judge rather than only an exact verifier. This produces about 600K reasoning examples; approximately 200K general examples complete the later SFT corpus.1
The six dense students are fine tuned on that curated 800K dataset without an RL stage in the reported distillation experiment. Their Qwen or Llama starting checkpoints and licenses must be named separately. They do not inherit MLA, expert routing, or the full R1 parameter count.12
What the training report does not reveal
The original report describes objectives, reward categories and the main stage order, but does not disclose the full corpus, exact RL rollout budget, complete learning rate and batch schedules, or an R1 specific compute ledger. A plausible GRPO implementation is not a faithful reproduction of this run. Later R1 papers or releases can have different disclosures and should not silently overwrite the identity studied here.1
Inference and evaluation budgets
The model generates reasoning tokens followed by its answer, so longer responses consume real decode time and output budget. The network remains an autoregressive decoder with a growing attention history. A visible reasoning trace is output text, not direct access to a complete internal thought process.14
The release recommends temperature 0.6, avoiding a system prompt, and placing instructions in the user message. These are recommendations for this release, not universal rules for all reasoning models.2
For most sampling benchmarks the report uses temperature 0.6, top p 0.95, and up to 32,768 output tokens. It averages correctness over repeated samples to estimate pass@1. Consensus from 64 samples is reported separately and consumes a much larger inference budget. R1-Zero’s AIME 2024 figures of 71.0 pass@1 and 86.7 consensus@64 should never be plotted as equivalent single attempt performance.1
For smaller models, Table 6 compares an internal Qwen 32B RL policy with the distilled Qwen 32B policy: AIME pass@1 is 47.0 versus 72.6 under the reported settings. That supports the utility of this teacher and dataset relative to this RL experiment, not a theorem that SFT distillation always beats RL.1
Failures and lessons
The authors report mixed languages and poor readability in R1-Zero. Longer traces and an illustrative self correction do not demonstrate that every trace is faithful, or establish an internal “aha” mechanism independently of the training and evaluation evidence.1
Why process rewards and tree search were not the chosen recipe
A process reward model needs meaningful step boundaries and reliable intermediate labels. The team found automated labels inadequate, human labeling difficult to scale, and repeated reward hacking costly to correct. They still found process rewards useful for reranking or guided search; their negative result concerns the additional overhead in this large scale RL setup.7
MCTS used tagged reasoning steps and a value model to search for training answers. The language branching space was much larger than a board game’s; capped expansion could get stuck in local optima, and an unreliable fine grained value model limited iterative improvement. This does not establish that all tree search is ineffective.7
The January model also trails V3 on function calling, multiturn interaction, role play and JSON output. Few shot prompts degraded performance in the authors’ tests. Slow software engineering evaluation limited extensive RL in that domain; asynchronous evaluation is described as future work, not an implemented component of this R1 recipe.7
The full R1 code and weights are released under MIT; distilled variants retain obligations from their underlying model families. Training data and a complete training system are not supplied by that license grant.82
The central takeaway is to evaluate a reasoning model as policy plus reward history plus inference budget. Compare the structural costs with DeepSeek-V3, and the switch between thinking and direct answering with Qwen3.
References
-
DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 22, 2025. Sections 2 to 4 and Tables 2 and 6; original version inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22
-
DeepSeek-AI. R1 release repository; model downloads, evaluation settings, usage recommendations and variant license notes. ↩ ↩2 ↩3 ↩4 ↩5
-
DeepSeek-AI. DeepSeek-V3 Technical Report, Section 4; source of the inherited base training budget, not an R1 specific run. ↩
-
DeepSeek-AI. Original R1 configuration, inspected October 10, 2026. ↩ ↩2 ↩3
-
DeepSeek-AI et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, Nature 645, 633–638, September 17, 2025. Figure 2 on page 635. © The Author(s) 2025, CC BY 4.0. Figure artwork unchanged; surrounding article text and original caption removed. This later illustration does not silently replace the January report’s numerical protocol. ↩ ↩2
-
DeepSeek-AI. R1 report, Section 2.2.1, Equations (1)–(3). Policy ratio, clipped surrogate, group normalization and KL reference are reported separately from Section 2.2.2 reward modeling. ↩ ↩2 ↩3 ↩4
-
DeepSeek-AI. R1 report, Sections 4.2 and 5: process rewards, MCTS, prompting, general capability and software engineering limitations. ↩ ↩2 ↩3
-
DeepSeek-AI. R1 MIT License, inspected October 10, 2026. ↩
