Distillation is not one final compression step shared by every model. It can supply pretraining targets, teach reasoning through curated responses, or consolidate several specialist policies. These recipes differ in both the supervision and the capability they try to transfer.123
This page inspects distillation disclosures, not every aspect of each model’s full training system. Sources were checked October 14, 2026. Results are author-reported; no checkpoint was trained or benchmarked here.
Six recipes, six different contracts
Read the training contract
6 recipes| Model and study | Stage and student | Disclosed signal | Do not infer |
|---|---|---|---|
| Gemma 2 | Pretraining of 2B and 9B models | Teacher next-token probabilities.1 | The 27B model uses the same distilled-pretraining recipe. |
| Gemma 3 | Pretraining across the reported family | 256 probability-weighted sampled logits per position; restricted targets are renormalized.4 | These are simply the teacher’s top 256 logits. |
| DeepSeek-R1 | SFT of six dense Qwen/Llama students, 1.5B to 70B | Approximately 800K curated examples; no student RL in this reported experiment.2 | The students inherit the teacher’s MoE architecture. |
| Qwen3 | Lightweight-model post-training | Teacher responses first, then teacher logits on student rollouts.5 | A named KL direction when Section 4.5 only says KL. |
| Qwen3-VL | Strong-to-weak post-training | Text-only distillation of the language backbone, within a larger multimodal recipe.6 | This stage directly distills visual perception. |
| DeepSeek-V4.1-Flash | Final post-training consolidation | Full-vocabulary on-policy distillation using over 40 teachers and asynchronous generation.3 | Teacher count is the number of MoE experts. |
No matching recipes. Try another term or reset the search.
Scroll to compare. Focus view gives the table more room.
The comparison is a synthesis of the cited reports, not a quality ranking. Follow the study links for the wider architecture; use the original reports in the references for the training disclosures.
Gemma: distillation can be part of pretraining
Gemma 2 trains the 2B and 9B students against teacher probabilities rather than only observed next tokens. That changes the supervision during large-scale pretraining, not just the final instruction stage. Its controlled 7B-to-2B ablation uses 500B tokens; that experiment is not the complete released-model recipe or a measurement of the final 9B model’s distillation gain.1
Gemma 3 reduces target handling by sampling 256 logits, weighted by teacher probabilities, then assigning zero target probability outside the samples and renormalizing. The report also keeps its vision encoder frozen during language model training and precomputes image embeddings. Distillation here is not evidence that all visual layers are being updated.4
The engineering question is which information survives this target approximation. The target mass outside a retained set, how the set was chosen, and the normalization rule matter. The practice chapter works through the storage implications without presenting its calculation as Gemma’s measured storage cost.
DeepSeek-R1: a reasoning teacher, dense students
The expanded R1 report documents roughly 800K supervised examples: reasoning data curated through sampling and checks, plus general tasks. Six students start from Qwen or Llama checkpoints. Appendix B.4.3 describes 2–3 SFT epochs; Appendix F explicitly separates these students from the teacher’s RL training. These are response targets, not a disclosed soft-logit objective.2
The transferable lesson is the stage separation: expensive teacher learning can produce a reusable training corpus. The student remains its own architecture and is evaluated as its own model. It does not inherit the teacher’s internal computation merely by learning its responses.
Qwen: demonstrations, then student experience
Qwen3’s lightweight path first learns teacher responses, then matches teacher logits on student-generated sequences. In the original report, the named teachers are Qwen3-32B or Qwen3-235B-A22B.5 In Figure 1, follow the separate lightweight-model branch.
Figure 1. Teacher training and student training are distinct paths. Qwen Team, original report Figure 1; attributed figure-region excerpt for analysis.57
The report’s controlled comparison starts from the same off-policy-distilled 8B checkpoint. On-policy distillation uses 1,800 GPU hours, versus 17,920 for the compared RL run, with higher reported scores. These are incremental experiment costs, excluding teacher creation and earlier training. The finding does not establish a universal tenfold saving.5
A VLM can improve without direct visual distillation
Qwen3-VL explicitly describes its strong-to-weak stage as text-only training of the language backbone. It reports reasoning benefits on text and multimodal tasks. That is compatible with an already visually grounded model using improved reasoning; it is not evidence that text alone teaches a model to read previously unreadable pixels. The surrounding recipe also contains multimodal pretraining, SFT and RL.6
For a VLM, ask which bottleneck changed: seeing the evidence, connecting it to language, or reasoning over evidence already represented. A score alone does not identify that mechanism.
Visual transfer has several possible targets
| Target | Representative method | What it teaches |
|---|---|---|
| Image–text relationships | TinyCLIP | Mimic cross-modal affinity, with teacher-weight inheritance in the studied compression recipe.8 |
| Visual relations plus outputs | LLaVA-KD | Combine visual–textual distribution matching with relationships between visual features.9 |
| Visual evidence through teacher language computation | Switch-KD | Feed student visual features into the frozen teacher pathway and supervise the resulting probability relationships.10 |
These are related but different settings. CLIP-style embedding alignment is not the same objective as training a generative visual assistant.
Figure 2. Visual features are tested through language predictions. Sun et al., Switch-KD v1, original Figure 2; complete figure-region excerpt for analysis.1011
For image and text , the hybrid route in Figure 2 is:
Here, is the student vision encoder; and are the frozen teacher projector and language model. The route requires compatible feature interfaces, not arbitrary model pairs. Switch-KD adds Dynamic Bi-directional Logits Difference loss, selecting informative regions adaptively.10
One implementation consequence: frozen teacher weights do not imply detaching the student features. A gradient used to train those features must still pass through the frozen teacher computation. The detached teacher-target branch in the basic KL example has a different role.
The reported 3B-to-0.5B experiment improves the average over ten multimodal benchmarks by 3.6 points. That aggregate does not establish gains on every visual category or on different model pairs.10
Recent consolidation: many teachers, one student
DeepSeek-V4.1-Flash describes a final full-vocabulary OPD stage across domains, using over 40 teachers, including architecturally heterogeneous teachers. Asynchronous generation improves rollout efficiency. The system can change teachers and mixtures while samples remain in flight.3
This is policy consolidation, not the MoE router choosing 40 experts at inference. The report does not disclose a complete reproducible schedule or isolate the full model’s gain to OPD alone. Operational flexibility also makes version tracking important: a sample needs its generating policy and teacher configuration, not merely a prompt and response.
Beyond autoregressive response transfer
DiffusionGemma’s August 2026 report uses a joint sampler-distillation and RL stage after SFT. Its online teacher produces higher-step denoising trajectories to improve generation quality while teaching useful output in fewer steps. Here, the compressed resource is the sampling trajectory, not necessarily the parameter count.12
DistillSpec offers another boundary case: the student is a draft model that helps a larger target model decode faster. The target remains in the inference loop. This differs from replacing the teacher with a standalone student; the relevant metric includes draft acceptance and verification cost.13
Two other domains broaden the lesson. Speech: Distil-Whisper uses selected teacher pseudo-labels to train a smaller speech recognizer. Retrieval: RocketQAv2 uses dynamic listwise distillation between a retriever and re-ranker. There, the shared events are candidate passages rather than next vocabulary tokens. These are brief connections, not complete studies of either system.1415
Read any future recipe with the same questions
Identify the teacher, its information access, the student initialization, training stage, target, prefix source and objective. Then ask what was measured under controlled conditions. Mark absent quantities as undisclosed rather than filling them in from a neighboring paper or a model’s predecessor.
Continue with Practice and Evaluation to turn those questions into a concrete experiment.
References
-
Gemma Team. Gemma 2: Improving Open Language Models at a Practical Size, 2024, Sections 3.2 and 5; separate distilled 2B/9B pretraining from the 27B recipe. ↩ ↩2 ↩3
-
DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, expanded January 4, 2026 edition, Appendices B.3.3, B.4.3 and F. The original January 2025 report has different section and table numbers. ↩ ↩2 ↩3
-
DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, inspected release report, Section 5.2.4, PDF pages 31–32. Distillation disclosure inspected October 14, 2026. ↩ ↩2 ↩3
-
Gemma Team. Gemma 3 Technical Report, March 25, 2025, Sections 1.3, 2.1 and 2.4; sampled teacher targets, frozen vision encoder and image embeddings. ↩ ↩2
-
An Yang et al. Qwen3 Technical Report, May 14, 2025, Figure 1, Section 4.5 and Section 4.7, Table 21. GPU hours are the compared additional stages, not total model costs. ↩ ↩2 ↩3 ↩4
-
Shuai Bai et al. Qwen3-VL Technical Report, November 27, 2025, Sections 4.1 and 4.3; text-only strong-to-weak distillation within multimodal post-training. ↩ ↩2
-
Qwen3 figure extraction and attribution record, original Figure 1; bounded critical/research reuse, not an MIT artwork license. Additional use assessed in this collection’s figure record. ↩
-
Kan Wu et al. TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance, ICCV 2023, Section 3. ↩
-
Yuxuan Cai et al. LLaVA-KD: A Framework of Distilling Multimodal Large Language Models, ICCV 2025, multimodal/relationship supervision and three-stage recipe. ↩
-
Haoyi Sun et al. Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models, April 16, 2026 edition, Sections 3–4 and original Figure 2. Reported results are not a local reproduction. ↩ ↩2 ↩3 ↩4
-
Figure sources and reuse records, Switch-KD extraction, analysis purpose and documented reuse boundary. ↩
-
Gemma Team. DiffusionGemma Technical Report, August 2026, Section 5, sampler distillation and reinforcement learning. ↩
-
Yongchao Zhou et al. DistillSpec: Improving Speculative Decoding via Knowledge Distillation, ICLR 2024, draft alignment and task/decoding-dependent divergence choices. ↩
-
Sanchit Gandhi, Patrick von Platen and Alexander M. Rush. Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling, November 2023, abstract’s pseudo-label construction and selection. Only this broad connection is covered here. ↩
-
Ruiyang Ren et al. RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking, EMNLP 2021, revised 2023, abstract’s dynamic listwise distillation. Only this broad connection is covered here. ↩

