Knowledge distillation trains a student model using supervision from a teacher. The teacher might provide an answer, probabilities over possible answers, or internal representations. The transfer is learned through training, not simply by shrinking the teacher’s parameter file.12
The useful question is broader than “How do we make a model smaller?” It is: Which capability should transfer, through what signal, and how will we know the student learned it? Students can even have the same architecture and size as their teachers, as studied in Born Again Networks.3
Figure 1 maps the learning contract and its evaluation boundary.
Figure 1. Transfer a useful capability, then test it independently. Original synthesis of teacher supervision, student rollouts and same-size transfer.143
The idea in one minute
A correct answer tells the student what to predict. A probability distribution also exposes alternatives. An explanation can expose a route to the answer. Teacher feedback on the student’s own partial response can address states that polished demonstrations never visit. These are different learning signals, not interchangeable names for the same procedure.546
Three distinctions organize this collection:
- Signal: generated text, token probabilities, features or relationships.
- Experience: reference prefixes, teacher responses, current student responses, or replayed responses from an older student.
- Objective: the discrepancy or task criterion used to update the student.
We use this as an editorial map, not a universal taxonomy. Teacher access, student architecture and available compute constrain which combinations are practical. “On policy” describes whose behavior supplies the training experience; it does not uniquely specify a loss.4
Explore the collection
Foundations
Understand soft targets, temperature, KL divergence and the difference between token and sequence distributions.
Modern Methods
Compare supervision signals, student rollouts, reasoning transfer, context distillation and recent research directions.
LLM and VLM Recipes
Inspect what Gemma, DeepSeek and Qwen actually disclose, with a closer look at visual supervision and multimodal transfer.
Practice and Evaluation
Design a reliable pipeline, check the loss, compare fair baselines and measure the complete cost of the resulting capability.
Read in order for a connected explanation. If you already know the objectives, start with the recipes and use the foundations as a reference.
What this research asks beyond compression
What actually transfers? A student that sounds more helpful may not know more. Gudibande et al.’s imitation experiments exposed this distinction in their tested models; their conclusion is not a verdict on every later distillation method.7
Which “best method” fits the setting? We compare evidence under its teacher access, student size, task and budget. A recent method beating its evaluated baselines is not a universal state of the art.
Does the complete system improve? Better task accuracy can be offset by longer responses, expensive teacher queries or a deployment backend that does not exploit the smaller architecture. The Edge AI collection examines the deployment side; this collection examines the transfer itself.
Local and Cloud Model Collaboration examines cooperation during inference, rather than transferring capability through training.
Scope and evidence
Primary sources were inspected through October 14, 2026. This is a selected research synthesis, not an exhaustive catalog or a reproduced training run. Recent preprints remain identified as such. Model recipes distinguish disclosed details from missing settings and broader interpretation.
Model Deep Dives supply the wider architecture context. Training capable small models connects distillation with data curation and domain adaptation. Document AI and Agents provide application settings where preserving evidence and behavior matters more than imitating style.
References
-
Geoffrey Hinton, Oriol Vinyals and Jeff Dean. Distilling the Knowledge in a Neural Network, 2015, Sections 1–2. ↩ ↩2
-
Adriana Romero et al. FitNets: Hints for Thin Deep Nets, ICLR 2015, intermediate representation supervision. ↩
-
Tommaso Furlanello et al. Born Again Neural Networks, ICML 2018, same-architecture students in the studied vision and language settings. ↩ ↩2
-
Rishabh Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024, Sections 3–4. ↩ ↩2 ↩3
-
Yoon Kim and Alexander M. Rush. Sequence-Level Knowledge Distillation, EMNLP 2016, Section 3. ↩
-
Cheng-Yu Hsieh et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes, ACL 2023, rationale supervision. ↩
-
Arnav Gudibande et al. The False Promise of Imitating Proprietary LLMs, 2023, targeted evaluations versus human impressions in the tested imitation setting. ↩