05 Training and Adaptation 3 min read 684 words
Knowledge Distillation
Knowledge distillation is the process of transferring the intelligence from a large, complex model ("Teacher") to a smaller, more efficient one ("Student"). This is the secret to the high performance of small models in late 2025.
The Teacher-Student Paradigm
In 2025, small models (e.g., Llama 4 8B, Gemini 3 Flash) are not trained on raw web data alone. They are trained on Synthetic Data generated or curated by a much larger model (e.g., GPT-5.2 or Llama 4 405B).
| Model | Role | Intelligence Source |
|---|---|---|
| Teacher | Large (100B+ Params) | Pretraining on 50T+ tokens |
| Student | Small (1B - 8B Params) | Teacher's filtered logic/output |
How Distillation Works
1. Hard Label Distillation
The student learns from the teacher's final predictions (e.g., the answer to a question).
2. Soft Label Distillation (Temperature Scaling)
The student learns from the teacher's probability distribution (Logits). This is much richer because it tells the student not just the right answer, but which wrong answers were "almost" right.
# Distillation Loss (KL Divergence):
Loss = KL_Div(Teacher_Logits / T, Student_Logits / T)
Where T is the Temperature (typically 2.0 - 5.0).
Feature vs. Output Distillation
Output Distillation (Standard)
Student matches the teacher's text responses.
- Pros: Easy to implement via API.
- Cons: Only learns behavioral surface patterns.
Self-Distillation from Proof (SDP)
The 2025 Reasoning Breakthrough. Models like o1 and DeepSeek-R1 use SDP to improve without new human data.
- Generation: The model generates 100 possible solutions to a hard math/code problem.
- Verification: A rule-based system (compiler/calculator) identifies the 1 correct solution.
- Distillation: The model is fine-tuned on the "Chain of Thought" (CoT) that led to that correct solution.
Result: The model "distills itself" by keeping only the high-quality reasoning paths.
Quantization-Aware Distillation
Standard quantization (e.g., 16-bit to 4-bit) causes a small drop in accuracy. The Fix: Use Knowledge Distillation during the quantization process. The 16-bit model acts as the teacher, guiding the 4-bit model to minimize its error. This is how we get 4-bit models that match 16-bit performance in 2025.
Interview Questions
Q: Why is a distilled 8B model better than an 8B model trained from scratch on the same tokens?
Strong answer: Training from scratch (Pretraining) on raw web data is noisy; the model spends a lot of capacity learning to navigate that noise. A distilled model, however, is trained on a "purified" curriculum. The teacher model acts as a high-quality filter, providing structured logic, clear explanations, and a cleaner distribution of language. Essentially, the teacher provides "hints" through its logit distribution that tell the student exactly which features of the language are most important to learn.
Q: What are the risks of using GPT-4o as a teacher to distill a Llama student?
Strong answer:
- Model Collapse: If the student only sees the teacher's output, it may lose the "long tail" of creative or diverse knowledge and only learn the teacher's narrow biases.
- License Violations: Most proprietary models (OpenAI, Anthropic) have clauses forbidding the use of their outputs to train "competing" models. In 2025, this is a major legal risk for enterprises distilling their own models.
- Linguistic Mimicry: The student might learn to sound confident (like the teacher) without actually having the same level of logical depth, leading to confident but incorrect hallucinations.
References
- Hinton et al. "Distilling the Knowledge in a Neural Network" (2015)
- Gou et al. "Knowledge Distillation: A Survey" (2021)
- DeepSeek. "DeepSeek-R1: Incentivizing Reasoning Capability" (2025)
Key takeaways
01
Soft labels teach more than answers
Matching the teacher's full logit distribution at temperature 2.0 to 5.0 through KL divergence tells the student which wrong answers were nearly right, not just which one was correct.
02
Hidden-state matching needs open weights
Feature distillation copies the teacher's internal representations rather than its text, which the page calls the current standard — and which an API-only teacher cannot provide.
03
A verifier lets a model distill itself
Generate a hundred solutions, let a compiler or calculator pick the correct one, then fine-tune on the chain of thought that produced it: reasoning gains with no new human data.
04
API teachers carry legal and collapse risk
Proprietary terms typically forbid training competing models on outputs, and a student can inherit the teacher's confident tone without its depth, yielding assured hallucinations.