Knowledge Distillation
Train a small student model on a large teacher's soft output probabilities — not just hard labels — so it learns the teacher's uncertainty structure, not just its answers.
Intent & Description
🎯 Intent
Compress a large expensive model’s knowledge into a smaller deployable one without training from scratch — soft labels carry far richer signal than one-hot targets.
📋 Context
Training a small model from scratch on the same task consistently underperforms the teacher because hard labels encode only the correct answer. The teacher’s output probability distribution over all classes encodes which wrong answers are “almost right” and why — richer signal the student can learn from.
💡 Solution
Run the teacher on the training data at temperature τ > 1 to soften its output distribution (spreading probability mass across near-correct classes). Train the student using a weighted combination: α × KL(student || teacher soft labels) + (1-α) × cross-entropy(student || hard labels). The soft labels teach the student the teacher’s learned similarity structure. At inference the student uses τ = 1.
Real-world Use Case
📌 TL;DR
Train the small model to match what the big model almost said, not just what it said. Soft labels carry the teacher’s uncertainty structure — that’s the signal that matters.
Advantages
- Student trained on soft labels outperforms same-architecture models trained on hard labels alone
- Soft probability distributions encode the teacher’s learned class similarity structure
- Works across different student and teacher architectures — no structural coupling required
Disadvantages
- Requires teacher inference to generate soft labels over the full training set — adds upfront cost
- Student cannot exceed teacher quality — distillation compresses, it does not amplify
- Temperature τ is empirical and dataset-dependent — needs tuning per task