Progressive Distillation
Distill through intermediate model sizes rather than jumping from the largest teacher to the smallest target — each stage is a shorter compression jump with a better-matched teacher.
Intent & Description
🎯 Intent
Direct distillation from a very large teacher to a very small student loses too much quality in one step. Progressive distillation closes the capacity gap in stages, with each intermediate model serving as teacher for the next.
📋 Context
Distilling a 70B teacher directly to a 1B student is a 70x compression in one pass — too large a jump for the student to faithfully approximate the teacher’s distribution. Quality degrades sharply. Mid-size intermediates provide smoother knowledge gradients.
💡 Solution
Define a distillation chain: Teacher (70B) → Intermediate (13B) → Intermediate (7B) → Target (1B). Each step is a manageable compression ratio where the student and teacher are close enough in capacity for effective knowledge transfer. Each stage uses standard knowledge distillation with soft labels. Intermediate checkpoints are themselves deployable production models.
Real-world Use Case
📌 TL;DR
Don’t jump from 70B to 1B in one step — use intermediates. Each compression step is a shorter fall, and the final model lands better. You get a model family as a byproduct.
Advantages
- Better final quality than direct distillation at the same target compression ratio
- Intermediate checkpoints are independently deployable — a 13B and 7B model as byproducts
- Smoother knowledge gradient — each student has a capacity-matched teacher, not a 70x-larger one
Disadvantages
- Multiple training runs multiply total compute cost proportionally
- Pipeline management complexity grows with chain length
- Errors in an intermediate model can compound and degrade all downstream stages