Knowledge Distillation — Roadmap Step & Resources
Training a smaller 'student' model to mimic a larger 'teacher' model
- Level: advanced
- Estimated time: 1 days
- Roadmap: Deep Learning & Neural Networks
Before this step
Interview questions
- What does a student model learn from a teacher's soft labels that it wouldn't from hard labels alone?
Study resources
- Knowledge Distillation Tutorial (Article) — The official PyTorch tutorial on training a smaller student model from a larger teacher.
- Knowledge Distillation Explained (Video) — A practical explanation of compressing a model via teacher-student training.
Part of
- Model Optimization & Deployment for Deep Learning — section
- Deep Learning & Neural Networks roadmap — the full learning path