Distillation
In short
Distillation trains a smaller model to imitate a larger one, or an ensemble of models, so the small model can be deployed instead of running the large one for every user.
- Also called
- Knowledge distillation, Model distillation
- Topic
- How models are trained
- Level
- Some background helps
- First described
- Geoffrey Hinton, Oriol Vinyals and Jeff Dean, March 2015, in "Distilling the Knowledge in a Neural Network", a write-up of work from the NIPS 2014 Deep Learning Workshop1.
Hinton, Vinyals and Dean gave the method this name in a 2015 paper, a write-up of work from the NIPS 2014 Deep Learning Workshop1. The starting point is a cumbersome model, either an ensemble or one large heavily regularised model. Running the whole ensemble for every user is expensive. Distillation transfers what it has learned into a small model that is more suitable for deployment1. They note that Caruana and collaborators had already compressed an ensemble into one model, and they develop a different technique1.
Their recipe uses soft targets: the class probabilities from the large model, not only the single correct label. They raise the temperature of the final softmax so those probabilities are softer, then train the small model at the same high temperature to match them. After training, the small model runs at a temperature of 11. Matching the large model's logits, the numbers that go into the softmax, is a special case of this1. They also add a term that matches the true labels, because the small model cannot match the soft targets exactly1. They report surprising results on MNIST, and say the method significantly improved the acoustic model of a heavily used commercial speech system1.
Language-model distillation often skips that temperature trick. DeepSeek's R1 paper fine-tunes smaller open models on 800,000 samples written by DeepSeek-R1, using supervised fine-tuning only, with no reinforcement learning stage on the distilled models2. The six released models are based on Qwen and Llama, from 1.5 billion to 70 billion parameters2. DeepSeek-R1-Distill-Qwen-7B scores 55.5% pass@1 on AIME 2024. The 32 billion Qwen distill scores 72.6% pass@1 on AIME 2024, 94.3% on MATH-500 and 57.2% on LiveCodeBench2. In the same paper, that 32 billion distill beat a 32 billion model trained with large-scale reinforcement learning instead of distillation2.
Distillation is not the same as serving the large model. Hinton's point is to avoid running the cumbersome model for every user1. The student also does not automatically equal the teacher. On AIME 2024 pass@1, DeepSeek reports 79.8% for DeepSeek-R1, 72.6% for the 32 billion distill and 55.5% for the 7 billion distill. In that table, GPT-4o-0513 scores 9.3%2.
Example
A service does not have to call a huge ensemble of models for every request. Distillation is the training step that tries to put that ensemble's judgement into one smaller model that is practical to run1.
Why it matters
Running the biggest model for every request is expensive. Distillation is how a lab tries to keep much of that model's behaviour in a smaller one it can actually serve1.
Questions people ask
Is distillation the same as fine-tuning?
It is usually done by fine-tuning. The distinctive part is the target. The student is trained to match a teacher model, not only to match human labels1. DeepSeek's language-model version is supervised fine-tuning on answers the large model wrote2.
Does the student match the teacher?
Not exactly. Hinton notes that the small model cannot exactly match the teacher's soft targets1. On AIME 2024, DeepSeek reports 79.8% pass@1 for R1 and 55.5% for its 7 billion distill2.
Is distillation just making a model smaller?
The aim in the original paper is a smaller model that is easier to deploy than the ensemble or the huge model1. The training signal is imitation of that teacher. Storing the same model's weights in fewer bits is a different idea.
Is chatbot distillation the same recipe as Hinton's?
The shared idea is that a smaller model learns from a larger one. Hinton matches softened class probabilities from a softmax1. DeepSeek's R1 distills are fine-tuned on 800,000 text samples the large model wrote, with no reinforcement learning stage2.