How Distillation Works: From Teacher to Student
The last lesson covered why you'd want a smaller model. This one covers how. The process is plainer than you might expect: have the teacher answer questions, record how it got there, and train the student on that material. The difficulty isn't in the pipeline — it's in the details.
Assemble the question set
Collect questions that cover the target domain. You can curate them by hand or have a model generate them. This step sets the ceiling on what the student can do: whatever domain you never ask about, the student never learns.
The teacher answers, and you keep the full reasoning
The word that matters is "full." Keep only the final answer and the student just memorizes conclusions; keep the intermediate reasoning chain and it has a chance to learn how to think. That is also why reasoning models make particularly good teachers — they already write their thinking out.
Take the probability distribution, not just the final answer
Every time the model emits a word, what exists inside it is a full probability table over candidate words. Taking only the top entry throws away a lot of information; keeping the whole table transfers a completely different amount. The next section expands on this.
Train the student on that material
The training data has three parts: the original questions, the teacher's complete answers, and the probability distribution at every step. The student's objective is to bring its own output distribution as close as it can to the teacher's.
Evaluate and iterate
Measure how far the student is from the teacher on a test set, then go back and adjust the question set's coverage and the training configuration. This usually takes several rounds.
Say you have an image classification question where the correct answer is cat. Conventional training tells the model: cat is right, everything else is wrong. Distillation hands the teacher's judgment to the student as-is.
There is one more trick involved in practice. If the teacher is too confident, its probability distribution comes out very sharp — something like 98% / 1% / 1% — which is not much different from a hard label, so the advantage of soft labels never shows up.
The fix is to divide by a number T when computing the probabilities, the temperature. The larger T is, the flatter the distribution, and the clearer the relative relationships between words become. This is the same mechanism as the Temperature covered in Part 2, just used for a different purpose: there it was to make output more varied, here it is to pass more information along to the student.
This is the hardest part of the section to picture in the abstract, so here it is to drag directly. Below is the judgment a teacher gives when it looks at a photo of a cat, and T is yours to set.
The pipeline on its own stays abstract. Here is a public example: when DeepSeek released R1 in January 2025, it also open-sourced six distilled models, all trained on reasoning data generated by R1.
One detail in this list deserves attention: the students are not built on DeepSeek's own base models — they were picked from other companies' open-source releases. Four of the six use Qwen2.5, two use Llama3.
DeepSeek never explained the choice, but you can work backwards to a few requirements. The base has to ship open weights under a license that permits further training, or you cannot release what you build. The size range has to be complete enough to cover everything from 1.5B up to 70B in one pass. And the community tooling has to be mature, so training and deployment do not mean rebuilding the wheel. Not many options met all three at the time.
In the evaluations DeepSeek published, the 32B distilled version beat OpenAI o1-mini on several math and code benchmarks, and the company described it as reaching a new best result among the dense models of that period.
That covers most of what distillation buys you: cheap, fast, able to run locally. The next lesson covers the cost. The student learns everything the teacher has, including the teacher's bad habits.