Why Make Models Smaller: The Motivation Behind Distillation
The last section just finished explaining that capability emerges more readily in larger models. So why is the whole industry working so hard to make models smaller? Because capability is not the only constraint. Cost, latency, and whether your data is allowed to leave the building are three things that rule flagship models out of real projects.
Costthe gap is orders of magnitude, not percentages
Drop the parameter count by an order of magnitude and the compute per inference usually drops by an order of magnitude too. The common industry figure is that inference on a small model can cost one to two orders of magnitude less than on a flagship. Exactly how much less depends on size, quantization level, batch size, and deployment method. Don't memorize the numbers; remember that this is a difference in magnitude, not a discount.
Speedrunning locally removes the whole network round trip
Calling a remote flagship model means a network round trip, then queueing, then generating token by token. Waiting several seconds is normal. A small model running locally cuts out the network leg entirely, so the first token arrives much faster. Conversational products are especially sensitive to this: waiting three seconds and waiting three tenths of a second are two different experiences.
On-premise and compliancesome data is simply not allowed off the internal network
Medical records, case files, internal code, unreleased financial data — for this kind of content the compliance requirement is that it never leaves the internal network. However capable the model on the other end is, it is unusable, because the first step already fails. A small model deployed locally is the only route. The control described back in the first section becomes a hard constraint here.
What does "one to two orders of magnitude" turn into on the invoice at the end of the month? Drag in your own product's request volume and see.
"Small" is a relative term, but the test is concrete: does it fit on the GPU you have. There is an interactive tool later in this chapter dedicated to this calculation; here is the formula it uses.
Use 2.6 for FP16 and 0.65 for INT4. The coefficient already includes runtime overhead such as the KV Cache, so don't multiply it in again afterwards.
Here are a few sizes on that basis. The last column is whether it runs on a 24 GB consumer GPU after INT4 quantization.
The conclusion is blunt. Quantized, the 8B size runs on a gaming GPU from several years ago; flagship sizes will not fit no matter how you quantize them. Which turns the question into a different one: can a small model's capability be brought up to standard.
There are two ways to close the capability gap. One is to have the small model learn from scratch on its own, repeating the whole cycle of data, compute, and trial and error. The other is to find a large model that has already learned and let it teach. The second is knowledge distillation.
The difference from conventional training lies mainly in how much information gets transferred at a time.
Training data provides the correct answer. Right earns credit, wrong loses it. What comes back from one item is basically a single "right" or "wrong." Whether it was wildly wrong or off by a hair is invisible in the label. The signal is sparse, which is why it takes enormous data and enormous compute to make up for it.
The teacher provides not just the final answer but its leanings across all the candidates. On the same item, the student can see which options the teacher considered close and which were nowhere near. The same piece of data carries far more information, so the student learns much faster.
How that information is actually transferred and how the training math works is what oss-6 takes apart. For this section, just hold on to the motivation: distillation saves money and time; it does not create new capability.
On 20 January 2025, DeepSeek released R1 and open-sourced a set of distilled small models alongside it. The approach was to generate reasoning data with R1 and then train smaller models on that data. Some of the distilled versions are built on Llama as the base, others on Qwen.
This one is worth noting because it is a publicly verifiable third-party choice. Which bases a company picks when it does distillation says more than any marketing claim. R1 uses the MIT License, and the official announcement states explicitly that training other models through distillation is permitted. Only because that was written down in plain terms did anyone dare take this route seriously.
The naming convention for that batch of models and how it was done in detail come in the next section.
2. Whether something can ship comes down to VRAM first, and the formula is parameter count times the precision coefficient.
3. Distillation has a large model teach a small one, in order to avoid the cost of training from scratch.
4. Small models have a capability ceiling, and they inherit their teacher's defects. Put both on the table before you decide.
The next section covers how distillation actually works: how the teacher sets the questions, how the student checks its answers, and which engineering pitfalls sit in between.