Open Source Special Topic · How Large Models Get Smaller

The Cost of Distillation: Models Are Getting More Alike

The last lesson described distillation as the student learning from the teacher. The problem is that what the student learns is everything the teacher has: the skills, the verbal tics, the biases, even its sense of its own identity. When an entire industry is learning from the same few teachers, the result is that everyone starts to look the same.

Start with What the Research Says

Generative Monoculture in Large Language Models

Wu & Black · arXiv:2407.02209 · 2024

The study found that once multiple models are distilled from the same base, output diversity drops sharply. On tasks that should have many reasonable answers, such as writing book reviews or generating code, different models converge on strikingly similar results. The authors describe this state as a monoculture.

Measuring and Understanding LLM Identity Confusion

Xu et al. · arXiv:2411.10683 · 2024-11

The study covered 27 models and found that 25.93% of them show identity confusion: asked "who are you," they wrongly claim to be a different model. The authors also point out that this kind of error damages user trust more than an ordinary logical mistake does.

Both papers are publicly searchable on arXiv under the IDs above. Verified 2026-08-07.

The Disputes Already Public in the Industry

There is no consensus yet on whether distilling someone else's model crosses a line, but public accusations have already been made.

Anthropic has publicly stated that it found a vendor distilling Claude at "industrial scale." OpenAI has also accused DeepSeek of acquiring capability by distilling the GPT series. These are all one-sided accusations; the accused parties have not admitted them, and no third party has issued any ruling. They are here to make one point: the boundaries in this field are still taking shape. Most vendors' terms of service prohibit using their output to train a competing product, but proving it technically is very hard.

Based on Anthropic's public statements and reporting by Reuters and other outlets. This only states that the dispute exists; it makes no factual finding about the accusations themselves. Verified 2026-08-07.

Three Symptoms You Can Observe Yourself

Symptom 1

Verbal tics inherited wholesale

Certain sentence patterns show up at high frequency in one leading model, then start appearing across a great many models at once, until they become the fingerprint people use to identify AI writing.

Typical patterns "This is not just a technical problem, it is also a product problem"
"Let's dive deeper into this topic"
"It's worth noting that there are several key points here"

Symptom 2

Formatting quirks inherited too

Some models habitually attach an English gloss to nouns when writing Chinese, and the models trained on them write the same way, even when nothing about the context calls for it.

Typical output "This concept (Concept) is extremely important; we need to pay attention to performance (Performance) and safety (Safety)"

Symptom 3

Even the identity gets learned

The most blatant kind. Ask a model who it is, and the name it reports may be the teacher's.

Dialogue User: Which model are you?
Model: I am ChatGPT, developed by OpenAI… (when it is in fact another company's product)

These are not bugs. They are the inevitable outcome of distillation. The training data is the teacher's output, so the student speaks the way the teacher speaks. You cannot inherit the capability without the habits, because in the training data the two simply cannot be separated.

The More Concentrated the Ecosystem, the Worse This Gets

Earlier in this chapter we noted that one open-source family already has more than 200,000 derivative models. From an ecosystem standpoint that is proof of influence, but seen from another angle, it also means 200,000 models share the same set of underlying assumptions.

Biases, knowledge blind spots, and stylistic tendencies inside the base model pass down the distillation chain layer by layer. Change one training strategy upstream and tens of thousands of downstream models change with it. This structure has a familiar name in software engineering: a single point of dependency.

The most direct product impact: multi-model cross-validation may be fake. Plenty of teams call two or three models from different vendors at the same time and have them check each other to cut the error rate. That approach only holds up if they make different mistakes. If the lineage of these models traces back to the same teacher, they will likely go wrong in the same place together, and go wrong in exactly the same way. What cross-validation gives you then is not safety — it is a false sense of safety.

That sounds alarmist until you try it once yourself. Below are six products from six different companies, with different names, different vendors, and different marketing.

What You Can Do

Check the lineage when you select a model

The model card usually states what the base is. When you design multi-model redundancy, favor combinations built on different bases, rather than just different vendor names.

Do not build product differentiation at the model layer

If your competitive edge is "our answers are better," and everyone is using models with closely related lineage, that edge is not solid. Real differentiation usually comes from data, workflow, and understanding of the use case.

Treat verbal tics as something you have to manage

If your product has brand-voice requirements, the default output will most likely carry the upstream model's stylistic habits. Fixing that takes prompt constraints and post-processing; expecting a different model to solve it usually does not work, because they all behave this way.

The first half of this chapter laid out what open source means, how models get smaller, and where the costs land. The next two lessons are hands-on: first work out how large a model your machine can run, then actually get it running.