Emergence: Why Abilities Show Up All at Once
Bigger models are stronger — that sounds obvious enough. Reality is a little more complicated. Some abilities score near zero while a model is small, and then, once the parameter count crosses a certain order of magnitude, they are simply there. The phenomenon is called emergence, and it is the starting point for every tension in the sections that follow.
Put parameter count on the horizontal axis and the score on some task on the vertical axis, and you see two kinds of trajectory. One climbs steadily with scale, gaining a little with every doubling of parameters. The other hugs the floor for a long stretch and then suddenly lifts off at some point.
The trouble lies in the left half. Before the jump, this curve looks exactly like "this approach is a dead end." With only the early data in hand, there is no way to extrapolate whether it will lift off later, and no way to predict where. Nor is there currently a theory that can tell you in advance at what scale a given ability will appear.
That reads flatly enough. Dragging it once is what makes it land. Below, the model scale is yours to set: push it up from 0.5B and watch six abilities switch on one at a time.
All of the abilities below get cited as examples of emergence in published research and industry discussion. The scales in parentheses are order-of-magnitude references only. Change the architecture, the training data, or the evaluation method, and the same ability shifts noticeably.
The argument runs like this. Multi-step math problems are usually scored on whether the final answer is exactly right; one wrong step anywhere and the whole item scores zero. That scoring is discrete, and it compresses continuous progress into a binary outcome. A model that has improved from wildly wrong to almost right still scores zero. Only when it clears that last bit of the threshold does the score jump off the floor all at once.
Switch to a continuous metric — for instance, the log probability of the correct answer token by token — and the curve from the same set of experiments becomes a smooth climb. The jump disappears, which says the jump came from the ruler, and that the model itself was improving steadily all along.
Discrete metric
Only a clean pass counts. All the progress along the way is recorded as zero and cashed in at the end, which looks exactly like a sudden jump.
Continuous metric
Partial credit counts too. Progress across the same set of models is recorded in full, and the curve turns into a smooth climb.
This rebuttal does not overturn the phenomenon entirely. What it establishes is something more specific: when you see a step-shaped curve, first confirm whether the step comes from the model or from your ruler. For anyone building a product, that reminder is more useful than the debate itself. If your acceptance criterion is "it only passes if it runs end to end," you will most likely see a step in your own data too.
Run your own test before moving to a bigger model
Benchmarks measure general tasks, and your scenario may sit right near some threshold. Moving up one size class might solve it instantly, or change nothing at all. Spend half a day building a test set of twenty or thirty samples and run it on real data; that beats reading ten benchmark writeups.
Don't force a prompt to cover what a small model cannot do
When a task stays wrong no matter how you rewrite the prompt, try one size class up first. If the bigger model gets it right immediately, those elaborate prompts were only papering over a gap in scale. Prompts like that are usually long and brittle, and they break the moment you change model or version.
When something "suddenly works," don't conclude too fast either
The dispute above applies to your own tests as well. Cross-check with a more lenient scoring scheme — credit per step, or credit per field. The conclusion only holds if both rulers show the jump.
Emergence shows that scale buys capability. It also leaves a question open: if bigger models are stronger, why is the whole industry working so hard to make models smaller? Squeezing hundreds of billions of parameters down to a few billion costs exactly the kind of capability this section describes. What compression buys, and whether the price is acceptable, is where the next section starts.