Programming Fundamentals · Complexity: Is This Code Worth It?

Why Longer Context Costs More: The O(n²) Bill

Last lesson you got the Big-O ruler—remember that red line taking off? Today we look at the most famous O(n²) inside LLMs—the attention mechanism. You'll suddenly get a pile of old questions: why long chats get slower, why the context window is a “window” not a “warehouse,” why everyone hustles on context compression.

Quick refresh · What attention is doing

Earlier lessons covered this: when an LLM generates each new token, it has to “look back” at every previous token, assign attention weights, and decide what comes next. One token scanning all tokens—translate that into last lesson's language: n tokens, each looking at n, for n × n “eye contacts” total. That's an n×n table. Drag the slider and see it drawn.

Interactive 1 · Attention matrix: what n² looks like

Each row below means “one token looking at every token”; the dark diagonal is it looking at itself. Watch the cell count on the right: the slider moves at a steady pace, but the number jumps harder and harder—that's the feel of “quadratic growth.”

8 tokens
each row = one token looking at every token
TOKEN count n
8
Eye contacts n²
64
= total cells in the attention matrix
n from 4 to 48 is only 12×; cells jump from 16 to 2304—144×. Real chats often hit tens of thousands of tokens—scale this picture up a thousand times in your head, and you'll know what the GPU is computing for you.
Interactive 2 · Same question, two contexts

Same ask—“summarize the key points for me.” One person trims history to 10k tokens first; the other dumps a full 100k-token record in. Tokens differ by only 10×—watch how far the bills diverge. Watch the “attention compute” row: it's not ×10, it's ×100.

Trim camp 10k tokens
Keep only paragraphs relevant to the question, then ask
Attention compute1×
Time to first token (illustrative)≈ 1 s
Input cost this turn (illustrative)≈ ¥0.1
Stuff-it-all camp 100k tokens
Dump the entire history into context as-is
Attention compute100×
Time to first token (illustrative)≈ tens of seconds
Input cost this turn (illustrative)≈ ¥1+
※ Latency and cost are order-of-magnitude sketches, not exact quotes; models and tiers differ a lot
One-line core takeaway: 10× the context, 100× the attention compute. Cost mostly tracks token count (×10 at least; long-context tiers often surcharge), while latency and VRAM pressure track compute—so “stuffing more never hurts” fails for LLMs: every token you stuff gets looked back at by every later token, over and over.
Connecting the dots · This explains three things from earlier lessons
🐢

① Why long chats get slower

The longer you chat, the bigger n, and the more “look-backs” each new token needs. Lag isn't a network issue—it's n² snowballing in the background.

🗜

② Why context compression exists

Compaction summarizes old turns into a short chunk, then continues. Trade a bit of detail for a much smaller n—halve n, cut compute by three quarters. Worth it.

💾

③ Why KV Cache saves money

Cache attention results already computed for the prefix so the next turn doesn't redo them (sister part ds-6 covered this). Precisely because the raw compute is the expensive O(n²), the cache discount is so valuable.

Take this into daily use. Next time a long AI chat lags or the bill spikes, you know what to do: start a new chat, have it summarize before continuing, recall only relevant passages in RAG. Same math behind every trick: shrink n a little, and n² shrinks a lot.

What this lesson wants to share