Cost Optimization
KV Cache: Trading Space for Speed
In every conversation turn, the model runs Attention computation over all historical Tokens. KV Cache stores the already-computed K/V matrices so only new Tokens need to be computed next turn.
How It Works
Analogy:
Without cache = rewriting the entire textbook from memory every class
With cache = textbook is archived; today you only review today's new notes
Without cache = rewriting the entire textbook from memory every class
With cache = textbook is archived; today you only review today's new notes
Without KV Cache
Every turn: System Prompt + full history is fully recomputed
Cost of turn N = sum of all Tokens from turn 1 to N
Cost of turn N = sum of all Tokens from turn 1 to N
With KV Cache
K/V of historical Tokens is cached; only this turn's new Tokens are computed
Cost of turn N ≈ just the new Tokens added this turn
Cost of turn N ≈ just the new Tokens added this turn
Turn-by-Turn Comparison Demo
Mode:
System Prompt
Redundant Computation
Cache Hit
New Computation
Click "Next Turn" to start · each block = fixed token count, SYS always constant
0
Turns Shown
0T
Without Cache (cumulative)
0T
With Cache (cumulative)
—
Savings Rate
Conclusion: The more conversation turns there are, the greater the KV Cache savings. Keeping your System Prompt constant is the simplest and most effective optimization.
Key Reminders
The longer the System Prompt, the more valuable KV Cache becomes.
A 5,000-Token System Prompt across 1,000 conversation turns saves approximately 80% of total cost with KV Cache. Keeping System Prompt constant = cache hit.
A 5,000-Token System Prompt across 1,000 conversation turns saves approximately 80% of total cost with KV Cache. Keeping System Prompt constant = cache hit.
⚠️ Hidden Pitfall: Distributed Inference Servers
Cloud LLMs typically run on multiple inference servers. Your request may be routed to a different node each time, and that node has no cache of your previous context — implicit cache will always MISS.
Cloud LLMs typically run on multiple inference servers. Your request may be routed to a different node each time, and that node has no cache of your previous context — implicit cache will always MISS.
Coming up next: Explicit caching: guarantee a cache hit with one line of code, with Claude / Qwen / OpenAI code examples and a hit-rate simulation comparison.