Architecture Layer: KV Cache Caveats
In commercial cost engineering, KV Cache is one of the most underrated levers. Hit vs miss can mean up to 90% cost difference—but a few invisible traps sit inside. You might never hear them until someone tells you.
An LLM is “Token pushes Token”: it doesn't care what you asked—only what came before. That means—if two requests share the same prefix, the model is recomputing the same work. KV Cache stores those intermediate results and reuses them on the next identical prefix: cheap storage for expensive live compute—“trading space for time.”
Example: you ask “2 + 4 = ?”, it gets 6. Ask “3 + 4 = ?”—prefix changed, start over. But “2 + 4 + 1 = ?”—prefix “2 + 4” is intact, so it starts from 6 and gets 7. As long as the prefix doesn't change, the cache can hit. DeepSeek, Qwen, and Zhipu all support this (look back at Lesson 3's price sheet: cached price is only 1/5 of standard price). It's a must-check when the author picks an LLM API.
The previous request already cached the full prefix. Tap the three “this request” cases below and see hits (green) vs recomputes (red).
Looks like Token savings—it's a landmine
Some products mount tools by intent to save every Token: WeatherTool for weather, nothing for small talk. The catch is the backend template—take Qwen 3: any request with a tools parameter gets a tool-description block inserted after the System Prompt; if you didn't write a System Prompt, the template adds one and then inserts. Change tool state (on→off, A→B) and the Prompt-head prefix changes—hundreds of thousands of cached Tokens go up in smoke.
Recommended: keep System Prompt stable + don't use the tools field; or waste a few Tokens and keep the full tool set mounted forever—protect the cache hit. System Prompt stability matters more than saving those Tokens.
In multi-turn and long-text products, many use a “sliding window” on long history—keep the last N turns, drop the rest. That's lazy, and it breaks. A sliding window is a FIFO queue: every scroll changes the prefix; the cache never hits.
The author's product Banxie (伴写) (generate later text from earlier text) started with a fixed 800-character rolling window—expensive and prone to “amnesia.” Later it switched to chapter cache: when the AI detects a topic shift (scene change, new chapter), it inserts a separator; the system decides what to compress into a summary and what to keep verbatim—better to keep the prefix stable and hit the cache than let the AI lossily compress on its own.
| Metric | Old (sliding window) | New (chapter cache) |
|---|---|---|
| KV Cache hit rate | ~10% (prefix always changing) | ~80% (prefix stability) |
| Logical coherence | Poor (frequent amnesia) | Good (summary + full chapters) |
| Token cost | High (recompute) | Low (cache reuse) |
| Question | Design decision |
|---|---|
| What must be “kept forever”? | Put it in the Stable zone as the cache prefix |
| What can be “compressed and archived”? | Replace originals with a summary; control window size |
| What needs “load on demand”? | Split by chapter / topic and mount dynamically |
| How to spot “compressible boundaries”? | Design a separator mechanism so the AI marks topic shifts |
Stable prefix → cache hit → save up to 90%. When picking an API, “does it support context caching?” is a must-check.
Don't switch tools dynamically: tool state changes rewrite the System Prompt template and invalidate the entire cache. Better keep the full set mounted.
Sliding windows kill the cache: replace with “Stable prefix + summary archive + chapter mount”—hit rate 10% → 80%.
Source: Adapted from the author's internal team share “AI Token Cost Engineering Strategy” hands-on section “03｜Architecture Layer.” DeepSeek disk-cache pricing: official announcement; review “Token pushes Token” in LLM Fundamentals · Base Model.