KV Cache Is an Interface

Play first, then the lecture. The ribbon below is a token sequence of system prompt plus tool schemas (numbers are teaching abstractions). Hit Play and the demo walks through common moves: resend unchanged, change one persona word, add a tool, append conversation, and jitter plugin load order. Each step marks from which token the cache breaks; the meter on the right accumulates the recompute tokens you overpay. Top-right switches DSH mode vs control mode — the jitter step diverges completely.

KV Cache in plain talk. While handling a request, the model caches intermediate results (keys and values) per token. On the next request, if the leading token sequence matches the previous one character for character, that prefix’s work can be reused and the provider discounts the hit. On DeepSeek’s public pricing, cache-hit input tokens are about an order of magnitude cheaper than misses (exact multiplier per the official price page).

The key is “character for character.” Cache matches by prefix: from the first differing token onward, everything is invalid. What’s at the front of an agent request? System prompt plus tool schemas — often thousands of tokens, on every request. Change one persona word, reshuffle tool order, or stuff the current time at the front, and the cache breaks there; every later request recomputes at full price.

So DSH’s conclusion: the prompt prefix is the interface of the model-as-API; its stability is a compatibility promise, maintained like a public API. In engineering terms, three practices.

First: bake it into doc discipline. In the local snapshot, of 268 package READMEs under packages, 215 carry a fixed #### KV Cache effect section. Anything that can appear in a model request must document three parts: what the model sees (What the model sees), token cost (Token effect), and cache impact (KV Cache effect). Take the tool-schema section of packages/core/tools/README.md, line 145:

The same file, lines 186–188, states the inverse: tool-call history and results are append-only; new content trails the reusable prefix and does not break existing cache. What hurts the cache and what doesn’t is written as lookup-ready doc entries.

Second: normalize tool order with a central list. Tool schemas dominate the prefix; their order used to follow plugin registration. Plugins load concurrently, so registration order jitters with the environment — DSH saw divergent request headers in CI (Agent Note 2026-07-06-explicit-tool-order, problem section). Order changes request bytes; bytes change cache — so it must be governed explicitly: a config toolOrder list sets a single order, the list must contain exactly one <unlisted-tools> rest marker, and without a list it falls back to lexicographic order. Normalization runs inside assemble(), before the waterfall; registration order no longer appears anywhere observable.

Third: strict interpolation — throw rather than ship a bad prompt. Persona is a template; variable groups like {{model}} are interpreted strictly against the registry: unknown variables, missing values this assemble, or malformed brace groups all throw. The turn fails immediately; no request is sent. The reason is direct: silent tolerance means sending a quietly mutated prefix, quietly invalidating cache, and possibly quietly changing behavior. Loud failure is cheaper.

Below is the core tool-ordering logic. At the top of the function (lines 165–168) a reserved-name check: if a tool provider uses the reserved name <unlisted-tools>, it throws immediately. Then the sort itself — watch two failure branches: no list → lexicographic fallback; toolOrder names an unregistered tool → throw. Throws happen at assemble time, before any request is sent.

  if (toolOrder === undefined) return tools.sort(compareToolNames)
  const unknown = toolOrder.filter(name => name !== TOOL_ORDER_REST && !knownNames.has(name))
  if (unknown.length > 0) {
    throw new Error(`toolOrder lists unregistered tool${unknown.length > 1 ? 's' : ''} ${unknown.map(name => `"${name}"`).join(', ')}; known tools: ${[...knownNames].sort().join(', ') || '(none)'}`)
  }
  const listed = new Set(toolOrder)
  const rest = tools.filter(tool => !listed.has(tool.name)).sort(compareToolNames)
  return toolOrder.flatMap(name =>
    name === TOOL_ORDER_REST ? rest : tools.filter(tool => tool.name === name))
}

Two edge cases worth remembering — both from Agent Note 2026-07-06-explicit-tool-order. After a hot reload, registration order changes — does tool order change? No: the central list normalizes before the waterfall; registration order is nowhere observable. What if toolOrder has a misspelled tool name? The Note’s consequences section is precise: the turn fails at assemble — no steps opened, no request headers logged, no adapter call; every turn fails the same way until config is fixed; the process itself keeps running.

Strict interpolation is three consecutive throw branches — none let anything through. First, format: if the variable name fails the naming regex, throw “malformed prompt variable reference”; even empty names like {{}} are called out in comments and take this format-error path. Second, registry: if the name isn’t registered, throw “unknown prompt variable”, listing every registered name. Third, values: registered but no value this assemble — throw. All three catch the same thing: a quietly mutated prefix.

Source: interpolation branches in packages/core/system-prompt/src/index.ts lines 277–290, verified on 2026-08-13.

Line 283’s Object.hasOwn has a deliberate twist: ordinary property access on {{constructor}} would walk the prototype chain to Object builtins and mistake them for registered variables. Own-property checks treat prototype-chain names as unregistered. That’s how you stop a quietly allowed bad prompt.

Claude Code: Comments in restored source restored-src/src/tools/AgentTool/prompt.ts lines 57–64 record a real incident. The subagent list used to be nested in the tool description; async MCP connect, plugin reload, and permission-mode switches all mutated that list — one description change invalidated the entire tool-schema cache. That single issue was 10.2% of fleet-wide cache_creation tokens. The fix converges with DSH’s idea: move the volatile list out of the static prefix into a separate attachment message, keep the tool description cacheable (source: claude-code-sourcemap-main/study/chapters/05-multi-agent.md lines 106–121). Timing differs: Claude Code fixed it after seeing 10.2% on the bill; DSH governed order at the CI-jitter stage and spread the discipline across 215 docs.

Grok Build: takes the static-template path. The system prompt is decrypted and rendered from a pregenerated template (crates/codegen/xai-grok-agent/src/prompt/template.rs), then AGENTS.md and skills are appended (module split in same-dir mod.rs). Templates are compile-time fixed, so the prefix is naturally stabler than dynamic assembly — the static path’s built-in advantage. The cost is flexibility: DSH’s model where plugins contribute segments, variables, and tools on the fly doesn’t exist here. Whether Grok has equivalent per-package cache-impact docs: not seen in the verified local snapshot; based on public evidence this stays unknown.