Programming Fundamentals · Hashing & Caching: Space for Time

Caches: The Invisible Discount on Your AI Bill

Last lesson's hash table solved “how to find instantly.” This one solves another: don't compute the same thing twice. The “cache hit half-price or even 90% off” on model price sheets—every cent saved on the bill—is the same way of organizing: store results you've computed, fetch next time.

Start with an everyday scene

You once computed “37 × 89 = 3293.” Next day someone asks again—do you redo the long multiplication? No, you just give the answer. A cache is the computer's “just give the answer”: store results by key in last lesson's hash table; same key next time, one-step fetch. Hash tables handle “where & how to find”; caches handle “what's worth keeping.” Together they're the full “trade space for time.”

Interactive 1 · KV Cache: don't recompute the same prefix twice

Every time a large model generates a token, it “looks back” at all prior tokens and computes attention intermediates for each. The key: if the prefix is identical, those intermediates are identical—so why recompute them in round two? Each square below is a token. Play “Round 1,” then “Round 2.” Watch for how fast the green squares appear in round two: they weren't computed—they came from cache.

Computing (burning compute) Computed this round Cache hit, skip compute Not yet

Round 1 conversation

System prompt + question ①—every token computed from scratch

Round 2 conversation

Prefix (system prompt + all of round 1) unchanged

Play round 1 first—watch each token light up as it's computed

Round-2 compute comparison (squares = tokens to compute)

Without KV Cache

0 tokens

With KV Cache

0 tokens

KV Cache stores “already-computed attention intermediates” (each token's Key and Value—that's where the name comes from), not the answer itself. Every extra round lengthens the prefix and saves more—real system prompts are often thousands of tokens; recomputing every round is paying full price every round. On vendor price sheets, “cached input tokens at 10% off” means exactly these green squares.

A review intuition you can use immediately: KV Cache only accepts an “identical prefix.” If your app changes one character of the system prompt every round (e.g. stamps the current time at the front), the cache misses every round and the bill multiplies. Put changing content at the prompt's end, fixed content at the start—a one-minute architecture habit that saves real money.

Interactive 2 · the semantic-cache bill

KV Cache saves on “the same opening.” There's a fiercer move: same question—don't recompute the whole answer. A support bot gets asked “how do I return this?” 10,000 times a day—wordings vary, meaning doesn't. Treat “meaning” as the key (vector similarity; lesson ten), and on a hit return the stored answer with zero model calls. Drag the hit-rate slider—watch for the monthly bill.

Scene: a support bot gets 10,000 questions/day; each LLM call costs about ¥0.02; 30 days a month. Cache hits return the stored answer—no call fee.

Semantic cache hit rate 0%

This month you pay

¥6,000

10,000 real calls every day

Cache saves you

¥0

That's the semantic-cache strategy from the Harness core part saving money

⚠️ But—what if the return policy changes?

The cache still holds a standard answer generated under the old policy. The bot will keep serving it earnestly for days or weeks—wronger than no cache, and more confidently. Engineers say: the hard part of caching isn't storing—it's knowing when to invalidate (jargon: “cache invalidation,” one of CS's two hard problems). Common moves: TTL (e.g. expire in 24 hours), or purge related entries the moment policy updates. Design that step before any cache—or the money you save comes back as support tickets.

Its real form in the AI world

“Don't recompute” is everywhere. These four things you enjoy every day are the same structure underneath.

🧠

KV Cache

Standard kit for LLM inference: attention intermediates for prefix tokens are computed once. Without it, long chats simply don't run.

💬

Semantic cache

Treat “question meaning” as the key; similar asks reuse the answer. High-volume support can cut call fees in half or more.

🌐

Browser cache

Images and styles download once and stay local; the second page load is instant—half of why browsing feels fast.

🗺

CDN

Store content ahead of time in the nearest POP so users everywhere feel like hitting a local server. Cache + geography—same move.

What this lesson wants to share

Cache = don't recompute: store results by key in a hash table, fetch next time—last lesson's structure starts earning money here
KV Cache keys on prefix: fixed content at the prompt head, changing bits at the end—hit rate writes the bill
Semantic cache is fiercer: same-meaning asks skip the model entirely; savings scale linearly with hit rate in high-volume scenes
Three cache questions: what to store (results worth reuse), where (RAM / disk / near the user), when to invalidate (the hardest)
Review lens: when every request re-calls the model / recomputes, ask “why isn't this cached?”