Long-term Memory
The Cost Problem of Memory Injection
You stored 1,000 memory entries — do you stuff them all into the system prompt every conversation? Or retrieve only the relevant few on demand? Full injection is simple but costly and noisy; on-demand retrieval saves money but can miss things.
Drag the slider to feel the impact of memory volume
Full Injection (dump everything into prompt)
Injected tokens—
Cost per call—
Noise ratio—
Implementation complexity Minimal
On-demand Retrieval (inject only relevant ones)
Injected tokens—
Cost per call—
Noise ratio—
Implementation complexity Requires retrieval system
Recommended Strategy
Recommended Approach for Memory Injection
User sends message
New conversation starts
→
Semantic retrieval
Find relevant memories
(3–10 entries)
(3–10 entries)
→
Precise injection
Only inject relevant ones
into system prompt
into system prompt
→
Model replies
High-quality response
with context
with context
Core principle: The value of memory lies in surfacing the most relevant few entries each time — how many you've stored doesn't matter. Full injection is viable when memory is small, but as memory grows, on-demand retrieval is the only scalable solution.
More memory isn't always better. Stuffing 1,000 entries into every prompt makes it harder for AI to find what matters. A good memory system works like a competent secretary: they don't haul the entire filing cabinet into the meeting room — they place only the three documents needed for today's meeting on the table in advance.