Context Management
Local Compression vs. LLM Compression
There are two paths to compressing context: local processing (regex replacement, truncation) is zero-cost but crude; LLM summarization costs money but is precise. Which to choose? Both — but in the right order.
Comparison of the Two Approaches
| Dimension | Local Compression | LLM Compression |
|---|---|---|
| Mechanism | Regex matching, character truncation, template replacement | Have another LLM read and write a summary |
| Cost | Zero (pure local compute) | Costs money (one API call) |
| Latency | <1ms | 1–5 seconds |
| Quality | Crude; may drop key information | Precise; preserves core semantics |
| Best for | Tool output, JSON results, repetitive content | Multi-turn conversation summaries, complex context distillation |
Case Study: Two Ways to Compress the Same Conversation
本地压缩
Original conversation (10 turns · 4,200 tokens):
User: Find me high-speed trains from Beijing to Shanghai
AI: Sure, searching for you… (200-word detailed reply)
[Tool] 12306 results: G1 07:00–11:28 ¥553, G3 08:00–12:35 ¥553, G7 09:00–13:28 ¥553… (15 results total, 800 words)
AI: Found 15 trains, recommending G1… (300-word analysis)
User: G1 looks good, check if there's a business class seat
[Tool] Seat query result: {JSON data, 500 words}
AI: G1 business class: 3 seats available, ¥1,748… (200 words)
User: OK, business class it is. Also search hotels
[Tool] Hotel search results… (600 words)
AI: Recommending Pudong Shangri-La… (400 words)
User: Book that hotel and make me a travel checklist
User: Find me high-speed trains from Beijing to Shanghai
AI: Sure, searching for you… (200-word detailed reply)
[Tool] 12306 results: G1 07:00–11:28 ¥553, G3 08:00–12:35 ¥553, G7 09:00–13:28 ¥553… (15 results total, 800 words)
AI: Found 15 trains, recommending G1… (300-word analysis)
User: G1 looks good, check if there's a business class seat
[Tool] Seat query result: {JSON data, 500 words}
AI: G1 business class: 3 seats available, ¥1,748… (200 words)
User: OK, business class it is. Also search hotels
[Tool] Hotel search results… (600 words)
AI: Recommending Pudong Shangri-La… (400 words)
User: Book that hotel and make me a travel checklist
Compressed result
Click the button above to see
LLM 压缩
Original conversation (10 turns · 4,200 tokens):
(Same original conversation as on the left)
(Same original conversation as on the left)
Compressed result
Click the button above to see
The Right Order: Free Methods First, Then Paid
Recommended Context Compression Pipeline
1
Local Truncation
Drop tool output
Truncate long JSON
Truncate long JSON
→
2
Template Replacement
Replace repetitive
structures with placeholders
structures with placeholders
→
3
Check if Enough
Still over the window?
Proceed to next step
Proceed to next step
→
4
LLM Summarization
Pay AI to
refine the context
refine the context
Compress as much as possible with free methods first; only pay for AI help when you've exhausted local options — that order cannot be reversed. Local compression and LLM compression are sequential stages in a pipeline, not an either-or choice.