Token Cost Engineering · 13 / 13 · Finale

Minimalism in the Age of Compute

After these layers of optimization, you've already beaten about 90% of rough-cut AI products. This finale stitches the whole series into one checklist—and talks about the more important thing behind “saving money.”

Full recapInformation densityFurther reading
Saving Tokens is really about raising information density

Looking back, we covered BPE, pricing tiers, tier-jump traps, the Agent bill, YAML formats, compression algorithms, KV Cache, stop sequences… It all looks like saving money and peeling cost down. Dig deeper, though: saving Tokens is, at root, raising information density. Once you filter format noise, document fluff, and repeated computation, what you feed the model is signal. Higher density means attention scatters less—and hallucinations drop too.

High SNR = high intelligence

There's a side effect too: speed. Fewer Tokens means faster time to first token and shorter end-to-end latency—in consumer products, that decides whether users keep going. Next time you review an engineering plan, run this check: is every Token here contributing value to the final result? If not, consider cutting it. Leave compute for real thinking—that's the aesthetics of lean computing in the AI age.

Full checklist: 13 lessons on one sheet
Read the bill

Introduction: Token cost is a triple map of finance, latency, and quality; AI commercialization is a bet against users. BPE: Chinese carries a ~2× Token tax by nature. Price sheet: T0/T1/T2—the Three Tiers; cached price is only 1/5 of standard price.

Tier-jump traps

GLM 200 cliff: two extra output Tokens and even input gets a retroactive price bump. Qwen 32k red line: cross it and they bill the whole request at the higher tier—use budget-aware truncation. Image tax: 32-pixel alignment + the resolution curse; tier by task.

Agent

Input-dominated: every turn rereads full history; I/O Ratio 62:1; total grows roughly with the square. Four traps: tool truncation at a 2k cap, thinking-mode tiers, three circuit breakers, history as “fixed + summary + last 3 turns.”

Four layers

Syntax layer: decorative Tokens eat 10–20%; use YAML/CSV/Minified JSON. Semantic layer: dynamic Few-Shot saves 87.5%; LLMLingua-2 compresses 5–20×. Architecture layer: stable prefixes hit KV Cache and save ~90%—don't swap tools dynamically, don't use a sliding window. Output layer: negative constraints cut ~30% fluff, polish with Diff, stop sequences.

Three universal red linesThresholdAction
Single-call input< 32k TokensBudget-aware truncation (RAG, multi-image, multi-turn history)
Agent turns< 10 turnsCircuit-breaker mechanism as backstop
I/O RatioWatch > 50:1Agent is spinning—check the workflow first
Further reading · eighteen topics

The original talk came with reading materials grouped by theme—from context engineering and inference-framework guts to economics models—covering the primary sources behind every key claim in this series. Expand as needed.

1. Closed-source “context engineering” and VRAM optimization
Breaks down how Manus uses a “Mask, Don't Remove” strategy to lift KV Cache hit rate from 20% to 95%—core reading on why closed-source models can cost less in Agent scenarios.
How decisive KV Cache is for cost in production, plus cache invalidation problems and fixes under vLLM distributed deployment.
2. Inference framework guts: vLLM vs SGLang
Deep dive into how SGLang's RadixAttention manages VRAM as a tree—and why multi-turn chat often favors it over vLLM.
Benchmarks for multi-turn and shared-prefix scenarios, quantifying throughput and VRAM utilization gaps between the two frameworks.
3. Hardware layer: Huawei Ascend 910B vs NVIDIA
Extremely detailed GPU spec database: H20, H100 vs Ascend 910B on memory bandwidth and interconnect—one of the most authoritative sources on the “memory wall.”
Architecture trade-offs from 910A to 910B—and why peak FLOPs diverge from real performance.
Why MLA-architecture models can't open graph mode on 910B—the soft spot in operator adaptation.
4. Theory: attention sink
Why the first few Tokens soak up so much attention weight—theoretical basis for why Prefix Caching can lock generation paths and cut diversity.
5. VLM visual economics and resolution research
Core source for the “resolution curse” and visual Token redundancy up to 85%: how dynamic resolution selection cuts cost without cutting accuracy.
How LLMs See Images and What It Really Costs You (Medium)
GPT-4o's patching (512×512 tiles and 85/170 Token billing)—how to read OpenAI's stepped cost structure.
Google Gemini API Pricing & Tokenization (Google Cloud official docs)
Gemini's flat 258-Token rate and 768px large-image tiling—key basis for comparing cost-effectiveness with GPT-4o.
6. Agent cost traps and SWE-bench evidence
Stanford SWE-bench postmortem: source of the “$4.60 per problem” and “edit-generation stage = 60% of cost” numbers.
Documents Agent “input-dominated” behavior (input Tokens ~95%) and the “inverse scaling” paradox.
7. Prompt compression, RAG, and hallucination
Microsoft's classic Prompt compression work: compression ratio (5x vs 20x) vs accuracy drop—and why over-compressing loses logic.
RAG cost trade-offs: economics of Re-Ranking costing up to 5000× vector retrieval.
Benchmarks of hallucination rates under long context and adversarial inputs.
8. Core theory & architecture: model routing
One of the foundational papers on “model routing”: train a router on preference data to cut cost while keeping high performance.
“Task decomposition” so small on-device models work with cloud LLMs—must-read for hybrid edge-cloud architecture.
Uses perplexity as a dynamic threshold to decide when to enter “deep think” mode—a trigger mechanism for routing.
9. Enterprise case studies
How Notion routes requests to different backend models by task type (writing vs Q&A)—“product architecture is routing.”
Business numbers for AI support handling 2/3 of chats: average handle time from 11 minutes down to 2.
In latency-sensitive code completion, how local context plus cloud models enable millisecond routing.
10. On-device and classifier techniques
Design trade-offs for a 3B on-device model, 4k context limit, LoRA adapter fine-tuning.
Limits of sub-1B models and “Deep and Thin” architecture optimization.
Why the simplest regex and keyword checks are sometimes the most efficient routers.
Train an efficient intent classifier with very few samples—a core piece of low-cost routers.
11. MCP protocol core and architecture philosophy
Official definitions and design philosophy for MCP's three primitives: resources, tools, prompts.
Code Execution vs. Tool Calling (Anthropic engineering blog)
Original research on “cutting Token use ~98% via code execution,” with a detailed comparison of the two modes.
12. Big-tech games and strategic splits
How Microsoft “degrades” MCP into a plugin source for Semantic Kernel—embrace-and-extend strategy.
Official docs currently support read-only ops only: Microsoft's caution (and liability avoidance) on AI agent writes.
How OpenAI defines an “AI README” to control the instruction interaction layer for Agents.
13. Security risk and enterprise governance
“Shadow Agents,” data leakage, and how MCP can bypass traditional DLP.
Case studies of typosquatting and malicious MCP packages.
Intranet pivoting risk and missing fine-grained auth (Confused Deputy) details.
14. Future architecture: Agent gateways
Latest “gateway-first” practice for MCP security problems—and how Agent Gateway differs from classic API Gateway.
15. Core frameworks and methodology
How BootstrapFewShot, MIPROv2, and friends work: optimize Prompts programmatically.
Foundational work on iterative test-and-self-correct flows for code generation.
16. Frontier algorithms
How LLM self-evaluation filters “similar but harmful” Few-Shot samples.
Assertions in LLM pipelines for auto-correct and backtracking—key to Flow Engineering stability.
17. Economics models and ROI analysis
Counterintuitive finding: where RAG can get more expensive than fine-tuning at scale.
Practical tool to estimate SFT payback from Token prices and call volume.
18. Enterprise monitoring and infrastructure
Deep guide to monitoring TTFT, Token efficiency, and spotting anomalous query patterns.
How to set similarity thresholds to balance Cache Hit Rate vs accuracy.
Closing note

Saving Tokens = raising information density: after noise is filtered, attention focuses, hallucinations drop, and speed goes up.

One review standard: is every Token here contributing value to the final result? If not, cut it.

Leave compute for real thinking—that's the aesthetics of lean computing in the AI age.

Source: This series is adapted from the author's internal talk AI Token Cost Engineering Strategy Share. To go deeper on the engineering side, continue with the RAG, Agent, and context-engineering chapters in the Advanced Hands-on Part.