Hallucinations & Mitigations
RAG Costs & Optimization Strategies
Every RAG query costs extra. Without optimization, costs spiral out of control as usage grows.
Sources of Additional Cost
Document Embedding
Low (one-time)
Run once when indexing documents; reused thereafter
Query Embedding
Low
~$0.014/1M Tokens per query
Vector Retrieval
Medium
Significant latency at large knowledge-base scale
Prompt Expansion
High
500–2000 extra Tokens injected per query; LLM costs double ⚠
The most critical optimization: identify intent first — determine whether RAG is even needed. 70% of conversations don't actually require document retrieval; answering directly with the LLM is faster and cheaper.
PM Response Strategies
Build retrieval hit-rate evaluation: know what percentage of queries successfully retrieve the correct document
Enforce source citation display: lets users verify answers and incentivizes better knowledge-base quality
Clean the knowledge base regularly: RAG quality ceiling = knowledge base quality
Four Optimization Strategies (click to expand)
Keyword Triggering (Filtering)
Cost savings: skip 30–70% of queries
First check whether the question actually needs retrieval. "What's today's date?" doesn't need a doc lookup — answer directly; "What's our refund policy?" triggers RAG.
Implementation: Use an intent classifier or simple rules to pre-filter and skip unnecessary retrieval pipelines.
Implementation: Use an intent classifier or simple rules to pre-filter and skip unnecessary retrieval pipelines.
Model Routing (Tiered Processing)
Overall LLM cost reduction: 60–80%
Use a small (cheaper) model for simple questions; escalate to a flagship model only for complex ones. Don't waste GPT-4 on "Hello."
Implementation: Tiered complexity scoring + model cascade configuration (small model as fallback, large model on demand).
Implementation: Tiered complexity scoring + model cascade configuration (small model as fallback, large model on demand).
Semantic Caching
High-frequency queries: 50% latency & cost reduction
Reuse the same retrieval result for similar questions. "Refund policy" and "how do I get a refund" produce nearly identical results — no need to re-query.
Implementation: When query vector similarity ≥ 0.95, return cached results directly and skip the entire RAG pipeline.
Implementation: When query vector similarity ≥ 0.95, return cached results directly and skip the entire RAG pipeline.
Precise Chunking Strategy
Accuracy improvement: 20–40%
Document chunk granularity directly affects retrieval quality. Too large injects redundant Tokens; too small loses context.
Best practice: ~512–800 Tokens per chunk, with title/paragraph boundaries, preserving semantic integrity.
Best practice: ~512–800 Tokens per chunk, with title/paragraph boundaries, preserving semantic integrity.