Advanced Finale
Contextual Retrieval: Better RAG
The core assumption of RAG is "retrieve the right Chunk, get the right answer." But in practice there is a fundamental problem: the chunking process itself loses critical information. Contextual Retrieval is designed to fix exactly this.
The Core Problem with Traditional RAG
The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LLM to generate an answer. This workflow has one fatal flaw:
Chunks become ambiguous when stripped of context
Once a document is split into Chunks, each Chunk loses its positional information in the original document. Text that was crystal-clear in context may be completely unintelligible on its own.
Typical Example
"The company's Q2 revenue increased by 3% over the previous quarter."
Which company? Which year? What was the Q1 baseline? Is this growth rate good or bad for the industry? All this critical context is lost during chunking. The vector search may find this Chunk, but the Chunk itself carries severely insufficient information.
Full Document
Split into Chunks
Context Lost
Fuzzy Retrieval
The Core Idea of Contextual Retrieval
Use an LLM to prepend a context prefix to each Chunk before Embedding
The idea is elegantly simple: before vectorizing each Chunk, have an LLM read the entire document and generate a brief contextual description as a prefix for each Chunk. This way, every Chunk carries the necessary context when retrieved.
BEFORE -- Bare Chunk
"The company's Q2 revenue
increased by 3% over the
previous quarter."
Who? When? No way to know.
AFTER -- Chunk with Context
"This chunk is from the company's
2024 Annual Report, specifically
the Financial Performance section.
The company's Q2 revenue
increased by 3% over the
previous quarter."
The LLM-generated prefix automatically adds source, time, and section.
Three-Layer Progressive Optimization
LAYER 01
Contextual Embeddings
Vectorize each Chunk after prepending its context prefix. The prefix contains document title, section position, key entities, etc. During retrieval, each Chunk carries its own context for more precise semantic matching.
LAYER 02
Contextual BM25
Traditional BM25 keyword retrieval also benefits from the context prefix. Keywords in the prefix (e.g., company names, years) allow BM25 to match Chunks that would otherwise be missed due to missing context. Vector retrieval + BM25 dual-path recall, complementing each other's blind spots.
LAYER 03
Reranking
After retrieval, use a Reranker model to re-rank candidate Chunks. The Reranker can more precisely judge the relevance between a Chunk and the query, surfacing the most relevant results. All three layers combined yield the best results.
Performance Data
Retrieval Failure Rate Reduction
What does a 67% reduction in retrieval failure rate mean? Suppose that previously 30 out of every 100 retrievals failed to find the right Chunk (failure rate 30%). After optimization, the failure rate drops to about 10% — two-thirds of retrieval errors are eliminated. For production systems that rely on RAG, this is a qualitative leap.
Cost Trade-offs
No Free Lunch
Increased pre-processing cost: Each Chunk requires an additional LLM call to generate its context prefix. For large-scale document libraries, this pre-processing cost is significant.
Prompt Caching can reduce cost: Different Chunks from the same document share the same document-level context. Using Prompt Caching avoids repeatedly sending the entire document.
Best suited for high-accuracy scenarios: If your RAG system demands very high accuracy (e.g., legal document retrieval, medical Q&A, financial compliance queries), the extra pre-processing cost is worth it. For high-tolerance scenarios (e.g., casual recommendations), it may not be cost-effective.
RAG is not just about chunking plus vector retrieval — every Chunk must carry its own context. The core insight of Contextual Retrieval: the quality bottleneck in retrieval lies in the informational completeness of the Chunk itself; switching to a stronger embedding model helps only marginally. Restoring the lost context to each Chunk can reduce retrieval failure by two-thirds.