Hallucination & Mitigations

RAG: Let the Model "Take an Open-Book Exam"

PM perspective: When do you need RAG? How do you design the retrieval pipeline? Where are the real pitfalls?

Strategy 02 / 04 · Prevention

Choose a Business Scenario

Scenario A · Legal / Compliance

Contract Review / Regulatory Q&A Assistant

Regulations update frequently; the model's training data has a cutoff, making it easy to cite outdated or repealed clauses.

Scenario B · Healthcare

Drug Information / Medication Advice Assistant

Medication plans are highly personalized; the model's general knowledge cannot reflect a specific patient's condition or hospital protocols.

Scenario C · Product Docs

SaaS Product Technical Support Bot

Products iterate rapidly; version numbers and navigation paths change often, so the model only remembers the old version it was trained on.

PM Decision Framework: When is RAG Required?

✅

Time-sensitive knowledge: Policies, products, or prices update frequently — the model's training data can't keep up

✅

Private knowledge base: Internal company docs, product manuals — the model has never seen them

✅

High cost of errors: Legal, medical, or financial contexts — one wrong answer can cause serious harm

⚠️

Not needed for chat/creative scenarios: RAG can actually make responses feel rigid and overly constrained

Conversation Comparison

Without RAG (Closed-Book)

With RAG (Open-Book)

Without RAG: Closed-Book Exam

Question → × Guessing from training memory

The context contains only the question itself, so the model can only predict each Token based on impressions from its training data

With RAG: Open-Book Exam

Real document chunks + Question → ✓ Accurate answer

The retrieved documents become part of the context, directly shifting the probability distribution of each subsequent Token

At its core, a model is Token predicting Token — what comes next depends entirely on what came before. RAG works by "injecting retrieved real documents as high-quality context": with the same model, change the context and you change the probability distribution of everything that follows. The model hasn't gotten smarter — it just has a better starting point.

⚠ Limitations: RAG Is Not a Silver Bullet

Retrieval itself can fail

When retrieved chunks are inaccurate, the model treats the wrong document as ground truth — hallucinations don't decrease; they gain the appearance of an authoritative source

The model still fills gaps beyond the retrieved docs

The retrieved context window is limited; anything outside it is still completed from training memory, mixing RAG + hallucination in ways that are harder for users to detect than pure hallucination

PM mitigation strategy

RAG quality ceiling = knowledge base quality. Build retrieval hit-rate evaluation, and use mandatory citation display so users can verify sources