Hallucinations & Mitigations

Mitigation 3: Temperature & Top-P

PM Perspective: How to configure sampling parameters when integrating APIs — recommended values and risk boundaries for different business scenarios. Technique 03 / 04 · Prevention.

Recommended Parameters by Business Scenario

⚖️

Legal / Compliance / Customer Service

Recommended: T=0.1 Top-P=0.8

📊

Report Writing / Summarization

Recommended: T=0.3 Top-P=0.9

💬

General Chat / Q&A Consulting

Recommended: T=0.7 Top-P=0.95

✍️

Creative Writing / Brainstorming

Recommended: T=1.2 Top-P=1.0

Current Parameter Position

Temperature

ConservativeBalancedDivergent

Top-P

Narrow poolModerateFull set

Parameter Tuning × Output Quality

TemperatureControls randomness level

0.10

Top-PControls candidate token pool

0.80

Estimated output quality under current parameters

Token-by-Token Probability Distribution

Low Temp T = 0.1 — Sharp distribution

6 months

3 months

1 year

…

The highest-probability token is almost locked in,
model output is stable and predictable

High Temp T = 2.0 — Flat distribution

6 months

3 months

forever

anything

…

Low-probability tokens can still be selected,
hallucinations and unexpected tokens increase

Every Token generation step, the model first computes logits for candidate tokens, then applies softmax to get a probability distribution. Temperature scales logits proportionally before softmax: smaller T widens the gap between values, sharpening the distribution; larger T narrows the gap, flattening it. This is the most direct way to intervene in per-step sampling randomness, but it cannot distinguish which steps should be conservative and which can be creative. That's why factual Q&A needs a consistently low temperature, while creative scenarios can open it up.

⚠ Limitation: Lowering the parameter does NOT eliminate hallucinations

Low temperature locks in "most likely" — most likely ≠ most correct

T=0 only makes the model pick the highest-probability token each step. But if the training data had an incorrect answer to that question, the highest-probability token is still the wrong answer — and the output will be consistently, confidently wrong

Affects randomness only — not the knowledge boundary

Temperature controls "how to sample each step," but cannot inject knowledge the model doesn't have: things the model doesn't know, it will still fabricate at low temperature — just more consistently

PM Mitigation Strategy

Parameter tuning is the cheapest first line of defense, but you can't rely on it alone. For scenarios with high factual accuracy requirements, you must pair it with RAG or human review