Hallucinations & Mitigations

Prompt Engineering: Constraining Hallucinations with Instructions

A PM's perspective: how to reduce model fabrication via System Prompt when writing requirement documents and integrating AI APIs.

Technique 01 / 04 · Prevention

Select Business Scenario

Scenario A · Customer Service

E-commerce / Finance Customer Service Bot

Users ask about product details and refund policies — the model easily fabricates non-existent rules.

Scenario B · Content Creation

AI Writing Assistant / Document Generation

When generating reports, the model may fabricate data or cite non-existent regulatory documents.

Scenario C · Knowledge Q&A

Enterprise Internal Knowledge Base Q&A

When employees ask about internal processes, the model fills gaps with generic knowledge, which may not match actual company policy.

PM Key Takeaways

Loading…

Conversation Comparison

Unconstrained Prompt

Constrained System Prompt

No System Prompt

User Question → Fabricated answer…

The model predicts tokens directly from training memory; when it hits knowledge gaps, it fabricates fluently.

With a Constrained System Prompt

User Question + If unsure say you don't know → I'm not sure…

The blue constraint words become part of the context, raising the probability of the "admit I don't know" sequence.

When the model predicts the next token, every token in the current context influences the probability distribution. The constraint words injected via System Prompt act as "high-quality prior context": they raise the probability of the "I'm not sure" sequence and lower the probability of fabrication sequences. The essence: use carefully crafted prior context to directionally bias the probability distribution of subsequent output.

⚠ Limitation: The model doesn't know what it doesn't know

When constraint instructions work

When the model has some uncertainty about a question, constraint words can push it toward admitting it doesn't know.
→ Works for: questions beyond knowledge scope, ambiguous queries, time-sensitive information

When constraint instructions fail

When the model is highly confident in a wrong answer, its top-ranked token is already wrong, and constraint words cannot affect that probability.
→ Fails for: systematic errors in training data, outdated knowledge treated as fact

PM Countermeasure

For domains where "the model is highly confident but may be wrong," Prompt constraints alone are insufficient.
→ Combine with RAG (inject real knowledge) or human review as fallback