GenAI Field Guide

Where it fails

Hallucination

Models are trained and scored in ways that reward a confident guess over an honest "I don't know." They have no internal fact-check, and they fill gaps with plausible text. Hallucination is a structural property, not a bug that gets patched away.

≈2 min
An unanswerable question Training can still reward a guess Confident guess Sometimes scoredright “I don’t know” Can be scored wrong Make uncertainty acceptable Say: “I don’t know” is a valid answer
Training rewarded confident answers over honest abstention, so the model guesses. Your prompt can change the incentive.

What helps, ranked by effect (the author's judgment)

# Do this Why it works
1 Ground it. Give the source and say "answer only from the provided text; if it isn't there, say so." Largest single effect. Removes the need to guess.
2 Give permission to abstain. "If you are not sure, say 'I don't know'; a wrong answer is worse than none." Directly counters the guess-rewarding incentive.
3 Demand checkable citations, then check them. Page numbers, quotes, URLs. A fabricated citation is easy to catch.
4 Decompose. Ask for the list of claims first, then the answer. Smaller steps, fewer places to drift.
5 Second-pass review in a fresh chat with the sources attached. An independent reader catches what the first pass glossed.
6 Use a reasoning mode or lower creativity for factual tasks. Keep the creative settings for creative work.

What does not help

"Don't hallucinate." "Be accurate." "You are an expert." There is no evidence these change the error rate meaningfully. They change the tone, and a more confident tone on a wrong answer is worse.

Try it (demo)

Ask for five real references on a narrow topic and check them. Then re-ask with: "Only cite what you can quote verbatim from a source you retrieved; otherwise say none found." Compare the two lists.