Prompt Design for Grounded Answers Instead of Creative Ones
Building a retrieval augmented assistant that answers from company documents means fighting the model's own instincts. Left alone, it fills gaps with pretrained knowledge and writes it in exactly the same confident register as the sourced parts. So prompt design is where you decide whether an answer belongs to the documents or to the model. Here's the working set I use when the requirement is that every sentence traces back to a passage a reader can actually open.
Why Default Model Behaviour Works Against You
A base language model is tuned to produce a plausible answer. Never to announce that the retrieved context was thin. And the interesting failure in a grounded setup isn't an obviously wrong fact - those get caught. It's a fluent blend of document content and general knowledge, seamless enough that nobody can audit it afterwards. Creative behaviour and grounded behaviour need different instructions, not different amounts of politeness. What you optimise for here is narrow: every claim traceable to a passage you can show the reader.
Separating Retrieved Context From Instructions
Keep the system prompt, the retrieved passages and the user question in structurally distinct blocks. Delimiters that never show up in ordinary documents work best, and XML-style tags do the job fine, so the model knows exactly where evidence starts and stops. Label each passage with an identifier it can quote back - citation then becomes a formatting task rather than a memory task. Don't paraphrase or pre-summarise passages upstream. That's just a second place for drift to creep in. Put task instructions before the context and restate them briefly afterwards, since generation happens closest to the end of the prompt.
Wording That Constrains the Answer to the Sources
Positive instruction beats prohibition. Telling the model what to use outperforms a list of things to avoid, and asking for an answer built from the numbered passages beats asking for one that is "accurate" or "factual". Those adjectives mean nothing operationally.
- Permit the empty answer explicitly and give it a fixed shape, so declining is a valid completion instead of a failure state.
- Ban hedging language that blurs the line between quoted material and inference.
- Require an explicit marker when the model combines two passages into a conclusion neither one states.
Citations as a Structural Requirement
Ask for the passage identifier inline with each claim. A bibliography stuck at the end verifies nothing, because no reader maps it back sentence by sentence. Nobody. Structured output - JSON with an answer field and a sources array - lets you reject a response programmatically when a claim carries no identifier. Validate every identifier against the set actually sent in the prompt: a fabricated citation is trivial to catch and tells you a lot about prompt quality. A citation the reader can open in the original document is what separates a demo from something an auditor accepts.
Handling the Cases Where Retrieval Comes Back Empty or Contradictory
The prompt should distinguish three states: enough evidence, partial evidence, no relevant evidence. For partial evidence, ask for the answerable part plus a named gap, rather than a complete-looking answer with a quiet hole in it. And when two documents disagree? Present both with their sources instead of letting the model silently prefer the newer or longer one. Version and effective-date metadata carried alongside each passage lets it say which policy applied when - usually the real question anyway.
Tip: route the no-evidence case to a fixed response template instead of letting the model compose a fresh refusal each time.
Decoding Settings and Model Choice Are Part of the Prompt
Low temperature is the cheapest grounding intervention you have, and the first thing to check when answers start wandering. A smaller instruction-following model that stays inside the context often beats a bigger one with strong opinions of its own. I've seen it more than once. On-premise deployments pin the model version, which removes a whole class of silent regressions caused by provider-side updates. Treat context length as a budget, not a target: each extra passage raises the odds that an off-topic one drags the answer sideways.
Tip: keep the prompt, model identifier and decoding parameters in version control together, because changing any one of them invalidates your evaluation results.
Testing Whether a Prompt Is Actually Grounded
Build a fixed question set from real user questions, and deliberately include ones the corpus cannot answer. That negative set matters more than the positive one, because a prompt that answers everything is a prompt that invents.
- Sabotage test: remove the passage containing the answer and confirm the model declines instead of reciting it from pretraining.
- Check by hand, at least at first, that each citation resolves to a passage supporting the sentence it sits next to.
- Re-run the whole set after every prompt edit, model change and index rebuild.
Prompt changes are code changes. Treat them to the same review.
FAQ
Does a stricter prompt make the assistant less useful?
It makes it narrower, on purpose. Users trust an assistant that admits ignorance far more than one that is confidently wrong twice a week.
Can prompt design alone stop hallucination?
No. It cuts the rate substantially, but retrieval quality, chunking and citation validation carry the rest of the load.
Do these techniques transfer between models?
The principles do, the exact wording doesn't. Re-run the evaluation set after any swap, including a minor version bump.
What This Buys You
Grounding is a design constraint applied across prompt, retrieval, decoding and validation. Not a sentence bolted onto a system message. The prompt is the contract stating that answers belong to the documents rather than to the model. Separate the context, permit the empty answer, require resolvable citations, keep temperature low, and test against questions your corpus genuinely cannot answer. Then treat every prompt edit as a change that needs the evaluation set re-run before it reaches users.
