What Your Assistant Should Say When the Answer Is Not in the Docs

Every retrieval assistant runs into questions its corpus can't answer. I've watched teams burn weeks on chunk sizes and embedding models, then ship something that invents a policy the second someone asks about parental leave in a country the handbook never covered. Retrieval quality was never the problem. Nobody had decided, up front, what the system should say when it has nothing to say.

The Failure Mode Nobody Tests For

Eval sets get built from questions the documents already answer well. Which conveniently hides the behaviour that destroys trust fastest. A confident, fluent, wrong answer costs more than silence, because silence prompts a follow-up and fluency prompts action. And gaps aren't some edge case: onboarding material lags reality, deprecated pages linger, half of any operations policy lives in one person's head. So the design question isn't whether the assistant hits a gap. It's what comes out when it does.

Three Kinds of Missing, and They Need Different Answers

Dump every gap into one apology string and you throw away information the user could have acted on. We split it three ways:

• Nothing relevant returned. No passage clears the relevance floor. Hard refusal, plus a pointer to where the question actually belongs.
• Something adjacent returned. Passages are on-topic but miss the specific fact. Give the partial answer with an explicit boundary around what was and wasn't found.
• A contradiction returned. Two documents disagree, usually because one is stale and neither carries a date. Show both, side by side, let the reader judge.

Writing the Refusal So It Is Actually Useful

A good refusal names the scope that was searched, so readers learn the assistant's boundaries instead of guessing at them. It reports what sat nearby, not only what was absent. And it offers a next step: the owning team, the closest document, the request form. Hedged prose is the enemy here. Anything that reads like an answer invites the reader to fill in the rest themselves.

Tip: draft refusal copy with the people who own the content, not with the model. They know where each question actually goes next.

Making the Model Abstain Instead of Improvising

Prompt instructions are the weakest control you have. Tell a model to refuse without evidence and it'll still complete the pattern the moment retrieved passages look roughly plausible. Enforcement belongs in the pipeline, before generation starts. Four checks carry most of the weight:

1. Score floor: discard passages below a relevance threshold, so weak context never reaches the prompt.
2. Span coverage: require each claim to map to a retrieved span, which makes unsupported output detectable instead of invisible.
3. Contradiction detection: flag disagreeing passages instead of silently picking one.
4. Answer-source labelling: mark grounded content separately from general knowledge.

Citations as a Refusal Mechanism, Not Decoration

Attach the source and every response becomes something a reader can falsify in seconds. That's the whole point. Treat it as control flow, not formatting: if nothing can be cited, the system refuses. Far easier to enforce than any instruction about honesty. Cite at passage level, not document level, so verification means reading a paragraph instead of forty pages. The byproduct is worth having too - content owners get a list of what people ask and the docs never answered.

Turning Gaps Into a Content Backlog

Log each abstention with the question, the retrieved candidates and their scores. A bare counter tells you nothing you can act on. Cluster those questions and the missing documents jump out, and then you route each cluster to the team that owns the subject rather than to whoever happens to run the assistant. This loop is the strongest argument for the whole deployment: a refusal that produces a written answer next month has already paid for itself.

Tip: review the log on a fixed cadence, or it turns into a table nobody opens.

Where This Sits in Cloud and On-Premise Deployments

The abstention logic is identical either way. The logs are the sensitive artefact. User questions routinely carry more confidential detail than the documents they search, because people describe their actual situation when they ask. Running on customer infrastructure keeps that gap log inside the perimeter, and that's what makes it usable for regulated content. Corpus boundaries vary per install too, so the scope named in refusal text has to be configurable, not hardcoded. One more interaction to handle: a document the user lacks permission to read should produce a permission-shaped response, never a false gap.

FAQ

Does refusing more often make the assistant less useful?

Measured usefulness comes from answers people trust enough to act on without checking them. A lower answer rate with verifiable citations beats full coverage nobody relies on, because the second kind gets quietly abandoned after the first bad experience.

Should the assistant fall back on general model knowledge when the docs are silent?

Only when it's labelled and visually separated from grounded content. Internal policy questions have no correct general answer, so unlabelled fallback here is indistinguishable from fabrication.

How do we know the assistant is refusing for the right reason?

Build a test set of questions your corpus deliberately doesn't answer, then check those refusals alongside the correct answers on every change. Otherwise a retrieval tweak silently converts abstentions back into guesses.

What We Would Build First

Start with the relevance floor and the abstention log. Both are cheap, both change behaviour on day one. Write the refusal copy before you tune retrieval, because drafting it forces the scope question into the open where people can argue about it. Then treat the accumulated gap list as the deliverable that outlives the model, the vendor and the deployment target. An assistant that says clearly what it doesn't know is the only kind anyone can trust about what it does.