Why a Document Assistant Still Makes Things Up, and How to Stop It
An assistant wired into your company documents answers a policy question in three fluent sentences. The claim appears nowhere in the corpus. That is the failure teams hit first, and it splits into two separate problems: the model inventing content outright, and the model correctly summarising the wrong retrieved passage. There is a third case that almost everyone misses. The answer is right, but the source attached to it does not support it, and that erodes trust faster than a plain mistake ever does. Retrieval narrows what the model can draw on. It does not make invention impossible. Most fabrication you will meet is a plumbing problem before it is a model problem.
Retrieval Fails Before the Model Does
The model answers from what it was handed, so a bad response usually starts with a bad passage set. Semantic similarity is a weak filter for exact identifiers, part numbers, dates and policy version labels. And top-k cutoffs quietly discard the one paragraph that mattered, especially in long repetitive documents where everything looks a bit like everything else.
- Answer contradicts current practice: a superseded document outranked the live one.
- Answer is topically right, factually off: embeddings matched the subject, missed the specific entity.
- Answer truncates a rule: chunk split mid-table or mid-clause.
- Answer is invented wholesale: the corpus simply lacks it.
Chunking and Metadata Decisions You Make Once and Live With
Chunk boundaries encode assumptions about how a document gets read. Contracts, manuals and support tickets each want different treatment. Keep the structural context attached - section heading, document title, effective date - so the model knows what it is holding. Metadata filters on department, document type, validity window and access level cut the candidate pool before any scoring runs, which is cheaper and more reliable than hoping the ranker sorts it out.
Tip: store document date and status as filterable fields, not as text buried inside the chunk, so "current policy only" becomes a query constraint rather than a hope.
Grounding the Answer in the Passage, Not the Model
We instruct the assistant to answer only from the supplied passages and to say plainly when they do not cover the question. "I don't know" has to be an acceptable output, not a state the system contorts itself to avoid. Attaching sources changes reader behaviour too: people verify, and a wrong citation becomes visible immediately. Passage-level attribution beats pointing at a 60-page PDF. Extraction questions are safer than synthesis across many documents, and synthesis deserves tighter guardrails.
Verification Layers Before the Answer Reaches the User
- Does every claim map to a retrieved passage?
- Do quoted figures appear verbatim in the source?
- Does the cited chunk actually contain the supporting sentence?
- Is retrieval confidence above the threshold for answering at all?
When the best passage scores poorly, handing back the documents without a generated answer is the better product decision. Second-pass checking costs latency and tokens, so justify it per use case rather than switching it on everywhere by reflex.
Tip: log retrieved passage IDs beside every answer. Without that trace you cannot separate a retrieval miss from a generation error when someone reports a bad answer weeks later.
Measuring Whether It Actually Improved
Build an evaluation set from real user questions, including the ones the corpus cannot answer. Score retrieval and generation separately. A single end-to-end number hides one regressing while the other improves, and you will chase the wrong layer for a week. Track abstention rate as well, plus whether those abstentions were correct, because an assistant that never abstains is not being honest. Route user corrections back into the set, then re-run after every change to chunking, embeddings, retrieval parameters or prompts. These interact unpredictably.
Where the System Runs Changes What You Can Fix
Cloud and on-premise deployment constrain different things. Running entirely on the customer's own infrastructure keeps regulated or confidential documents inside the organisation's boundary, which in some sectors settles the architecture before anyone gets to talk about accuracy. Self-hosted setups also give full access to logs, the index and the evaluation loop, which makes fabrication far more direct to debug. One thing that is not negotiable: access control has to hold at retrieval time. Answering from a document the user cannot read is a more serious failure than hallucination.
Frequently Asked Questions
Does retrieval augmented generation eliminate hallucination?
No. It narrows the ground the model stands on and makes errors checkable by attaching sources, though a model can still misread a passage or stitch two unrelated ones together. The gain is that such errors become detectable rather than invisible.
Why does the assistant answer confidently when the documents say nothing?
Language models produce fluent continuations, and withholding one is not their default. Unless abstention is explicitly instructed, rewarded and enforced with a confidence threshold, the system prefers an answer to a refusal.
Should we fine-tune the model instead of improving retrieval?
Rarely as a first step. Fine-tuning teaches tone, format and domain vocabulary, but it will not keep the assistant current as documents change, and retrained knowledge cannot be cited back to a source.
What We Would Fix First
Fabrication in a document assistant is usually retrieval, chunking and missing abstention rather than a defective model. So inspect the retrieved passages for the failing questions before you touch the prompt. Then fix chunking and metadata, add abstention and verification, and measure. What makes the system trustworthy in daily use is visible sources and honest abstention, not a higher benchmark score. Build the trace and the evaluation set early, because every later fix depends on knowing which layer failed.
