Chunking Documents for Retrieval: Size, Overlap and Where to Cut

Retrieval quality gets decided long before a model ever sees a prompt. It gets decided the moment a document is split. A chunk is the smallest thing a retriever can hand back, which makes it also the smallest thing an answer can point at. Two failure modes keep showing up in the systems we look at: chunks too small to mean anything on their own, and chunks too big to rank precisely against a specific question. Embedding a passage squeezes it into one vector, so three topics crammed into one chunk blur that vector into something matching everything weakly and nothing well. And cutting is a citation decision too. Users click the source. A bad boundary drops them into the wrong paragraph.

What Chunk Size Actually Controls

Size trades recall against precision. Bigger chunks carry more context per hit; smaller ones rank sharper. The real ceiling isn't your embedding model's token limit - it's how much unrelated text you're willing to average into one vector. Budget downstream as well: chunk count times chunk size has to fit the answer prompt with room left over. Policy manuals, product specs, support tickets, contracts. Each wants a different target.

• Answers that quote a heading but never the rule beneath it.
• The same long page returned for every question, whatever was asked.
• Chunks whose text alone would not let a human answer.

Tip: read ten random chunks from your own corpus before you tune anything. Most size problems are visible without a benchmark.

Overlap: What It Fixes and What It Costs

Overlap protects facts that straddle a boundary, like a definition separated from the sentence that uses it. Think insurance against bad cut points, not a substitute for good ones. The costs are concrete: the index grows, near-duplicate passages fight for the same slots, and the retrieved set starts repeating itself. Once overlap goes up, deduplication at retrieval time stops being optional. Otherwise the model reads one sentence twice and quietly drops a different source that would have added something. Where structure is strong, with clear headings and numbered clauses, overlap can shrink toward zero. In unstructured transcripts or scanned prose it earns its keep.

Where to Cut: Structure Beats Character Counts

Use the boundaries the author already wrote. Our ranking, strongest first:

1. Document sections and headings.
2. Paragraphs.
3. Sentences.
4. A hard character or token limit, only as a last resort.

Never cut inside a table, code block, list item or numbered clause. Treat them as atomic and let them run over target. Headings should travel with the chunk they introduce, so the passage stands on its own out of context. Keep the breadcrumb too: document title, section path, page or clause number, effective date where versions exist. Semantic splitting on embedding distance sounds elegant, but it's slow, and in practice it rediscovers the paragraph breaks you already had.

Tip: prepend a short header line with the document title and section path to each chunk before embedding. Recall jumps noticeably on questions that name a document instead of describing what's in it.

Metadata, Parents and the Cut You Do Not Have to Choose

Small-to-big retrieval sidesteps most of the size argument. Index small chunks for ranking, then hand the model the surrounding parent section once a chunk wins. Precision and context stop competing, because each one is served by a different object. Store a stable chunk id, parent id, source path and position, so the assistant can point at the exact spot it used instead of waving at a general area. Metadata filters on product, language, department or document version narrow the search space before similarity even runs. For deployments on a customer's own infrastructure, keep the chunker deterministic: a reindex has to produce identical ids, or every stored reference breaks.

Testing a Chunking Strategy Instead of Arguing About It

Build a small question set from questions real users ask, and mark by hand the passage that should answer each one. Check whether that passage shows up in the retrieved set at all before you judge the generated wording. A retrieval miss and a phrasing miss need different fixes. One variable per run.

1. Freeze the question set.
2. Reindex with a single change: size, then overlap, then cut rule, then metadata.
3. Record retrieval hits.
4. Read the failures rather than the average.
5. Keep the failure notes.

Those failing cases are the only reliable guide you have once a new document type lands in the corpus.

FAQ

Is there a chunk size that works for every corpus?

No. The target follows document structure and question style, so start from natural sections and adjust based on the failures you actually see, not a number borrowed from someone else's system.

Do I still need overlap if I split on headings?

Usually much less, because a structural boundary already lines up with a topic boundary. Keep a small overlap for long sections that run on without any internal structure.

Should I rechunk everything when I change the strategy?

Yes, for consistency. Mixed chunking makes similarity scores incomparable across the index, so plan reindexing as a routine operation rather than an emergency.

What We Would Do First

The decision order we keep coming back to: respect the document's own structure, size to the questions being asked, add overlap only where boundaries are genuinely unreliable. Carry metadata and parent context through the pipeline so an answer can show exactly where it came from. Revisit chunking whenever the corpus changes character - it's a parameter, not a launch setting you fix once and forget. In most retrieval systems the cheapest improvement on the table is a better cut, not a bigger model.