Hybrid Retrieval: Combining Keyword Search With Embeddings

Retrieval augmented generation assistants live or die on which passages reach the model. We build these systems on company documents, and the same thing happens every time: dense vector search on its own leaves holes that only surface once real users start asking real questions. Bolting keyword search onto it isn't a refinement. It's basic coverage.

Why Pure Vector Search Disappoints on Company Documents

Embeddings capture meaning, but they blur exact strings. Part numbers, error codes, contract clause IDs, internal product names - all of it goes soft. Ask about "error E-4412" and back comes semantically adjacent troubleshooting prose, while the one paragraph that actually names the code sits below the cutoff. Corporate corpora are stuffed with low-frequency tokens the encoder never saw during training, so they land in vague regions of vector space. And the failure hides beautifully: the assistant still answers fluently, just from the wrong chunk. We usually catch it in acceptance testing, the moment domain experts start using their own internal vocabulary.

What Keyword Search Still Does Better

BM25 and its relatives score exact term overlap and reward rare terms - exactly where dense vectors trip. They're deterministic and explainable, so when someone disputes an answer we can point at the term that matched. No encoder dependency either, which keeps retrieval alive for jargon or for languages the embedding model handles badly. Indexing is cheap to run and cheap to redo, and that matters a lot for documents revised weekly. The weakness is just the strength inverted: paraphrases, synonyms and questions worded nothing like the source slide right past.

How Hybrid Retrieval Actually Works

Both retrievers run over the same chunk set, then the two ranked lists merge into one. Reciprocal rank fusion combines by position rather than score, which sidesteps the whole mess of comparing incomparable scales. Weighted score fusion normalises both distributions and blends them with a tunable alpha - fine, but it wants recalibration every time the corpus shifts.

1. Chunk once, index into both the vector store and the keyword store.
2. Retrieve top-k from each side independently.
3. Fuse the two lists into a single ranking.
4. Optionally rerank with a cross-encoder.
5. Pass the surviving passages to the model.

Over-fetch before fusion. Pulling thirty candidates per side and cutting to eight after reranking beats pulling eight from each.

Choosing a Fusion Strategy Without Guesswork

Start with reciprocal rank fusion. No score calibration needed and it behaves predictably from the very first query. Graduate to weighted fusion only after an evaluation set shows a consistent lean toward one retriever for your document type. Measure recall at the retrieval stage separately from answer quality, otherwise failures get pinned on the wrong component. And reranking often buys more improvement than fusion tuning does - you pay for it in latency per query.

Tip: build a small labelled question set from genuine user questions before you touch alpha. Without one, tuning is just nudging numbers until the last demo query behaves.

Where Hybrid Retrieval Costs You

Two indexes now. Both monitored, both kept in sync as documents get added, edited or deleted. Query latency climbs: two retrievals, plus fusion, plus optional reranking. More moving parts to inspect when an answer looks off, too. Single-mode retrieval is still the honest choice in a few cases:

• Small, homogeneous corpora where one method already covers the questions.
• Natural-language-only content with no codes, IDs or part numbers.
• Prototypes running on a tight latency budget.

Tip: log retrieved chunk IDs and their per-retriever ranks for every production query. Without that trace, later tuning is guesswork.

Hybrid Retrieval and Answer Provenance

Attaching a source to every answer helps only if the retrieved passage genuinely contains the claim. Keyword matching hands you an auditable reason why a document surfaced, and that strengthens the citation the user sees. Chunk metadata - document title, section, revision date - should survive fusion intact, so the final answer can name where it came from. Trust arrives much faster when someone opens the source and finds the matched phrase sitting right there. For regulated content, an answer without verifiable provenance is worth nothing.

Deployment Considerations: Cloud or On-Premise

Keyword indexes run comfortably on modest hardware, so the on-premise cost question is really about the embedding model and the reranker. Self-hosted deployments lean toward smaller open encoders, which makes the keyword half of hybrid retrieval more valuable, not less. Cloud deployments can afford bigger encoders and hosted reranking, tilting things back toward the dense side. Either way, the index and the documents stay wherever the customer's policy says they stay. Plan your reindexing windows too: swapping the embedding model means recomputing every single vector, while the keyword index sits there untouched.

FAQ

Does hybrid retrieval remove the need for good chunking?

No. Both retrievers score whatever chunks you hand them, and a chunk that separates a definition from its explanation fails in both modes equally.

Can we add keyword search to an existing vector-only system?

Yes, and it's an additive change. Index the same chunks in a keyword store, fuse the results at query time, leave the generation layer alone.

How do we know the hybrid setup is actually better?

Compare retrieval recall on a fixed question set before and after the change. Judging answer quality by impression tells you nothing reliable.

Summary: Treat Retrieval as the Product

Most disappointing assistants fail at retrieval, not generation - the model answers competently from passages that were never the right ones. Hybrid retrieval isn't a clever trick. It's coverage for two known, complementary failure modes. Begin with reciprocal rank fusion, build an evaluation set, add reranking, tune weights last. Decide this at project start, because retrofitting a second index later costs more than putting it in from day one.