Vector Database or Plain Keyword Search: When Each One Wins

Every retrieval project starts as a purchasing question and ends as an editorial one. Teams ask which technology to buy. The honest answer depends on what their users type and what their documents actually say. Keyword search matches literal tokens, vector search matches meaning through embeddings, and neither wins outright. A language model can only reason over the passages you hand it, so retrieval is where correctness is decided. Pick wrong and you get confident answers built on the wrong document, which only looks better than no answer at all.

Where Keyword Search Still Wins

Exact identifiers are the easy case. Part numbers, SKUs, error codes, invoice references, statute paragraphs, chemical names. Support engineers and legal staff already know the internal vocabulary and they type it precisely. Rare tokens get flattened by embeddings, because a model trained on general text has no useful representation of "TX-4471-B". And operationally it stays simple: no embedding step, no model version to pin, cheap reindexing, and a matched term anyone can point at. On small corpora a tuned BM25 index answers everything and adds nothing to maintain.

Tip: if you can predict the exact string a user will type, a keyword index will beat an embedding almost every time.

Where Vector Search Earns Its Keep

Vocabulary mismatch is the everyday reason we reach for vectors. Someone writes "my card got declined" while the document says "authorization failure at settlement". Long natural-language questions often carry no decisive keyword at all. Multilingual collections need a question in one language to reach a document written in another, and departments writing over many years leave a trail of inconsistent terms for one concept. Conceptual asks about policy or procedure hinge on topic, not wording. Chunking matters as much as the model here. A vector represents whatever text you stored together, nothing more.

The Failure Modes Nobody Demos

• Vector search always returns something; the nearest neighbour to nonsense still looks like a valid hit.
• Semantic similarity ignores negation, dates and versions, so a retired policy scores as high as the current one.
• Keyword search fails in the opposite direction, returning zero results when the user's word is a synonym of the indexed one.
• Upgrading an embedding model shifts the space, forcing a full reindex.
• Filters and permissions reason cleanly over keyword indexes but need explicit metadata handling in vector stores.

Tip: test both methods with real questions pulled from your support inbox, never with questions written by the team building the system.

Hybrid Retrieval as the Default

We usually run both and merge. Production logs mix identifier lookups with vague conceptual asks, so why pretend otherwise? Reciprocal rank fusion or weighted scores combine two ranked lists without forcing them onto one scale, and a reranking pass over the merged candidates often buys more quality than swapping embedding models. Hybrid costs more to operate, so treat it as a decision, not an unexamined default.

1. Filter on metadata: document type, date, department, access level.
2. Retrieve twice, keyword and vector.
3. Fuse the two ranked lists.
4. Rerank the merged candidates.
5. Cut to a token budget and pass to the model with sources attached.

What Your Infrastructure Decides for You

Existing search infrastructure frequently supports both modes already, which quietly removes a component from the architecture. On-premise work adds constraints: the embedding model runs locally, and GPU availability shapes what is realistic. Managed vector stores in the cloud are cheap to start with and harder to leave later. Index size, update frequency and latency budget narrow the field faster than any quality benchmark ever will. Whatever you pick has to return identifiable document references, not just scores, so the answer can carry its source passage back with it.

How We Decide in Practice

Take a few hundred real questions and sort them into exact lookup, conceptual or mixed. Build the keyword baseline first. It is cheap, and it shows how much headroom actually exists. Measure retrieval separately from generation, because a correct answer over a wrong passage is luck. Signals pushing toward vectors: paraphrase, multiple languages, inconsistent terminology, long questions. Signals pushing toward keywords: identifiers, expert users, stable vocabulary, small corpora. Then reassess once the assistant is live, because real usage always differs from what the team imagined.

FAQ

Do we need a dedicated vector database, or is a vector extension on our existing database enough?

For most corpora, an extension on the database your team already runs is enough, and far easier to operate. Dedicated stores start paying off at large scale with high query volume. Weigh the cost of one more service against familiarity.

Will a better embedding model fix bad retrieval?

Rarely. Chunking, metadata and a reranking pass are usually higher-leverage than the model itself. If passages are split badly or lack dates and document types, no embedding recovers that.

Can we run semantic search entirely on our own infrastructure?

Yes. Local embedding models are practical given modest GPU capacity, and index storage is predictable. Regulated environments often start here precisely because no document leaves the building.

Summary

Keyword search wins on precision and exact tokens. Vector search wins on paraphrase and concept. Most real corpora contain both kinds of question, which is why hybrid retrieval keeps turning up as the endpoint. The decision belongs to your documents and your users, not to a benchmark table. And whichever you choose, the answer should arrive with the passage it came from, so a reader can check it.