Choosing an Embedding Model for a Company Knowledge Base
Every retrieval augmented assistant rests on one component nobody talks about at the demo stage: the embedding model. It decides which passages come back when someone asks a question, and everything downstream inherits that decision. We treat the choice as an engineering problem with measurable answers. Not a matter of picking whatever topped a leaderboard last month.
What the Embedding Model Actually Decides
The model turns document chunks and user questions into vectors. Retrieval quality is capped by how cleanly it separates your material - and no generation model, however strong, rescues a weak retriever. The assistant only answers from the text it was handed. Bad retrieval means bad citations, not just vague prose, because the cited source is whatever came back from the index. Keep three jobs distinct: chunking, embedding, reranking.
Start With Your Corpus, Not the Leaderboard
Public benchmarks measure general web and academic text. Your knowledge base is contracts, tickets, SOPs, spec sheets and meeting notes. Part numbers, product codenames and internal acronyms behave badly in general purpose models, and mixed-language material needs a multilingual model even when the interface is English only. Write down these corpus traits before shortlisting anything:
- Languages present, including occasional ones
- Document types and their relative volume
- Average chunk length after your chunking rules
- Share of tables and code blocks
- Update frequency and sensitivity level
Hosted API or Self-Hosted Weights
A hosted API gives you a strong baseline with no GPU to babysit. The price is that every document and every query leaves the network. Open weight models run on the customer's own hardware, and for regulated or confidential material that is frequently the only acceptable answer. Latency differs too: an API call adds a round trip that a warm local GPU does not. Switching later means re-embedding everything, so treat this as semi-permanent and version the index.
Dimensions, Context Window and Index Cost
Vector width drives storage and memory in the database, and wider is not automatically better at finding things. Models with truncatable dimensions let us trade a little accuracy for a much smaller index, which starts to matter once the corpus grows. The input limit sets a hard ceiling on chunk size - anything longer gets silently cut. Tip: measure index size and query latency at your real document count, never on a few hundred sample chunks.
Build an Evaluation Set From Real Questions
Collect what the people who will use the assistant actually ask. Badly worded examples included. Label the passage that answers each one, then score candidates on recall at k instead of subjective spot checks. Fifty to a couple hundred labelled questions usually separates the serious contenders from the rest.
- Gather real questions from future users
- Mark the gold passage for each
- Embed the corpus with every candidate model
- Run identical queries against each index
- Compare hit rate at k, then inspect misses manually
Where a Reranker Fits
A cross encoder reads query and passage together, reorders a wider candidate list and catches what vector similarity alone misses. The usual pattern: retrieve broadly with a fast model, rerank the top candidates, pass a small set to generation. In practice this buys more accuracy than upgrading embeddings does, and you pay for it with extra inference per query. Tip: widen k and add a reranker before swapping models - far cheaper than re-embedding everything.
Practical Selection Path
Shortlist two or three models that satisfy the deployment constraint, then rank them by measured retrieval quality on your own set. Keep the pipeline model-agnostic: store model name and version alongside every vector, so migration is a rebuild rather than a rewrite. Re-evaluate as the corpus grows and new document types show up. And watch hybrid search, because keyword matching still wins on exact identifiers and error codes.
FAQ
Do we need to fine-tune an embedding model on our documents?
Usually no. Better chunking, hybrid search and a reranker deliver more for less effort, and fine-tuning hands you a retraining obligation every time the corpus shifts.
What happens if we change the embedding model later?
The whole corpus gets re-embedded and the index rebuilt. Queries and stored vectors always have to come from the same model, so a partial migration produces silent nonsense.
Can one model serve documents in several languages?
A multilingual model can, but verify cross-language retrieval against your own evaluation set rather than trusting the label.
Summary
The order that works: deployment constraint first, then corpus fit, then measured retrieval quality on questions your users really ask. Benchmarks narrow the shortlist. A labelled evaluation set picks the winner. Design the pipeline so the model can be swapped without touching application logic, because sooner or later it will be.
