Reranking Retrieved Passages: The Step Most Teams Skip

Most retrieval augmented generation pipelines stop at the vector search and hope the top five chunks are the right five. They usually are not. Reranking sits between retrieval and generation, reorders what came back, and quietly fixes a class of wrong answers that no amount of prompt fiddling will touch. Cheap to add. Easy to skip.

Why Vector Search Alone Leaves Answers on the Table

Embedding search squashes a whole passage into one vector, so it matches topic, not intent. Ask about cancellation deadlines and you get back every paragraph that mentions cancellation. Fine. But the retriever was tuned for recall across millions of chunks, not for deciding which five passages answer this question. Here is the pattern I keep running into: the right document sits at rank 12, the model never sees it, and the answer comes out confidently incomplete. And no, bigger context windows do not save you. Stuff twenty passages in and you dilute the signal and blur attribution.

What a Reranker Actually Does Differently

A cross-encoder pushes the query and the passage through the model together, so it can weigh how the specific wording of the question relates to the specific wording of the text. Bi-encoders do the opposite: embed each side separately, let them meet only at a cosine similarity. That joint interaction is expensive, which is exactly why reranking cannot replace retrieval. It scores candidates. It does not search a corpus. Two stages: the retriever casts a wide net, the reranker orders what came up in it.

Where Reranking Fits in a RAG Pipeline

The stages run in order: chunking and indexing, query handling, first-stage retrieval, reranking, context assembly, generation, source attribution. Reranking lands after retrieval and before prompt assembly, which is what makes it additive - you bolt it on without reindexing a thing. It also sharpens the citation you attach, because the passage the model leaned on is more likely to be genuinely relevant. Hybrid retrieval widens the candidate pool, reranking cleans it up. They compose well.

Choosing a Reranker: Hosted API, Open Model, or None

• Hosted reranking API - fastest to try, but passages leave your infrastructure.
• Open cross-encoder, run locally - full data control, needs a GPU or accepts CPU latency.
• LLM-as-reranker prompt - most flexible, most expensive per query.

On deployments that live on a customer's own infrastructure, the open cross-encoder is usually the only one clearing data residency, so you pick model size against whatever hardware you were given. Multilingual corpora need a multilingual reranker. An English-trained one degrades quietly rather than failing loudly, which is worse.

Tip: rerank with an LLM prompt on a handful of test queries first, confirm it helps on your corpus at all, then swap in a dedicated model for production latency.

The Latency and Cost Budget Nobody Plans For

Reranking cost scales linearly with candidates: scoring a hundred is roughly ten times the work of scoring ten, and it all lands on the critical path before the first token streams. The levers I reach for, in order:

1. Shrink the candidate pool.
2. Cap passage length fed to the cross-encoder.
3. Batch scoring on GPU.
4. Cache reranked results for repeated queries.

Perceived latency matters more than total. Generation streams, so people watch it happen. Reranking cannot, so they just sit there in silence.

Tip: set a candidate ceiling per deployment, not globally. A 200-document handbook and a 400,000-document archive need different pools.

How We Measure Whether It Helped

Build a small labelled evaluation set from real user questions with the correct source passage marked. Fifty honest examples beat a synthetic thousand, every time. Track recall at the cutoff you actually pass to the model, plus mean reciprocal rank or NDCG to catch ordering gains that recall hides. Measure retrieval separately from generation, or a prompt change and a reranker change will take credit for each other. And watch for the failure the averages miss: a reranker that improves overall scores while systematically demoting a document type your users depend on.

When Skipping Reranking Is the Right Call

Small, clean, single-domain corpus where the retriever already returns the right passage first? You gain nothing but latency and one more moving part. Hard latency ceilings, voice interfaces especially, make the extra round trip cost more than the accuracy buys back. Fix upstream problems first - bad chunk boundaries, missing metadata filters, unparsed tables. No reranker repairs those, because the correct text was never a candidate in the first place. Your evaluation set is the honest test. If the numbers do not move, leave it out and revisit as the corpus grows.

Frequently Asked Questions

Does a bigger context window make reranking unnecessary?

No. Passing everything through dilutes the relevant signal among near-misses, raises cost per query, and weakens attribution, since you can no longer tell which passage the answer actually rested on.

Can I rerank with the same model that generates the answer?

At low volume, sure, and hosting one less component is a real benefit. At scale, the token cost and added latency usually justify a dedicated cross-encoder.

How many candidates should the retriever pass to the reranker?

Start in the tens and tune against your evaluation set. There is no universal number, only the linear relationship between candidate count and cost.

Summary: A Cheap Stage With an Unfair Reputation

Reranking is one of the few RAG improvements that needs no reindexing and no change to how documents are chunked or stored. Do it in sequence: build the evaluation set, measure current retrieval, add reranking behind a flag, compare, then tune the pool for latency. It sharpens source attribution as much as answer quality, which matters the moment users start checking where a claim came from. Teams skip it because it is invisible in demos and only shows up in the questions that quietly get answered wrong.