Cost and Latency in a RAG Pipeline: Where the Time Actually Goes

Every RAG demo feels fast until real documents and real users show up. Then someone asks why an answer took nine seconds, and the honest answer is usually that nobody measured the parts. We build these assistants for a living, and the same thing happens over and over: teams optimise the stage they can see instead of the one that is actually costing them. Knowing where the time and money go inside a pipeline is the difference between a system you can tune and one you can only apologise for.

The Stages a Question Passes Through

One request is not one operation. It embeds the query, searches the vector index, maybe reranks the candidates, assembles a prompt, generates an answer, then resolves citations back to source documents. Generation eats most of the wall clock in almost every deployment we have seen, because it is the only stage producing tokens one at a time. Retrieval often feels slow when the real culprit is a network hop or a cold index. Instrument each stage separately before you touch anything. A single end-to-end number hides the bottleneck instead of showing it to you.

Retrieval Is Cheap Until You Make It Clever

Plain vector search over a warm index is genuinely quick. The cost arrives with the additions. A cross-encoder reranker runs a second model over every candidate chunk. Query rewriting turns one request into two model calls before retrieval even starts.

• Hybrid search: better recall on names, codes and rare terms; costs a second index lookup and a merge step.
• Reranker: sharper ordering of candidates; costs a full model pass per chunk.
• Query rewrite: handles vague or conversational questions; costs a round trip before search starts.
• Multi-hop retrieval: answers questions spanning documents; multiplies every stage below it.

Tip: measure answer quality with and without the reranker on your own corpus. On narrow document sets it often changes nothing at all.

Context Size Is the Cost Lever Nobody Watches

Every retrieved chunk is billed as input and pushes time to first token further out. Passing twenty chunks instead of five rarely improves grounding. It does reliably raise the bill. Chunk size and overlap quietly decide how much redundant text you pay for on every single question, forever. And the evidence you retrieved is not alone in there: system prompts, tool schemas and conversation history all compete for the same budget. Prompt caching helps, sure, but only where the stable part of the prompt is genuinely stable from one request to the next.

Streaming, Perceived Speed, and What Users Feel

Time to first token matters more to people than total completion time. Retrieval happens entirely before that first token shows up, so every millisecond spent there lands directly on the person waiting. Streaming turns a long wait into a readable one. It does not fix a slow search stage. Citations resolved after generation can be attached progressively instead of blocking the answer.

Tip: show retrieval progress in the interface. Naming the documents being searched buys patience that a spinner never will.

Cloud Models Versus Your Own Infrastructure

Hosted models mean per-token pricing, no capacity planning, and latency that depends on somebody else's queue. Self-hosting means fixed hardware cost, predictable latency under known load, and owning the problem yourself when traffic spikes. Data residency and document sensitivity usually settle the argument before cost gets a word in.

1. Where are these documents legally allowed to live?
2. How many requests per day, realistically?
3. How much variable latency can the use case absorb?
4. Who operates the stack after launch?
5. What headroom is needed at peak?

Split setups are perfectly legitimate: embeddings and index on your machines, generation hosted. Or the other way round.

Indexing Cost: The Bill That Arrives Once and Then Again

Embedding a document set is a one-off expense right up until you change the embedding model, the chunking strategy or the documents. Re-embedding a large corpus is a migration, not a config change, so plan it like one. Incremental indexing on updates keeps ongoing cost proportional to churn rather than to corpus size. Storage and memory for the index scale with chunk count and vector dimension, and both of those are yours to control. Keep the source text stored next to the vectors so re-chunking never means re-scraping. Learned that one the hard way.

How We Profile a Pipeline in Practice

We log per-stage timings and token counts on every request from day one, not after the first complaint lands. Compare median against tail latency, because the tail is what users actually report. Track cost per answered question rather than cost per token. The second number flatters you; the first tells the truth. Keep a fixed set of real questions as a regression suite and run it whenever models, chunking or retrieval settings change. Then kill any stage that cannot justify its latency, and measure again.

FAQ

Does a bigger context window remove the need for retrieval?

No. It raises input cost and dilutes attention across irrelevant text. Retrieval keeps the prompt small and the citation precise, which is the whole point of the architecture.

Is a smaller generation model enough for RAG?

Often, yes, because the evidence sits in the prompt rather than in the weights. Test it against your own questions before you assume you need the biggest option on the menu.

What is the fastest single change to reduce latency?

Cut the number of retrieved chunks, then check whether answer quality moves at all. Usually it does not.

What This Means When You Build or Buy

Latency and cost are design decisions taken chunk by chunk, not tuning you sprinkle on at the end. Ask any vendor for per-stage timings and cost per answered question instead of a demo response time. Keep the pipeline simple enough that you can explain where a slow answer came from without guessing. The right architecture is the cheapest one that still answers correctly with a source attached.