On-Premise RAG: What You Give Up and What You Get Back
Nobody asks "cloud or on-premise" out of curiosity. The question shows up attached to a specific folder. Contracts, patient records, source code, case files - something with a rule bound to it. On-premise RAG means the documents, the index, the embedding step and the model inference all stay inside infrastructure the customer controls. Cloud is not the insecure option here, and on-prem is not automatically the safe one. Both are engineering choices with different failure modes, and this is a walkthrough of what actually changes on each side of that line.
What Stays Inside When You Run It Yourself
Self-hosting moves every layer that touches document text behind your own boundary. And don't skip the query log - the questions staff ask reveal contracts under negotiation, incidents in progress, deals nobody has announced yet. Components that come in-house:
- Parser and OCR stage
- Embedding model weights
- Vector index and reranker
- Generation model and orchestration layer
- Logs and evaluation data
The practical test of any on-prem claim: draw the network boundary, then check whether anything crosses it. Telemetry and model updates included.
What You Give Up
Model choice narrows first. The strongest hosted models are not distributed as weights, so you work with open-weight families and accept a real gap on hard reasoning and very long contexts. You also own the upgrade path. No silent improvements arrive overnight, and every library or model change becomes a project with testing attached. Hardware turns into a planning constraint - GPU memory decides what fits, and concurrency is bounded by what sits in the rack. Monitoring, backups, index rebuilds, driver versions, on-call: all yours. Procurement stretches the time to a first working assistant, too.
What You Get Back
You gain a clean answer to the question auditors, clients and works councils actually ask. Where does the text go, and who can read it. Contractual life gets simpler as well: no processing agreement chain, no cross-border transfer analysis, no subprocessor list to re-review every year. Cost takes a predictable shape - capital spend and known power draw rather than per-token billing that grows with adoption. Latency and availability sit under your control, air-gapped sites included. And the best part? You can index the sensitive corpus instead of a sanitized subset. That is usually where an assistant becomes genuinely useful.
Where the Split Deployment Wins
Real architectures rarely sit at either extreme. The useful question is which stage handles the sensitive text. One common shape keeps embeddings and the vector store local while a hosted model generates, with redaction limiting what reaches the prompt. The inverse works too: the restricted corpus stays fully local, and hosted models serve only public-facing content.
- Fully hosted - speed to value, no infrastructure team required
- Hosted model, private index - corpus stays put, generation borrowed
- Fully self-hosted - regulatory or contractual boundary on all text
- Air-gapped - no outbound network permitted at all
Tip: classify the corpus before choosing the architecture. One restricted folder should not drag the whole system on-prem if it can live in its own instance.
How We Decide With Customers
We start from the strictest document class in scope, not the average one, because that class sets the boundary. Then we ask who signs off - security, legal, a regulator, a client contract - and what evidence each of them accepts. Steady-state query volume matters, since the cost comparison flips somewhere between pilot traffic and organization-wide use. We check whether a team already operates GPU infrastructure, and when the answer is "we will hire for it" we treat that as a risk, not a plan. Citation requirements come up early too, because they push weight toward retrieval and permissions.
Tip: run the pilot on the deployment you intend to keep. Retrieval tuned against one model rarely transfers unchanged to another.
Frequently Asked Questions
Does an on-premise assistant give worse answers than a cloud one?
On the hardest reasoning tasks, yes - the open-weight gap is real and we say so. For grounded question answering over your own documents, though, chunking, retrieval quality and permission filtering usually dominate model choice.
What hardware do we actually need?
Sizing follows logic, not a catalogue number. Model size and quantization set GPU memory, concurrent users set throughput, and the index itself is comparatively cheap to host. Measure a pilot workload, then buy against it.
Can we start in the cloud and move on-premise later?
Yes, provided ingestion, retrieval and generation stay separate replaceable components. Migration cost sits in re-embedding the corpus and re-tuning prompts, so budget a re-evaluation pass instead of a lift and shift.
Making the Call
Read this as a trade, not a verdict: control and containment purchased with a capability ceiling and ongoing operational work. It reverses more easily in one direction than the other, so write down the assumptions that drove your choice. What holds either way is the thing that makes these systems worth building - answers grounded in your own documents with the source attached, so staff verify instead of trusting. Map the corpus, name the strictest class, and let that pick the deployment.
