Access Control in a Document Assistant: Who Sees Which File

A document assistant that answers from internal files inherits every permission problem the file store already has. Plus a few new ones. We build retrieval augmented generation systems, and access control is the question customers raise last and should raise first. Getting it right is architectural work, not a checkbox you tick the week before launch.

Why Retrieval Breaks the Usual Permission Model

A search box returns links. An assistant returns synthesized prose, which means a leak arrives as a fluent sentence rather than a visible file name. Nobody sees a red flag. Chunking strips folder context unless permissions travel with each fragment, and an embedding index is a flat namespace by default: every chunk equally reachable by every query. And answers get pasted into chat, forwarded, dropped into decks. One bad retrieval travels much further than one bad download ever did. So settle this before anything else: does the assistant enforce access, or inherit it?

Three Places to Enforce Access, and What Each Costs

1. Ingestion-time: one index per audience group. Easiest to reason about, smallest blast radius, and a genuine pain the day somebody changes teams and the whole thing re-indexes.
2. Query-time: single index, permission metadata per chunk, filtering inside the vector search. Modest latency cost, no re-indexing when membership changes.
3. Post-retrieval: fetch, then drop forbidden chunks. Weakest of the three, because a buggy filter has already put the text in memory.

Our default is query-time. We reserve separate indexes for the material that must never share storage: HR, legal, board papers.

Modelling Permissions as Chunk Metadata

Each chunk carries what it inherited from the source: group IDs, ACL entries, site or folder membership, sensitivity label. Store identifiers, not names. A renamed group should never quietly open a door. Deny rules have to be explicit too, because allow-list-only logic over-shares everywhere the source system leans on exceptions - and most of them do. Resolve the user's membership at query time from the identity provider. Never from a copy cached during ingestion.

Tip: treat an unlabelled chunk as private and log it, instead of letting it fall into the public tier.

Keeping Permissions Fresh After Ingestion

Permissions churn faster than documents do. Revocations, offboarding, archived projects, folders dragged into restricted spaces. Two sync patterns work here: a periodic full ACL crawl, and event-driven updates from the source system's change feed. Deletion has to reach three places, and teams routinely forget the third - the source record, the vector index, and cached answers or conversation history. Wire these events first:

• User deactivated or offboarded
• Group membership added or removed
• Document moved, its ACL edited, or sharing revoked
• Document deleted or archived

Citations, Snippets and the Leak Nobody Watches

Attaching sources is what makes the assistant trustworthy. It also creates the exact surface where a title escapes. A file name on its own discloses a client, a project code, a reorganisation nobody has announced yet. Filter citations with the same rule that filtered the chunk, and render only what the reader could open unaided. Conversation memory is the other blind spot: it inherits whatever earlier turns retrieved, so a shared thread can outlive the sharer's access.

Tip: make every citation a live link into the source system, so its own permission check becomes a second gate.

Cloud or Self-Hosted: What Actually Changes

The permission model is identical in both deployments. What differs is where the identity check happens and who keeps the logs. Self-hosted installs sit inside the existing perimeter and can reach internal directories that never face the internet. Cloud deployments need a clear answer on where embeddings live, because an embedding derives from the document and deserves equal handling. Regulated customers usually want the entire retrieval path on their own hardware, not merely the model. Neither choice removes per-chunk permissions. Network isolation guards the perimeter; it does nothing to guard colleagues from each other.

Testing That the Rules Hold

Build fixtures: test users at each tier, documents only one tier should reach. Then run these checks before go-live:

1. Query indirectly for restricted content - salary bands, an unannounced acquisition, a disciplinary case.
2. Revoke a user mid-session and confirm the next answer changes.
3. Re-run the whole suite after any chunker or connector change, because metadata drops silently.
4. Log retrieved chunk IDs per answer, so audits can reconstruct what was shown without storing answer text.

FAQ

Can the assistant just use the permissions from SharePoint or Google Drive?

Yes, when the connector reads their ACLs and keeps them synchronised. The risk sits in stale copies and in deny rules the connector does not support.

Does per-user filtering slow down retrieval?

Modern vector stores filter on metadata during the search itself. The genuine cost is recall when a filter is narrow, which you fix by retrieving more candidates.

What happens to answers already generated when access is revoked?

They persist, unless conversation history is scoped and rechecked. Treat history as retrievable content governed by its own rules.

Where We Land

Access control belongs in the design phase, before ingestion, not bolted on the week before launch. The workable default: permissions as chunk metadata, membership resolved at query time, citations filtered by the same rule. Separate indexes stay useful for the narrow set of material that should never share storage. Whatever the deployment, your assistant should be able to answer the audit question - which chunks did this person see, and why were they allowed?