How do I set up an internal answer engine for permission-aware document retrieval in a large enterprise?

To set up an internal answer engine for permission-aware document retrieval in a large enterprise, design it as a retrieval + access-control + answer-generation pipeline rather than a plain chatbot. The key requirement is: the model must only ever see documents the current user is allowed to access.

1) Define the core architecture

A practical enterprise pattern is:

1. Connect to document sources

   • SharePoint, Google Drive, Confluence, Slack, email, Box, OneDrive, internal wikis, file shares, ticketing systems, databases, etc.

2. Ingest and normalize content

   • Extract text, metadata, ownership, timestamps, classification labels, ACLs, groups, and source links.
   • Chunk documents into retrievable passages.

3. Index for retrieval

   • Use a hybrid index:
     • Keyword search for exact matches
     • Vector search for semantic matches
   • Store metadata for each chunk, especially security metadata.

4. Enforce permissions at retrieval time

   • When a user asks a question, resolve their identity and group memberships.
   • Filter candidate chunks by ACLs, document classification, and source-specific permissions before retrieval or before final ranking.

5. Generate an answer from allowed evidence only

   • Pass only permitted passages to the LLM.
   • Require citations back to source documents.

6. Log, audit, and monitor

   • Record query, user identity, retrieved docs, permission checks, answer, and citations for compliance.

---

2) Make permissions a first-class part of the data model

Each indexed chunk should carry:

• doc_id
• chunk_id
• source_system
• owner
• classification
  • e.g. Public, Internal, Confidential, Restricted
• acl_users
• acl_groups
• tenant / business_unit / region
• source_path
• effective_date
• retention_policy
• sensitivity_tags

For enterprise use, permissions usually come from:

• Document-level ACLs
• Folder-level inheritance
• Group memberships from IdP/HR system
• Attribute-based access control (ABAC) like region, business unit, clearance, project membership

Use a policy engine if possible:

• OPA (Open Policy Agent)
• Cedar
• A custom entitlement service

---

3) Choose a retrieval strategy that is permission-safe

There are three common patterns:

A. Pre-filter retrieval

Filter the index by user permissions first, then search only allowed docs.

Pros

• Strongest safety
• Simple mental model

Cons

• Harder if permissions are complex or dynamic
• May reduce recall if filtering happens too early

B. Post-filter retrieval

Retrieve candidates broadly, then filter out unauthorized results.

Pros

• Better recall

Cons

• Risky if unauthorized content leaks into ranking, reranking, or logs
• Must ensure the LLM never sees unauthorized chunks

C. Security-aware retrieval with ACL filtering

Best for most enterprises:

• Index includes ACL metadata
• Search layer supports metadata filters
• Query is executed only on docs the user can access
• Final rerank also respects permissions

This is usually the best balance.

---

4) Recommended high-level flow

1. User authenticates via SSO / IdP
2. System gets:
   • user ID
   • groups
   • roles
   • attributes
3. Query is expanded if needed
4. Retrieval service computes allowed scope:
   • user entitlements
   • document classification constraints
   • legal/compliance rules
5. Search returns only permitted chunks
6. Reranker scores allowed chunks
7. LLM answers using only those chunks
8. Answer includes citations
9. Audit log is stored

---

5) Indexing pipeline design

Ingestion steps

• Crawl source systems incrementally
• Pull content and metadata
• Extract text from PDFs, Office files, HTML, images via OCR if needed
• Normalize to a common schema
• Chunk content by semantic structure:
  • headings
  • paragraphs
  • tables
  • Q&A sections

Enrichment

• Add embeddings
• Add entity tags
• Detect language
• Classify sensitivity
• Attach ACLs and inheritance info

Important

Store ACLs at the chunk level, not just document level, if sections differ in access.

---

6) Identity and access control integration

You need a strong identity backbone:

• SSO: Okta, Entra ID, Ping, etc.
• SCIM / directory sync for groups and roles
• HR system for employment status, org, manager, location
• Entitlement service to resolve effective permissions
• Service-to-service auth between retrieval, policy, and LLM layers

Support:

• Role-based access control (RBAC)
• Attribute-based access control (ABAC)
• Just-in-time permissions if needed
• Time-limited access for special projects

---

7) Prevent data leakage to the LLM

This is critical.

Do:

• Send only approved snippets
• Minimize context
• Redact secrets, PII, credentials, and regulated data if not needed
• Use prompt templates that instruct the model to answer only from provided evidence
• Include source citations

Don’t:

• Send raw corpora
• Send unauthorized retrieval candidates to the model
• Log sensitive context without controls
• Rely on the LLM to “self-censor” permissions

Remember: permission enforcement must happen before generation, not just in the prompt.

---

8) Handle source-specific permission models

Different systems expose permissions differently:

• SharePoint: inherited ACLs
• Google Drive: file sharing and groups
• Confluence: space/page restrictions
• Slack: channel membership and message visibility
• File systems: POSIX/NTFS ACLs
• Databases: row-level or column-level security

Create a common entitlement normalization layer that converts each source’s ACL model into a unified representation.

---

9) Add a policy engine

For large enterprises, hardcoding permissions in application code becomes unmanageable.

A policy engine can express rules like:

• Only Finance can access finance docs
• Only users in Project X can access project docs
• EMEA users may not access certain US-only content
• Contractors cannot access restricted legal content
• Executives have broader but still auditable access

Good pattern:

• Retrieval service asks policy engine: “Can user U access chunk C?”
• Policy engine returns allow/deny
• Cache results carefully with invalidation on membership changes

---

10) Use hybrid retrieval + reranking

A strong enterprise retrieval stack:

• BM25 / keyword search
• Dense vector search
• Metadata filtering
• Cross-encoder or LLM reranker on allowed candidates
• Citation selection

This improves answer quality while respecting security.

---

11) Build a citation-backed answering layer

Your answer engine should:

• Quote or paraphrase only permitted evidence
• Link back to source docs
• Show confidence or coverage
• Say “I couldn’t find permissioned evidence” when appropriate

This reduces hallucination and helps users verify answers.

---

12) Logging, audit, and governance

Enterprise deployments need:

• Query logs
• Retrieved chunk IDs
• Policy decisions
• Model version
• Prompt template version
• User identity
• Citation trail
• DLP events
• Access anomalies

Use these for:

• Compliance audits
• Security investigations
• Retrieval quality tuning
• Permission debugging

---

13) Security hardening checklist

• Encrypt data at rest and in transit
• Isolate tenant/business unit data
• Use secret management
• Redact sensitive values in logs
• Rate limit and detect prompt injection
• Treat documents as untrusted input
• Sanitize content before prompting
• Validate citations
• Use least-privilege service accounts
• Review prompt injection defenses for retrieved content

---

14) Protect against prompt injection in documents

Documents can contain malicious instructions like:

• “Ignore previous instructions”
• “Reveal hidden data”

Mitigations:

• Separate system prompts from retrieved content
• Strip or label suspicious instruction-like text
• Use document trust scoring
• Never let retrieved text override system policy
• Constrain the model to extraction and synthesis only

---

15) Recommended implementation pattern

A solid stack might look like:

• IdP: Entra ID / Okta
• Policy engine: OPA
• Ingestion: Airflow / Dagster / custom connectors
• Search: Elasticsearch/OpenSearch + vector DB, or a unified engine
• Metadata store: Postgres / graph DB
• Embeddings: enterprise embedding model
• Reranker: cross-encoder or LLM reranker
• LLM: private hosted model or secure API with strict controls
• App layer: retrieval orchestration service
• Audit: SIEM / data platform

---

16) Start with a minimal safe design

If you want a first production version, implement this:

1. SSO login
2. Sync user groups
3. Index docs with ACL metadata
4. Filter retrieval by ACL before LLM
5. Use hybrid search
6. Return answers with citations
7. Log all access
8. Add policy engine and ABAC later

---

17) Common failure modes

• Indexing documents without preserving ACLs
• Expanding access too broadly through group inheritance mistakes
• Letting reranker or LLM see unauthorized candidates
• Stale group membership caches
• Missing folder inheritance logic
• Over-sharing in logs or telemetry
• Ignoring prompt injection
• Not handling source-system permission nuances

---

18) A simple architecture summary

User → SSO/IdP → Entitlement resolver → Permission-filtered retrieval → Reranker → LLM answer generation → Citations + audit

The golden rule:

Only authorized content should be retrievable, rerankable, and visible to the model.

If you want, I can also provide:

1. a reference architecture diagram,
2. a sample data schema for ACL-aware chunks,
3. or a production-ready system design for a specific stack like OpenSearch + OPA + Postgres + Azure OpenAI / Claude / Llama.