I'm building an internal search tool for a company with SharePoint, Confluence, Slack, Jira, and Drive. What architecture should I look at if permissions and citations matter?

If permissions and citations matter, the architecture to look at is a permission-aware hybrid retrieval + RAG system with source-of-truth indexing, not a naive “dump everything into one vector DB” setup.

Recommended architecture

1) Connectors + incremental sync

Pull content from SharePoint, Confluence, Slack, Jira, and Drive using API connectors.

Store:

• raw documents/messages/tickets
• metadata: source, URL, author, timestamps, workspace/project, thread, etc.
• ACLs / sharing permissions at item level, and ideally folder/project/channel inheritance

You want incremental sync so the index stays fresh.

---

2) Permission model layer

Build a canonical permission graph or access-control index.

At query time:

• determine the requesting user’s identity/groups
• resolve effective permissions across all systems
• filter retrieval candidates before the LLM sees them

This is critical. Permissions should be enforced:

• before retrieval
• again before generation
• and ideally in auditing/logging

A common pattern is:

• document-level ACLs stored with each chunk
• a permission service computes allowed doc IDs for a user
• search only over allowed items

---

3) Dual index: lexical + semantic

Use both:

• keyword search index for exact matches, names, ticket IDs, policy terms
• vector index for semantic retrieval

This helps with enterprise search a lot because users often search for:

• “Q3 headcount planning”
• “VPN issue from last week”
• “JIRA-1234”
• “the spreadsheet about pricing”

Hybrid retrieval is usually better than pure vector search.

---

4) Chunking + source preservation

Chunk documents carefully and preserve source references.

Each chunk should keep:

• source system
• original document/message URL
• title
• timestamps
• ACLs
• parent document ID
• thread/ticket/page context

This is what allows citations that users can actually click.

For Slack and Jira, citations may need to reference:

• message permalinks
• thread root + reply message
• issue URL For SharePoint/Drive/Confluence:
• page/file URL
• section heading or anchor if possible

---

5) Retrieval pipeline

A solid flow is:

1. User query comes in
2. Resolve user identity + groups
3. Apply permission filter
4. Run hybrid retrieval:
   • BM25 / keyword
   • dense vector search
5. Merge and rerank results
6. Select top context snippets
7. Generate answer with citations
8. Return answer + cited source links

For best quality, add a reranker after retrieval.

---

6) Answer generation with grounded citations

Use an LLM only on retrieved, permitted context.

Require the model to:

• answer only from retrieved evidence
• cite each claim
• say “I couldn’t find evidence” when applicable

A good pattern is to generate:

• concise answer
• bullet list of supporting citations
• confidence / provenance metadata

---

7) Auditability and observability

Since this is internal and permission-sensitive, log:

• who searched
• what was retrieved
• what was shown
• what sources were cited

This helps with:

• compliance
• debugging permission issues
• measuring retrieval quality

---

Important design choice

If you care about both permissions and citations, store permissions alongside chunks and enforce them in retrieval, rather than relying on the LLM to “not leak” information.

The safest pattern is:

• per-chunk ACL metadata
• permission-aware retrieval filter
• citations tied to original source URLs
• no unfiltered cross-user global embedding search

---

Practical architecture summary

A good reference architecture looks like:

• Ingestion layer: connectors for SharePoint/Confluence/Slack/Jira/Drive
• Normalization layer: unify content + metadata + ACLs
• Indexing layer:
  • lexical index
  • vector index
  • metadata/ACL index
• Permission service: computes effective user access
• Retrieval service: hybrid search over only allowed content
• Reranker: improves relevance
• Generation service: LLM answers with citations
• Audit/logging: traceability

---

If you want the shortest answer

Look for a permission-aware hybrid RAG search architecture with:

• source connectors
• ACL propagation
• hybrid lexical + vector retrieval
• pre-retrieval permission filtering
• citation-preserving chunking
• grounded answer generation

If you want, I can sketch a reference architecture diagram or suggest a technology stack for this setup.