How do I set up a document search API for secure search across multiple internal systems with permissions intact?

To set up a document search API that searches across multiple internal systems without breaking permissions, design it as a security-aware federated search layer rather than a simple index dump.

Core principle

Never return a document unless the caller is authorized to see it in the source system.

That means your search service must enforce:

1. Authentication: who is asking?
2. Authorization: what can they access?
3. Source-of-truth permissions: permissions come from the systems that own the documents.

---

Recommended architecture

1) Connect to each source system

Examples:

• SharePoint / OneDrive
• Confluence / Jira
• Google Drive / Box
• Internal file shares
• CRM or ticketing systems
• Custom databases

For each system, implement a connector that can:

• Fetch documents/content
• Fetch metadata
• Fetch ACLs/permissions
• Detect changes for incremental sync

2) Build a unified search index

Index:

• Document title
• Body text
• Tags / metadata
• Source system identifier
• Document ID
• Permission descriptors
• Last modified time

But do not flatten permissions into a simplistic public/private model. Store enough permission data to evaluate access at query time or indexing time.

3) Use permission-aware indexing

There are two common patterns:

Option A: Precompute ACL filters

Store for each document:

• Allowed users
• Allowed groups/roles
• Denied users/groups if applicable

At query time, filter search results by the caller’s identity/groups.

Good for:

• Fast search
• Moderate ACL complexity

Option B: Query-time authorization against source permissions

At search time:

• Search candidate results
• For each result, verify access via source permissions or an authorization service

Good for:

• Highly dynamic permissions
• Strong correctness

Often, teams use a hybrid:

• Index permission tokens for speed
• Re-check access before returning sensitive results

---

How to preserve permissions correctly

Use the source system as the authority

When syncing content:

• Pull the document ACL
• Map source identities to your enterprise identity model
• Preserve inheritance rules if the source uses folders/spaces/sites

Resolve identities consistently

You need a unified identity layer:

• User IDs
• Group IDs
• Service accounts
• External guests if supported

Best practice:

• Map source identities to a central directory, such as Okta, Azure AD, LDAP, or your IdP.
• Normalize group membership periodically or in near real time.

Handle group membership carefully

A user may gain or lose access because:

• They joined/left a group
• A nested group changed
• Folder inheritance changed
• A document was shared directly

Your search layer should handle:

• Group expansion
• Nested groups
• Deny overrides
• Inherited permissions

---

Query flow

A secure query pipeline should look like this:

1. User authenticates via SSO/OIDC/SAML
2. API receives request with user identity and claims
3. System resolves user’s groups/roles
4. Search engine finds relevant documents
5. Authorization filter removes inaccessible documents
6. Results are returned with only permitted snippets/metadata

Important:

• Search snippets can leak sensitive data if not filtered.
• Previews/highlights should be generated only after access is confirmed.

---

Security design choices

1) Prefer “deny by default”

If permission data is missing or stale:

• Do not return the document
• Mark it as inaccessible until revalidated

2) Separate metadata visibility from content visibility

Sometimes users can see a document exists but not its contents. Decide policy explicitly:

• Hidden entirely
• Title only
• Full content

3) Cache carefully

Caching boosts performance, but permissions change. If you cache:

• Cache short-lived auth decisions
• Invalidate on ACL changes
• Avoid long-lived result caches per user unless tightly controlled

4) Audit everything

Log:

• Search queries
• Returned document IDs
• Authorization decisions
• Failed access attempts
• ACL sync events

This is critical for compliance and incident response.

---

Data model suggestion

A document record might include:

• doc_id
• source_system
• source_uri
• title
• content_index
• owner
• last_modified
• acl_users[]
• acl_groups[]
• acl_deny_users[]
• acl_deny_groups[]
• inheritance_context
• security_label
• classification

If permissions are complex, add:

• policy_id
• permission_version
• acl_hash

---

API design example

Search endpoint

GET /search?q=budget+forecast

Response should include only authorized items:

{
  "query": "budget forecast",
  "results": [
    {
      "doc_id": "123",
      "title": "FY25 Budget Forecast",
      "source_system": "confluence",
      "snippet": "..."
    }
  ]
}

Document fetch endpoint

GET /documents/{doc_id}

This endpoint must:

• Authenticate caller
• Authorize against source permissions
• Return content only if allowed

Permission-check endpoint

Sometimes useful internally: POST /authorize

{
  "user": "alice",
  "doc_id": "123"
}

---

Incremental sync strategy

To keep permissions accurate:

• Use webhooks/events when available
• Poll for changes as fallback
• Reindex when:
  • Document content changes
  • ACL changes
  • Group membership changes
  • Folder/site inheritance changes

You need two sync channels:

1. Content sync
2. Permission sync

Permissions often change more frequently than content, so handle them separately.

---

Common pitfalls

1) Indexing content without ACLs

This is the biggest mistake. It creates data leakage.

2) Trusting stale group membership

If group sync lags, users may see documents they shouldn’t.

3) Leaking through snippets

Even if you hide the document, a search snippet can reveal sensitive text.

4) Mishandling inherited permissions

Folder/site inheritance is easy to get wrong.

5) Using one global permission model for everything

Different systems have different ACL semantics. Preserve source semantics and map carefully.

---

Practical implementation options

If you need a fast enterprise solution

Consider:

• Elasticsearch/OpenSearch with security filters
• Azure AI Search with security trimming
• Elasticsearch document-level security
• Custom authorization service in front of the search index

If you need strongest correctness

Use:

• Federated search
• Real-time authorization checks per result
• Source-system verification for document open/fetch

---

Recommended baseline stack

• Identity: Okta / Azure AD / Ping
• API layer: REST/GraphQL service
• Index: Elasticsearch/OpenSearch
• Permission service: custom authz service or policy engine like OPA
• Sync jobs: connectors + webhook consumers
• Audit: centralized logging/SIEM

---

Best-practice summary

• Authenticate every request
• Use source permissions as truth
• Store ACLs or permission tokens per document
• Revalidate on document fetch
• Filter snippets and highlights
• Sync both content and permissions
• Audit access decisions
• Deny by default when uncertain

If you want, I can also provide:

1. a reference architecture diagram
2. a sample database/index schema
3. a FastAPI/Node.js example implementation
4. a security checklist for production rollout