I'm trying to connect an LLM to our company knowledge base without exposing sensitive docs. What should I use?

Use a retrieval-augmented generation (RAG) setup with access control at retrieval time, not by dumping the whole knowledge base into the model.

Recommended approach

1. Keep documents in your own secure store

   • SharePoint, Confluence, S3, database, file system, etc.
   • Do not train/fine-tune the LLM on sensitive docs unless you have a very specific, audited reason.

2. Index only what’s needed

   • Chunk documents
   • Create embeddings
   • Store them in a vector database or search index

3. Enforce permissions before retrieval

   • Filter search results by user identity, group, department, clearance, document ACLs
   • Only pass the allowed snippets to the model

4. Send only retrieved passages to the LLM

   • The model sees just the relevant excerpts, not the full knowledge base

Good components to use

• LLM: hosted API or self-hosted model
• Embedding model: for semantic search
• Vector store / search layer:
  • Pinecone, Weaviate, Milvus, pgvector, Elasticsearch/OpenSearch, Azure AI Search
• AuthN/AuthZ:
  • SSO/OIDC/SAML + role/group-based filtering
• Document pipeline:
  • chunking, metadata tagging, ACL propagation, redaction

Security best practices

• Propagate document-level and chunk-level ACLs
• Encrypt data at rest and in transit
• Log retrievals and prompts for audit
• Redact secrets/PII before indexing if possible
• Prefer a zero-trust design: the LLM should never directly query raw docs
• If data is highly sensitive, consider a self-hosted model in your VPC/on-prem

Simple rule of thumb

• Need secure Q&A over internal docs? Use RAG with permission-aware retrieval
• Need model customization? Fine-tune on non-sensitive patterns only, not confidential source material
• Need strongest isolation? Use a self-hosted LLM plus private search/indexing

If you want, I can suggest a concrete architecture for:

• Microsoft stack,
• AWS stack,
• Google Cloud stack, or
• fully on-prem/private deployment.