How do I connect an LLM to our internal docs and databases so it can answer questions from our data?

To connect an LLM to your internal docs and databases, you typically build an RAG system — Retrieval-Augmented Generation — plus a tool/query layer for structured data.

1) High-level architecture

User question → router/orchestrator → retrieval/tools → LLM → answer

Common components:

• Documents layer

  • Ingest PDFs, wiki pages, tickets, spreadsheets, etc.
  • Chunk text, create embeddings, store in a vector database.
  • At query time, retrieve the most relevant chunks.

• Database layer

  • Expose safe, read-only access to SQL/BI systems.
  • Let the model generate or select queries through a controlled tool.
  • Retrieve rows/aggregations and feed results back to the LLM.

• LLM

  • Synthesizes the final answer from retrieved context and/or query results.
  • Should be instructed to answer only from provided sources when needed.

---

2) For internal documents: use RAG

Ingestion flow

1. Collect sources: SharePoint, Confluence, Google Drive, Notion, file shares, etc.
2. Parse and clean text.
3. Split into chunks:
   • usually 300–1,000 tokens
   • keep metadata like title, URL, department, timestamp, ACLs
4. Create embeddings for each chunk.
5. Store in a vector DB or search index:
   • Pinecone, Weaviate, Milvus, pgvector, Elasticsearch/OpenSearch, etc.

Query flow

1. User asks a question.
2. Embed the question.
3. Retrieve top-k relevant chunks.
4. Optionally rerank with a cross-encoder or LLM reranker.
5. Send the best chunks to the LLM with the question.
6. Return answer with citations.

Best practices

• Preserve document metadata and permissions.
• Use hybrid search: keyword + vector.
• Add reranking for better precision.
• Cite sources in the response.
• Refresh embeddings when docs change.

---

3) For databases: use controlled SQL/tool access

For structured data, don’t dump the whole database into embeddings. Instead:

Pattern A: Text-to-SQL

• LLM converts the user question into SQL.
• A query executor runs the SQL against a read-only replica or warehouse.
• Results are passed back to the LLM for interpretation.

Pattern B: Tool/function calling

• Define tools like:
  • get_customer_metrics(start_date, end_date)
  • run_sql(query)
  • lookup_order(order_id)
• The LLM chooses which tool to call.
• Your backend validates and executes safely.

Safety controls

• Read-only credentials.
• SQL allowlists or query builders.
• Row and column-level security.
• Query limits and timeouts.
• Human approval for risky queries.
• Logging and audit trails.

---

4) Recommended system design

A robust setup usually looks like this:

• Document retriever for unstructured sources
• SQL/tool layer for structured sources
• Orchestrator to decide which source to use
• LLM to summarize and explain results
• Permissions layer to enforce access control

Example routing:

• “What does the policy say about parental leave?” → docs RAG
• “What were Q3 sales by region?” → SQL tool
• “Why did support tickets increase last month?” → both docs + analytics

---

5) Security and governance

This is critical for internal data.

• Enforce document-level and row-level permissions
• Never expose secrets, credentials, or raw PII unnecessarily
• Mask sensitive fields
• Keep audit logs of prompts, retrieved sources, and outputs
• Add prompt-injection defenses for untrusted docs
• Use a secure network path to your data sources
• Consider tenant isolation if multiple teams use the system

---

6) Evaluation

Before production, test:

• Retrieval quality: Are the right docs being found?
• Answer correctness: Are responses grounded in data?
• Access control: Can users only see what they’re allowed to see?
• Latency: Is the system fast enough?
• Hallucinations: Does it refuse when evidence is insufficient?

Useful metrics:

• recall@k / precision@k for retrieval
• answer faithfulness
• citation accuracy
• SQL execution success rate

---

7) Practical implementation stack

A common stack:

• LLM: OpenAI, Anthropic, Azure OpenAI, local model
• Orchestration: LangChain, LlamaIndex, Haystack, or custom
• Vector store: pgvector, Pinecone, Weaviate, Milvus, OpenSearch
• DB access: SQLAlchemy, direct warehouse connectors, BI semantic layer
• Auth: SSO/OAuth, RBAC/ABAC
• Observability: tracing/logging/evals

---

8) Simple implementation approach

If you want the fastest path:

1. Start with one document source and one database.
2. Build RAG for docs.
3. Add a read-only SQL tool for a few approved tables.
4. Add citations and source links.
5. Add access control and logging.
6. Evaluate with real user questions.
7. Expand sources gradually.

---

9) Example interaction

User: “What is our refund policy for annual plans, and how many refunds did we issue last month?”

System:

• Retrieves policy docs from the vector index
• Runs SQL: refunds by month
• LLM merges the results into one answer with citations

---

If you want, I can sketch a reference architecture, provide a sample Python implementation with RAG + SQL tools, or suggest a stack for your existing environment.