How do I set up a RAG chatbot for searching internal docs with citations and fresh indexing?

Here’s a practical blueprint for setting up a RAG chatbot that searches internal docs, returns citations, and supports fresh indexing.

---

1) High-level architecture

A solid internal-doc RAG system usually has 5 parts:

1. Ingestion

   • Pull docs from sources like Google Drive, Confluence, SharePoint, Notion, Slack exports, PDFs, web pages, etc.
   • Convert everything to clean text + metadata.

2. Chunking

   • Split docs into searchable chunks.
   • Preserve structure like headings, page numbers, section titles, URLs.

3. Indexing

   • Embed chunks and store them in a vector database.
   • Also store metadata in a document store / search index.

4. Retrieval + generation

   • For a query, retrieve top relevant chunks.
   • Pass them to the LLM with instructions to answer only from sources.
   • Return citations pointing to exact chunks or source docs.

5. Freshness / reindexing

   • Monitor source changes and update the index incrementally.
   • Support scheduled syncs and near-real-time ingestion if needed.

---

2) Recommended stack

You can build this with many tools. A simple, reliable stack:

• Orchestration: LangChain or LlamaIndex
• Vector DB: pgvector, Pinecone, Weaviate, Qdrant, Milvus
• Primary metadata store: Postgres
• File processing: Unstructured, Apache Tika, pdfplumber
• Embeddings: OpenAI embeddings, Voyage, Cohere, or local models
• LLM: GPT-style model, Claude, or local model
• Auth: SSO / OAuth / enterprise identity provider
• App backend: FastAPI / Node.js
• Frontend: React / Next.js
• Observability: LangSmith, OpenTelemetry, custom logs

If you want the simplest production-ready path:

• Postgres + pgvector
• LlamaIndex or LangChain
• FastAPI
• Scheduled sync jobs

---

3) Document ingestion

What to store for every chunk

Keep both content and metadata.

Chunk fields:

• chunk_id
• document_id
• source_type (confluence, drive, pdf, etc.)
• source_url
• title
• section_heading
• page નંબર / paragraph / slide number if available
• chunk_text
• created_at
• updated_at
• acl or permission tags
• hash for deduplication and change detection

Why metadata matters

Metadata enables:

• citations
• filtering by access permissions
• filtering by source or date
• deduplication
• freshness updates

---

4) Chunking strategy

A good default:

• Chunk size: 300–800 tokens
• Overlap: 50–150 tokens
• Prefer splitting on:
  • headings
  • paragraphs
  • list boundaries
  • table boundaries if possible

For docs with structure:

• keep parent heading hierarchy
• include heading in each chunk
• preserve page numbers for PDFs

Example chunk text:

“# HR Benefits > Medical Coverage
Employees are eligible after 30 days...”

That makes citations and retrieval much better.

---

5) Embeddings and vector indexing

For each chunk:

1. Create embedding
2. Store vector in the vector DB
3. Store metadata alongside it

Best practice

Use hybrid search:

• Vector similarity for semantic matching
• Keyword/BM25 for exact terms, acronyms, policy codes, product names

This is especially helpful for internal docs because users often search for:

• team names
• ticket IDs
• policy numbers
• feature names
• exact phrases

---

6) Retrieval pipeline

A strong retrieval flow looks like this:

1. User asks a question
2. Normalize query
3. Optional query rewrite
4. Retrieve top chunks with:
   • vector search
   • keyword search
   • metadata filters
5. Rerank results
6. Send top sources to the LLM
7. LLM answers with citations
8. Return answer + cited chunks

Reranking

Use a reranker model if possible. It often improves relevance a lot.

Common pattern:

• retrieve top 20–50
• rerank to top 5–10
• generate answer from those

---

7) Citations: how to do them well

To get useful citations, don’t just cite “documents.” Cite specific chunks or source spans.

Good citation formats

• [1] Employee Handbook, p. 12
• [2] Confluence: Benefits Policy, section “Medical Coverage”
• [3] Drive doc: Q3 Planning, lines 45–62

How to implement

When retrieving chunks:

• keep source_url
• keep title
• keep page_number
• keep section_heading
• keep chunk_id

Then in the prompt, instruct the model:

• use only provided sources
• cite every factual claim
• if not present, say it’s not found

A typical answer prompt should say:

• “Answer using only the provided context.”
• “Include citations after each factual statement.”
• “If sources conflict, mention uncertainty.”
• “If the answer isn’t in the docs, say so.”

Very important

Make citations machine-verifiable:

• the backend should know which chunk IDs were used
• don’t rely only on the model’s text for citations
• render citations in the UI from the retrieved chunks

---

8) Fresh indexing / keeping docs up to date

This is one of the most important parts.

Option A: Scheduled batch sync

Run a job every N minutes/hours:

• list documents from source
• compare last_modified
• re-ingest changed docs
• delete removed docs
• update embeddings only for changed chunks

This is the easiest approach.

Option B: Event-driven sync

Use webhooks or event streams:

• Confluence page updated
• Drive file changed
• SharePoint doc updated

Then:

• ingest only that document
• re-chunk and re-embed
• update index immediately

Option C: Hybrid

Best for production:

• webhooks for near-real-time updates
• scheduled reconciliation job to catch missed events

Incremental indexing

Use document content hashes:

• hash full doc
• hash each chunk
• only re-embed changed chunks

This saves money and makes updates faster.

Deletions and permission changes

Don’t forget:

• if a doc is deleted, remove it from the index
• if permissions change, update ACL metadata and enforce access control at retrieval time

---

9) Access control and security

For internal docs, this is critical.

Enforce permissions at retrieval time

Do not rely only on the UI.

Store ACL metadata per doc/chunk:

• allowed users
• groups
• department
• tenant

At query time:

• filter retrieval by user identity
• only return chunks the user can access

Additional safeguards

• encrypt data at rest
• secure secrets
• audit queries
• log citations and retrieved sources
• avoid sending sensitive docs to a model if policy disallows it

If you have strict compliance requirements, you may need:

• private deployment
• VPC-hosted vector DB
• local embeddings and local LLMs

---

10) Prompting pattern for answer generation

A strong system prompt usually looks like:

• You are an assistant for internal documents.
• Only answer from the provided context.
• Cite every answer with source references.
• If the context is insufficient, say you don’t know.
• Don’t invent policy details.
• Prefer concise, direct answers.

Example output format

Answer:
Employees are eligible for medical coverage after 30 days of employment. [1]

Sources:
[1] Employee Handbook, p. 12, “Benefits” — chunk_abc123

This is easy to audit and trust.

---

11) Query flow example

User asks:

“What is our parental leave policy?”

System:

1. Search docs for “parental leave”
2. Retrieve policy chunks
3. Rerank
4. LLM generates:
   • summary of leave duration
   • eligibility conditions
   • citation links to HR policy doc

If the docs mention both “parental leave” and “family leave,” the assistant can summarize both and cite each.

---

12) Evaluation and quality checks

You should test your RAG system with a real eval set.

Evaluate:

• retrieval precision/recall
• citation correctness
• answer faithfulness
• refusal behavior when docs don’t contain answer
• permission enforcement

Build a test set

Use 50–200 real questions:

• policy questions
• how-to questions
• ambiguous questions
• questions with no answer in docs
• questions requiring recent updates

Metrics

• “Did it retrieve the right doc?”
• “Did it cite the right chunk?”
• “Was the answer supported by sources?”
• “Did it avoid hallucinating?”

---

13) Production tips

A few things that matter a lot in practice:

• Use hybrid retrieval
• Keep source metadata rich
• Reindex incrementally
• Enforce ACLs at retrieval
• Use reranking
• Make citations chunk-level
• Add a fallback for “I couldn’t find that in the docs”
• Log everything for debugging

---

14) Simple implementation outline

Ingestion job

• pull docs from source
• extract text
• chunk
• compute hash
• embed chunks
• upsert into vector DB
• store metadata in Postgres

Query service

• authenticate user
• retrieve matching chunks with ACL filter
• rerank
• build context
• call LLM
• return answer with citations

Sync job

• poll or receive webhook
• detect changed docs
• reindex changed chunks
• remove deleted docs

---

15) Minimal architecture diagram

Docs sources -> Ingestion -> Chunking -> Embeddings -> Vector DB
                                  \-> Metadata DB / ACL store

User query -> Auth -> Retrieval (hybrid + ACL + rerank) -> LLM -> Answer + citations

---

16) If you want the fastest path to a working MVP

Build this first:

• ingest one source, like Confluence
• use pgvector + Postgres
• chunk by heading/paragraph
• store source URL + page/section metadata
• hybrid search
• answer with chunk citations
• nightly sync
• basic ACL filtering

Then iterate:

• add more sources
• add webhooks
• add reranker
• add query rewriting
• add evaluation suite

---

If you want, I can also give you:

1. a reference architecture with specific tools,
2. a Python/FastAPI starter implementation, or
3. a LangChain/LlamaIndex example with citations and incremental indexing.