I'm building a chatbot for sales reps that needs CRM context and policy docs. What RAG stack would you recommend?

For a sales-rep chatbot with CRM context + policy docs, I’d recommend a hybrid RAG stack built around three things:

1. Structured CRM retrieval
2. Semantic + keyword retrieval over docs
3. Strict permissioning / source grounding

Recommended stack

1) Data sources

• CRM: Salesforce / HubSpot / Dynamics via API
• Policy docs: SharePoint, Google Drive, Confluence, Notion, PDF/docs
• Optional: email, call notes, support tickets

2) Ingestion + normalization

Use an ETL layer to convert everything into a common schema:

• Document chunks for policy content
• Entity records for CRM objects:
  • account
  • contact
  • opportunity
  • activity
  • product / pricing / entitlement

Good tools:

• LlamaIndex or LangChain for connectors + orchestration
• Unstructured for parsing PDFs/Office docs
• A small custom ETL for CRM objects

3) Storage / indexing

Use two indexes:

A. Vector index for semantic search

Best for policy docs and long-form content.

Options:

• Pinecone if you want managed and easy scaling
• Weaviate if you want flexible hybrid search
• pgvector if you want simpler infra and moderate scale
• Qdrant if you want strong open-source vector search

B. Keyword / lexical index for exact-match retrieval

Important for:

• policy clause lookup
• product names
• SKUs
• legal wording
• customer names, account IDs

Options:

• Elasticsearch / OpenSearch
• Or a hybrid vector DB like Weaviate

4) Hybrid retrieval strategy

For this use case, I would not use pure vector search.

Use:

• BM25 / keyword retrieval
• semantic vector retrieval
• metadata filters
• re-ranking

Best practice flow:

1. classify query intent
2. retrieve from CRM records + docs
3. merge results
4. rerank top candidates
5. generate answer with citations

5) Re-ranker

Very useful for improving answer quality.

Options:

• Cohere Rerank
• bge-reranker
• Voyage rerank
• A cross-encoder hosted internally

6) LLM layer

Use a strong instruction-following model that supports citations and tool use.

Examples:

• OpenAI GPT-4.1 / GPT-4o
• Claude
• Gemini
• Or a self-hosted model if required

7) Permissions and policy enforcement

This is critical for CRM + internal docs.

Add:

• row-level / document-level ACL filtering
• field masking for sensitive CRM fields
• audit logging
• tenant/user-based authorization before retrieval

8) Answer grounding

Force the model to:

• cite CRM fields and doc passages
• say “I don’t know” if evidence is missing
• separate factual data from recommendations

My preferred architecture

If I were building this today, I’d do:

• LlamaIndex for orchestration
• Postgres + pgvector or Qdrant for vector storage
• OpenSearch for keyword search if doc volume is large
• Cohere Rerank or bge-reranker
• OpenAI or Claude for generation
• Custom CRM ingestion layer with permission filters

If you want a simpler MVP

Use:

• LlamaIndex
• Qdrant
• Postgres
• OpenAI
• Unstructured
• Cohere Rerank

That’s a solid, practical starting point.

Design tips specific to sales reps

Sales questions often fall into these buckets:

• “What is the latest status of Acme’s deal?”
• “Can I offer a discount on this SKU?”
• “What’s our policy on legal redlines?”
• “Who owns this account?”
• “What did we promise the customer last week?”

So I’d build intent-aware retrieval:

• CRM queries go to structured tools/APIs
• policy questions go to document retrieval
• mixed queries use both and combine evidence

Best default recommendation

If you want one concrete stack:

LlamaIndex + Qdrant + OpenSearch + Cohere Rerank + OpenAI/Claude + custom CRM ACL filtering

That gives you:

• strong retrieval quality
• good scalability
• permission control
• easy expansion later

If you want, I can also give you:

1. a reference architecture diagram in text
2. a stack recommendation by budget
3. a minimal MVP vs production architecture