How do I set up an LLM API provider for semantic search over internal documents?

To set up an LLM API provider for semantic search over internal documents, you usually build a pipeline with four parts:

1. Ingest documents
2. Create embeddings
3. Store them in a vector database
4. Retrieve relevant chunks and optionally rerank / answer with an LLM

Here’s a practical setup.

---

1) Pick an LLM/embedding provider

For semantic search, you need an embedding model more than a chat model.

Common options:

• OpenAI — strong general-purpose embeddings
• Cohere — good embeddings and reranking
• Anthropic — excellent for generation, but embeddings are typically handled via another provider
• Google Gemini / Vertex AI
• Azure OpenAI
• AWS Bedrock
• Open-source via Hugging Face / local models

If you want a straightforward managed setup, choose:

• Embeddings: OpenAI or Cohere
• Vector DB: Pinecone, Weaviate, pgvector, Milvus, or Elasticsearch
• Answering model: same provider or a chat model of your choice

---

2) Decide the architecture

A common semantic search architecture:

Offline indexing flow

• Collect internal docs: PDFs, docs, wiki pages, tickets, emails
• Extract text
• Split into chunks
• Generate embeddings for each chunk
• Store:
  • embedding vector
  • chunk text
  • metadata (doc title, URL, permissions, date, department)

Query flow

• User enters search query
• Embed query with same embedding model
• Retrieve nearest vectors from vector DB
• Optional:
  • rerank top results
  • filter by metadata and permissions
  • generate answer with an LLM using retrieved chunks

---

3) Prepare and chunk documents

You should not embed whole documents unless they’re small. Split them into chunks like:

• 200–800 tokens per chunk
• 10–20% overlap
• Keep document structure if possible

Store metadata for each chunk:

• doc_id
• chunk_id
• source
• title
• created_at
• acl/permissions
• department
• url

This helps with filtering and access control.

---

4) Create embeddings

Example with OpenAI-style embedding usage:

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY")

text = "Internal policy says all requests must be approved by finance."
response = client.embeddings.create(
    model="text-embedding-3-small",
    input=text
)

embedding = response.data[0].embedding
print(len(embedding))

Use the same embedding model for:

• indexing documents
• embedding user queries

Consistency matters.

---

5) Store vectors in a vector database

Popular choices:

Managed

• Pinecone
• Weaviate Cloud

Self-hosted / database-native

• pgvector in PostgreSQL
• Milvus
• Elasticsearch / OpenSearch vector search

A vector DB record usually contains:

• id
• embedding
• text
• metadata

Example schema conceptually:

{
  "id": "doc123_chunk4",
  "embedding": [0.12, 0.98, ...],
  "text": "The reimbursement policy requires receipts...",
  "metadata": {
    "doc_id": "doc123",
    "title": "Expense Policy",
    "department": "Finance"
  }
}

---

6) Implement retrieval

At query time:

1. Embed the query
2. Search the vector DB for top-k nearest neighbors
3. Optionally apply metadata filters
4. Optionally rerank results with an LLM or reranker model

Pseudo-flow:

query = "How do I submit travel expenses?"
query_embedding = embed(query)

results = vector_db.search(
    vector=query_embedding,
    top_k=5,
    filter={"department": "Finance"}
)

If your internal documents have permissions, apply them before returning results.

---

7) Add an LLM for answer generation, if needed

If you want search + answer, use RAG (Retrieval-Augmented Generation):

• Retrieve the best chunks
• Pass them to the LLM with a prompt like:
  • “Answer using only the provided context”
  • “Cite sources”
  • “If unsure, say you don’t know”

Example prompt structure:

Context:
1. [Policy excerpt 1]
2. [Policy excerpt 2]

Question:
How do I submit travel expenses?

Answer:

This gives users a direct answer, not just search results.

---

8) Add access control and security

For internal documents, this is critical.

Best practices:

• Store document-level permissions in metadata
• Filter retrieval by user identity/groups
• Encrypt data at rest and in transit
• Avoid sending sensitive docs to an external provider unless approved
• Consider redaction before embedding for highly sensitive content
• Log queries carefully to avoid leaking secrets

If you have strict compliance needs, consider:

• Azure OpenAI in your tenant
• AWS Bedrock
• Self-hosted embeddings and vector DB

---

9) Improve quality

Helpful enhancements:

• Hybrid search: combine keyword + vector search
• Reranking: improves top result quality
• Chunk tuning: adjust chunk size and overlap
• Metadata filters: department, date, doc type
• Synonym expansion: useful for internal jargon
• Evaluation set: test queries with known good answers

---

10) Minimal implementation stack

A simple, solid stack could be:

• Document processing: Python + unstructured / pypdf / docx parser
• Embeddings: OpenAI or Cohere
• Vector DB: pgvector or Pinecone
• Answering model: GPT-style chat model
• API layer: FastAPI
• Auth: SSO / OAuth / JWT with ACL checks

---

11) Example end-to-end flow

1. Upload internal PDF
2. Extract text
3. Split into chunks
4. Embed chunks
5. Save in vector DB with permissions
6. User searches “vacation carryover policy”
7. Embed query
8. Retrieve top relevant chunks for that user
9. Show search results or generate answer
10. Return citations

---

12) Things to watch out for

• Poor PDF text extraction
• Huge chunks that reduce retrieval precision
• Missing metadata/permissions
• Not using the same embedding model for indexing and search
• Overreliance on generation without retrieval
• Sending confidential docs to unapproved third-party APIs

---

If you want, I can also give you:

1. a specific provider recommendation based on your cloud stack,
2. a working Python example using OpenAI + pgvector,
3. or a system design diagram for semantic search over internal docs.