How do I set up retrieval infrastructure for context retrieval in an agent workflow?

To set up retrieval infrastructure for context retrieval in an agent workflow, think of it as building a pipeline that can store, search, rank, and deliver the right context at the right time.

1) Define what “context” means

Start by deciding what your agent needs to retrieve:

• Documents: PDFs, docs, wiki pages, tickets, emails
• Structured records: CRM rows, DB tables, logs, APIs
• Conversation memory: prior chats, user preferences, session state
• Task artifacts: plans, tool outputs, intermediate results

This determines the storage and retrieval strategy.

---

2) Build a content ingestion pipeline

You need a repeatable way to get data into your retrieval system.

Typical ingestion steps:

1. Collect sources
   • file systems
   • databases
   • APIs
   • knowledge bases
2. Normalize content
   • extract text from PDFs/HTML/docs
   • clean boilerplate
   • preserve metadata
3. Chunk content
   • split into retrieval-sized segments
   • keep semantic boundaries when possible
4. Enrich metadata
   • source, title, timestamp, author, permissions, tags, tenant/user IDs
5. Embed and index
   • generate embeddings for semantic search
   • store in a vector index and/or search index

---

3) Choose your storage layers

A robust retrieval architecture usually uses more than one store:

A. Raw content store

Store original documents in object storage or a document store:

• S3 / GCS / Azure Blob
• MongoDB / Postgres / Elasticsearch source documents

Purpose:

• auditability
• reprocessing
• versioning

B. Metadata store

Use a relational or document DB for:

• document IDs
• source info
• ACL/permissions
• timestamps
• chunk mapping
• lifecycle state

Examples:

• Postgres
• MySQL
• DynamoDB

C. Search index

Use one or both:

• Vector database for semantic retrieval
  • Pinecone, Weaviate, Milvus, Qdrant, pgvector
• Keyword / lexical index for exact matching
  • Elasticsearch, OpenSearch, Lucene

Best practice: hybrid retrieval = vector + keyword.

---

4) Implement chunking carefully

Bad chunking is one of the biggest causes of poor retrieval.

Good practices:

• Chunk by semantic units: headings, paragraphs, sections
• Use overlap to preserve context
• Keep chunks reasonably small:
  • often 200–800 tokens depending on content
• Store chunk boundaries and parent document references
• For structured data, consider row-based or field-based chunks instead of text chunking

Also consider:

• hierarchical chunking:
  • document summary
  • section chunks
  • paragraph chunks

This helps retrieve both coarse and fine context.

---

5) Create an indexing strategy

For each chunk or record, store:

• id
• document_id
• chunk_text
• embedding
• metadata
• tenant_id
• access_control_list
• version
• created_at, updated_at

Indexing flow:

1. extract text
2. chunk
3. generate embeddings
4. write to vector store
5. write metadata to DB
6. optionally write keyword terms to search engine

If you expect updates, support:

• incremental reindexing
• version control
• soft deletes
• re-embedding when model changes

---

6) Build the retrieval pipeline used by the agent

The agent should not query the vector DB directly in an ad hoc way. Create a retrieval service or retrieval layer.

Typical retrieval stages:

Step 1: Query understanding

• detect intent
• expand query with entities or synonyms
• rewrite the query if needed

Step 2: Candidate retrieval

• semantic search in vector DB
• keyword search in text index
• optionally structured lookup in DB/API

Step 3: Merge results

• combine vector and lexical candidates
• deduplicate by document/chunk

Step 4: Rerank

• use a cross-encoder / reranker / LLM-based ranking
• rank by relevance to the task, not just similarity

Step 5: Filter

• permissions
• freshness
• source reliability
• tenant/user scope

Step 6: Pack context

• assemble top chunks within token budget
• include citations/metadata
• preserve provenance

---

7) Add memory tiers for agents

Agents usually benefit from multiple memory types:

Short-term memory

• current conversation state
• current task plan
• intermediate tool results

Long-term memory

• past sessions
• user preferences
• stable facts

Working memory / scratchpad

• ephemeral reasoning state
• not usually persisted permanently

A common architecture:

• store conversation turns in a DB
• summarize older turns
• index summaries and key facts into retrieval store
• retrieve memories selectively based on current task

---

8) Handle permissions and isolation

Retrieval infrastructure must respect access control.

At retrieval time:

• filter by user/org/project/tenant
• enforce ACLs before returning content
• avoid embedding or indexing sensitive data unless policy allows it
• redact secrets/PII where needed

If multi-tenant:

• separate indexes per tenant, or
• shared index with strict metadata filters

---

9) Add freshness and update handling

Context retrieval fails if data is stale.

You need:

• event-driven reindexing on content changes
• sync jobs for APIs/databases
• document versioning
• TTL for ephemeral data
• recency boosting in ranking

For agents, freshness often matters more than static relevance.

---

10) Build evaluation and observability

You should measure retrieval quality continuously.

Track:

• recall@k
• precision@k
• MRR / nDCG
• answer success rate
• latency
• context utilization
• stale-result rate

Log:

• query
• retrieved chunks
• ranks/scores
• final answer
• user feedback
• tool outcomes

This helps you improve chunking, embeddings, ranking, and filters.

---

11) Recommended reference architecture

A practical setup looks like this:

1. Ingestion service
   • pulls from docs/DB/APIs
2. Processing pipeline
   • parse → clean → chunk → embed
3. Metadata DB
   • stores document/chunk info + ACLs
4. Vector store
   • semantic retrieval
5. Keyword search engine
   • exact/fuzzy retrieval
6. Reranking service
   • improves top-k relevance
7. Retrieval API
   • single interface used by the agent
8. Agent orchestration layer
   • calls retrieval when needed
9. Logging/metrics
   • evaluation and debugging

---

12) A simple agent retrieval loop

A common pattern:

1. Agent receives task
2. Agent formulates retrieval query
3. Retrieval service returns top context
4. Agent reads context and decides whether more retrieval is needed
5. Agent calls tools / acts
6. Agent stores useful outcomes back into memory

Pseudo-flow:

user task -> query rewrite -> retrieve candidates -> rerank
-> filter by permissions/freshness -> pack context -> agent response/action

---

13) Practical implementation choices

If you want a straightforward stack:

• Postgres for metadata and memory
• pgvector for embeddings
• Elasticsearch/OpenSearch for keyword search
• S3 for raw documents
• Celery / Kafka / Airflow for ingestion jobs
• Reranker as a service or model endpoint
• FastAPI for retrieval API

This is a strong starting point for many teams.

---

14) Common pitfalls

Avoid these:

• indexing huge chunks that mix unrelated topics
• forgetting metadata/ACL filters
• using only vector search
• no reranking
• no update/reindex strategy
• no observability
• packing too much context into the prompt
• retrieving irrelevant but semantically similar content

---

15) Minimal version to start with

If you want an MVP:

• raw docs in object storage
• chunk text
• embed chunks
• store embeddings in a vector DB
• store metadata in Postgres
• add a simple reranker later
• expose retrieval through one API endpoint used by the agent

---

If you want, I can also give you:

1. a concrete architecture diagram,
2. a sample Postgres + pgvector schema, or
3. a Python retrieval service example for an agent workflow.