Do I need a dedicated infra stack for RAG?

Not always.

Short answer

You do not need a dedicated infrastructure stack for RAG if you’re starting small or proving value. A RAG system can often run on top of your existing app stack with a few added components:

• document ingestion pipeline
• vector store or search index
• embedding model
• retrieval + reranking
• LLM inference/API

When existing infra is enough

A shared or minimal setup is usually fine if:

• you have low to moderate traffic
• your document corpus is small to medium
• freshness requirements are not strict
• you’re okay with a bit of latency
• you’re using managed services for embeddings/search/LLM

Example:
Your app backend calls a hosted LLM, stores embeddings in an existing Postgres with pgvector, and runs ingestion as a cron/job worker. That’s often enough for a pilot.

When a dedicated stack starts to make sense

You’ll likely want more specialized infra if:

• you have large-scale retrieval or many tenants
• you need low latency and high throughput
• you ingest/update content frequently
• you need strict access control / isolation
• you’re mixing multiple retrieval sources
• you need observability, evals, and rollout safety
• you want to avoid coupling RAG performance to your main app DB

At that point, dedicated components like:

• a vector database/search engine
• separate ingestion workers
• an async queue
• caching layer
• metadata/security service
• evaluation/monitoring stack

become worth it.

Practical rule of thumb

Start with:

• your existing backend
• a managed LLM
• Postgres + pgvector or a managed vector DB
• background jobs for ingestion

Move to dedicated infra only when you can point to a bottleneck such as:

• retrieval latency
• indexing delays
• costs
• operational complexity
• reliability/isolation requirements

Recommendation

If you’re asking from a product/build standpoint:
Don’t overbuild infra for RAG upfront.
Build the simplest stack that proves answer quality, then harden it once usage and requirements justify it.

If you want, I can sketch:

1. a minimal RAG architecture,
2. a production-grade RAG architecture, or
3. help you decide based on your current scale.