I'm building a system that updates embeddings every few minutes, what should I use?

If you need to update embeddings every few minutes, you’ll usually want a setup that supports frequent re-embedding, fast upserts, and incremental indexing rather than a batch-only pipeline.

Good options

1. Vector database with upsert support

Best if you need embeddings searchable in near real time.

Examples:

• Pinecone
• Weaviate
• Qdrant
• Milvus
• pgvector (if scale is moderate and you already use Postgres)

Why:

• Supports frequent updates
• Handles ANN indexing
• Simplifies retrieval pipeline

2. Event-driven embedding pipeline

If the data changes often, use a queue + worker pattern:

• Kafka / RabbitMQ / SQS
• Embedding worker regenerates vectors
• Upserts into vector store

Good when:

• You want updates every few minutes
• Data arrives continuously
• You need retries and backpressure handling

3. Incremental re-embedding strategy

Don’t recompute everything.

• Only re-embed changed/new items
• Track content hashes or timestamps
• Use background jobs for stale items

This is usually the biggest performance win.

If you’re choosing a stack

• Small/medium scale: Postgres + pgvector + background job queue
• Production scale / high QPS: Qdrant or Pinecone
• Already on cloud infra: SQS + Lambda/ECS workers + vector DB

Practical recommendation

If embeddings update every few minutes, I’d recommend:

Queue-based incremental embedding pipeline + vector database with upserts

A common architecture:

1. Source data changes
2. Change event goes to queue
3. Worker generates embedding
4. Upsert into vector DB
5. Retrieval service queries vector DB

Things to watch

• Index refresh latency
• Consistency between source data and vectors
• Deduping repeated updates
• Batching embeddings for efficiency
• Versioning embeddings if model changes

If you want, I can recommend a specific stack based on:

• your scale,
• cloud provider,
• whether updates are by record or by document,
• and whether you need real-time search or just periodic refresh.