What should I use for a high-throughput embedding pipeline?

For a high-throughput embedding pipeline, the best choice usually depends on whether you mean generation, storage, or serving—but in practice, the stack most people land on is:

Good default choice

• Embedding model: a strong, efficient sentence-transformer / API embedding model
• Inference runtime: ONNX Runtime or TensorRT if self-hosting, or a managed embedding API if you want less ops
• Batching: essential
• Queue/streaming: Kafka, RabbitMQ, or SQS/PubSub
• Storage: FAISS, Milvus, pgvector, or Pinecone depending on scale and ops tolerance

If you want max throughput

• Self-hosted GPU inference
  • Model served with vLLM only if you’re using LLM-based embeddings; otherwise a dedicated embedding server is often better
  • For classic embedding models, use Triton Inference Server or ONNX Runtime
  • Use micro-batching and async workers
• Vector store
  • FAISS for fastest in-process indexing
  • Milvus / Qdrant for distributed search
  • Pinecone if you want managed and scalable
  • pgvector if you want simplicity and moderate scale

Common architecture

1. Ingest documents/events into a queue
2. Batch items by size or time window
3. Run embedding inference on GPU with batching
4. Write vectors + metadata to vector DB
5. Optionally keep a cache for duplicate content

Practical recommendation by use case

• Prototype / small scale: OpenAI embeddings + pgvector
• High throughput, self-hosted: sentence-transformers + ONNX/TensorRT + Milvus/Qdrant
• Fastest local search: FAISS
• Enterprise managed: Pinecone or Weaviate Cloud

Key throughput tips

• Batch requests aggressively
• Normalize and deduplicate inputs before embedding
• Use shorter chunk sizes
• Pin memory and use GPU where possible
• Separate embedding generation from indexing
• Measure token/second, not just requests/second

If you tell me:

• expected documents/sec,
• whether you need self-hosted vs managed,
• and whether this is for search or RAG,

I can recommend a specific stack.