How do I serve an open-source LLM in production with low latency?

To serve an open-source LLM in production with low latency, optimize the whole stack: model choice, inference engine, hardware, batching, and serving architecture.

1) Pick a model that fits your latency budget

• Use a smaller model if possible: 7B–8B is much easier to serve quickly than 70B+.
• Prefer models already optimized for inference:
  • Llama 3.x, Mistral, Qwen, Gemma, Phi
• Consider:
  • Quantization: 8-bit / 4-bit can significantly reduce memory and sometimes improve throughput.
  • Context length: long contexts increase latency a lot.

2) Use a fast inference server

Good production options:

• vLLM: excellent throughput, paged attention, continuous batching
• TensorRT-LLM: very low latency on NVIDIA GPUs, more engineering effort
• TGI (Text Generation Inference): solid production server from Hugging Face
• SGLang: good for structured/agentic serving and batching
• llama.cpp: great for CPU or edge, and also GPU in some setups

If you want a default recommendation for GPU production: vLLM is usually the easiest strong starting point.

3) Optimize decoding and batching

Latency is often dominated by token generation, not just the first response.

• Use continuous batching so the GPU stays busy.
• Limit max output tokens.
• Keep prompts short.
• Use streaming responses so users see tokens early.
• Cache prompt prefixes when possible.
• If your application has repeated system prompts or templates, use prefix/prompt caching.

4) Use the right hardware

For low latency:

• NVIDIA GPUs are the standard choice.
• More VRAM lets you use larger models and longer contexts.
• For best latency:
  • A100 / H100 / L40S / A10 can work depending on model size and traffic.
• CPU-only is usually much slower unless the model is small and quantized.

5) Reduce model load and memory overhead

• Quantize weights:
  • FP16/BF16 for best quality
  • INT8 or 4-bit for lower memory and often better throughput
• Avoid overly long KV cache growth:
  • Limit context window
  • Truncate or summarize history
• Use speculative decoding if supported:
  • A smaller draft model proposes tokens; the larger model verifies them
  • Can reduce latency noticeably

6) Serve with an architecture that matches traffic

A common production setup:

• API gateway
• Request router
• Inference pool with autoscaling
• Redis / cache layer
• Observability: logs, metrics, traces

Practical tips:

• Separate interactive low-latency traffic from bulk/offline jobs.
• Use multiple replicas behind a load balancer.
• Scale on queue depth, tokens/sec, GPU utilization, or p95 latency.

7) Tune for p95, not just average latency

Watch:

• Time to first token
• Tokens/sec
• p50/p95/p99 latency
• Queue time vs generation time
• GPU memory usage
• KV cache utilization

Low average latency can still feel bad if p95 is high.

8) Common deployment patterns

Best default for most teams

• Model: 7B–14B instruct model
• Server: vLLM
• Runtime: BF16 or 8-bit/4-bit quantized
• Features: streaming, continuous batching, max output limits, caching
• Infra: 1+ NVIDIA GPUs, autoscaling, observability

If you need the lowest latency on NVIDIA

• TensorRT-LLM with a carefully optimized model and static shapes where possible

If you need edge/CPU deployment

• llama.cpp with GGUF quantized models

9) Practical latency tips

• Keep prompts under control
• Use shorter system prompts
• Reuse conversation state efficiently
• Set strict max_new_tokens
• Return partial output early
• Pre-warm model replicas
• Avoid cold starts
• Pin model weights in GPU memory

10) A simple production checklist

• Choose a model small enough for your latency target
• Use vLLM/TGI/TensorRT-LLM instead of a raw Python loop
• Enable streaming
• Quantize if acceptable
• Implement batching
• Add caching where useful
• Monitor p95 latency and GPU memory
• Autoscale replicas
• Set token limits and timeout policies

If you tell me:

1. the model size you want,
2. your target p95 latency,
3. expected QPS,
4. and whether you have GPUs or CPU-only,

I can suggest a concrete serving stack and deployment architecture.