Which LLM API is best for low latency?

If your only goal is lowest latency, there usually isn’t one universal “best” LLM API — it depends on whether you mean:

• time to first token (TTFT)
• tokens per second (throughput)
• end-to-end response time
• global latency from your region

That said, in practice:

Often fastest in real-world usage

• OpenAI GPT-4o / GPT-4.1 mini: very fast and reliable, especially good TTFT and streaming.
• Anthropic Claude Haiku / Sonnet (fast modes): also very strong latency, especially for shorter prompts.
• Google Gemini Flash: typically one of the best choices for low-latency, high-throughput responses.
• Mistral small/medium models: often fast, especially if hosted close to your infra.

If you want the absolute lowest latency

The best choice is often not a hosted frontier API, but:

• a smaller open model
• self-hosted close to your users
• served with optimized runtimes like vLLM, TGI, or TensorRT-LLM

Examples:

• Llama 3.1/3.2 small variants
• Mistral small models
• Phi models

This can beat hosted APIs if you control deployment and infrastructure.

Practical ranking for low latency

For many users, a good default shortlist is:

1. Gemini Flash
2. GPT-4o mini
3. Claude Haiku
4. Self-hosted small model if you can optimize infra

What matters most for latency

• model size
• prompt length
• output length
• streaming support
• region proximity
• cold starts / rate limiting
• batching on provider side

Recommendation

If you want a simple answer:

• Best hosted API for low latency: Gemini Flash or GPT-4o mini
• Best possible latency overall: self-hosted small model near your users

If you want, I can give you a comparison table of OpenAI vs Anthropic vs Gemini vs Mistral specifically for TTFT, throughput, and cost.