I'm building an AI product with unpredictable traffic, how should I size GPUs?

For unpredictable AI traffic, the safest approach is usually not to size GPUs to peak load. Instead, size for a baseline, then add elastic capacity and a queueing/fallback strategy for spikes.

A practical way to size GPUs

1) Measure the workload first

You need these inputs:

• Requests per second or jobs per hour
• Average and P95 latency target
• Model type and size
• Tokens per request for LLMs, or input resolution/batch size for vision, etc.
• GPU throughput for your exact model stack:
  • tokens/sec/GPU
  • requests/sec/GPU
  • images/sec/GPU
• Memory footprint
  • model weights
  • KV cache / activations
  • batch size overhead

If you don’t know throughput yet, benchmark it on the GPU model you plan to use.

---

2) Convert traffic into required GPU capacity

A simple starting formula:

Required GPUs ≈ peak sustained load / throughput per GPU

Example for an LLM:

• Peak load: 1,000 tokens/sec
• One GPU serves: 200 tokens/sec
• Raw need: 5 GPUs

Then add headroom:

• 20–30% for normal variability
• More if traffic is bursty or models are large

So you might provision:

• 6–7 GPUs baseline
• Then handle spikes with autoscaling, queueing, or overflow to a secondary pool

---

3) Plan for burstiness, not just averages

If traffic is unpredictable, use one of these patterns:

Option A: Baseline + autoscale

• Keep a small always-on pool
• Scale up additional GPUs when queue depth, latency, or utilization crosses thresholds
• Best if your platform can start GPUs quickly enough

Option B: Queue and smooth

• Accept requests quickly
• Queue excess work
• Great when some latency is acceptable
• Lets you run GPUs hotter and with higher utilization

Option C: Tiered service

• Route priority traffic to reserved GPUs
• Send low-priority traffic to cheaper/spot/preemptible GPUs
• Good if you have premium and best-effort workloads

Option D: Multi-model routing

• Use a smaller/cheaper model for most traffic
• Reserve bigger GPUs only for hard cases or premium users

---

4) Don’t size to 100% utilization

In practice, GPUs perform better with some slack:

• 70–85% average utilization is often healthier than running at 95–100%
• Spikes, batching inefficiency, and memory fragmentation make “fully utilized” systems brittle
• Leave room for:
  • batching
  • retries
  • model reloads
  • traffic bursts

---

5) Separate compute sizing from memory sizing

A GPU can be “fast enough” but still unusable if it runs out of VRAM.

For LLM serving, size for:

• Model weights
• KV cache growth
• Concurrent requests / max context length
• Batching strategy

Sometimes you need fewer, larger GPUs not for compute, but for memory.

---

6) Use the right scaling signal

Avoid scaling only on CPU or average GPU utilization. Better signals:

• Queue length
• P95/P99 latency
• Time waiting in queue
• Tokens/sec backlog
• OOM/error rate
• GPU memory utilization

For LLMs, queue depth and latency usually work better than raw GPU utilization.

---

7) Add cost controls

Unpredictable traffic can get expensive fast. Use:

• hard caps on max GPU count
• rate limiting
• request budgets
• model fallbacks
• spot/preemptible GPUs for non-critical jobs
• scheduled scaling if you know daily patterns

---

Rule of thumb

If you’re early-stage and don’t know exact throughput yet:

1. Benchmark your model on 1 GPU
2. Estimate peak sustained demand
3. Provision enough GPUs for about 70–80% of expected peak
4. Add:
   • autoscaling
   • a queue
   • overflow/fallback model or provider

That usually gives a good balance of cost and reliability.

If you want, I can help you build a sizing worksheet

If you share:

• model name
• request type
• average tokens per request
• target latency
• expected peak RPS
• GPU type you’re considering

…I can estimate how many GPUs you need and suggest an autoscaling strategy.