I'm building a production inference system and want predictable GPU capacity

To get predictable GPU capacity in a production inference system, optimize for control, isolation, and admission rather than just raw throughput.

Practical approach

1) Use a fixed-capacity serving layer

• Run a known number of replicas per GPU or per node.
• Prefer dedicated inference GPUs over shared training/serving GPUs.
• Avoid autoscaling purely on CPU or average latency; it can hide saturation until too late.

2) Put hard limits on concurrency

• Set a max batch size and max in-flight requests per model replica.
• Use request admission control:
  • reject/queue when queue depth exceeds threshold
  • enforce deadlines/timeouts
  • cap per-tenant or per-model QPS

3) Make memory usage deterministic

• Preload models at startup.
• Warm up the model and compile kernels ahead of traffic.
• Reserve headroom for:
  • KV cache
  • temporary tensors
  • framework overhead
• Avoid dynamic shapes if possible; they make latency and memory less predictable.

4) Separate workload classes

• Split:
  • large models vs small models
  • interactive latency-sensitive traffic vs batch traffic
  • different tenants or SLAs
• This prevents one workload from consuming all GPU memory or compute.

5) Use batching carefully

• Microbatching improves throughput, but too much batching increases tail latency.
• Choose:
  • fixed batch windows, or
  • bounded dynamic batching with strict queue limits
• For predictability, keep batch policy stable and observable.

6) Monitor the right signals

Track per GPU and per model:

• utilization
• memory used / reserved
• queue depth
• p50 / p95 / p99 latency
• tokens/sec or requests/sec
• OOMs, retries, timeouts
• admission drops / throttles

7) Plan capacity with headroom

• Do not size to 100% theoretical utilization.
• Leave headroom for traffic spikes, fragmentation, and kernel variance.
• A common practice is to target a steady-state utilization band rather than maxing out.

8) Prefer schedulers with explicit placement

• Use node labels / affinities / topology-aware placement.
• Pin model replicas to GPUs.
• Avoid overcommitting GPUs if predictability matters more than utilization.

---

A simple operating model

For each model:

1. Measure average and worst-case request cost.
2. Decide a fixed per-replica concurrency limit.
3. Compute max safe requests/sec per replica.
4. Add 20–30% headroom.
5. Scale replicas only when the headroom band is consistently exceeded.

---

If you want the system to feel predictable to users

Prioritize:

• bounded queueing
• strict admission control
• fixed GPU reservation
• separate pools per SLA
• constant model shape / context limits

---

If you want, I can give you:

1. a reference architecture for predictable GPU inference,
2. a capacity planning formula, or
3. a Kubernetes deployment pattern for this.