GPU orchestration for inference

GPU orchestration for inference is the process of deciding which model runs on which GPU, when, and with what memory/compute share so you can maximize throughput, minimize latency, and use hardware efficiently.

Core goals

• Low latency for real-time requests
• High throughput for batch or streaming inference
• High utilization of GPU memory and compute
• Fairness / isolation across models, tenants, or workloads
• Reliability under spikes, failures, and model restarts

Main orchestration patterns

1. One model per GPU

Simplest setup.

• Easy to reason about
• Strong isolation
• Often wastes capacity if the model is small or traffic is variable

Best for:

• Large models
• Strict latency SLOs
• Production systems where simplicity matters

2. Multiple models on one GPU

Pack several inference workers on the same GPU.

• Better utilization
• Harder memory management
• Risk of contention and latency spikes

Best for:

• Smaller models
• Multi-tenant inference
• Bursty workloads

3. Model sharding / tensor parallelism

Split one large model across multiple GPUs.

• Needed when model doesn’t fit on a single GPU
• Adds communication overhead
• Requires careful placement and topology awareness

Best for:

• Very large LLMs
• High-memory models

4. Dynamic batching

Combine multiple requests into one GPU batch.

• Improves throughput
• Can increase latency if batch waits too long
• Often paired with admission control

Best for:

• LLM serving
• Image/text classification at scale

5. Elastic / autoscaled inference

Scale replicas up/down based on queue depth, GPU utilization, or p95 latency.

• Good cost efficiency
• Needs warm-up handling and load balancing

Best for:

• Cloud-native deployments
• Variable traffic

Key orchestration concerns

GPU memory management

• Model weights
• KV cache for LLMs
• Activations and intermediate buffers
• Fragmentation and memory spikes

Scheduling policy

Common policies:

• Round-robin
• Least-loaded
• Priority-based
• Deadline-aware
• Cost-aware / bin-packing

Placement awareness

Consider:

• GPU memory size
• NVLink / PCIe topology
• NUMA locality
• MIG partitions
• Mixed GPU types

Request routing

Route based on:

• Model availability
• Batch size
• SLA class
• GPU queue depth
• Cache locality

Admission control

Reject, defer, or shed load when:

• Batch queues get too deep
• Latency SLO is at risk
• Memory is insufficient

Common tooling

• Kubernetes + GPU operators
• NVIDIA Triton Inference Server
• vLLM / TGI / TensorRT-LLM
• Ray Serve
• KServe
• TorchServe (less common now for advanced LLM serving)

For LLM inference specifically

Important extra considerations:

• Prefill vs decode phases
• KV cache allocation
• Paged attention / memory paging
• Continuous batching
• Prefix caching
• Speculative decoding
• Request length-aware scheduling

A practical architecture

A common design looks like this:

1. Ingress/API gateway
2. Router
3. Scheduler
4. Inference workers on GPUs
5. Metrics + autoscaler
6. Fallback / retry / queue management

Good starting strategy

If you’re designing this from scratch:

1. Start with one model per GPU or replica
2. Add dynamic batching
3. Add queue-based routing
4. Add autoscaling
5. Optimize with sharding / packing / MIG only if needed

If you want, I can also give you:

• a reference architecture
• a Kubernetes-based design
• a scheduler algorithm
• or a GPU orchestration plan for LLM serving