How do I stop inference pods from OOMing on GPUs?

OOMs during GPU inference usually come from one of three places:

1. GPU memory exhausted by the model / batch / KV cache
2. CPU memory exhausted by preprocessing, tokenization, staging, or framework overhead
3. Kubernetes/container limits causing the process to get killed before the framework can recover

Here’s a practical checklist to stop it.

---

1) First confirm what is OOMing

Check whether it’s GPU OOM or container OOM

• GPU OOM usually shows up in app logs as:
  • CUDA out of memory
  • hipErrorOutOfMemory
  • failed to allocate
• Container / pod OOMKill shows up as:
  • OOMKilled
  • exit code 137
  • kubectl describe pod ... events mention memory limit exceeded

Inspect:

kubectl describe pod <pod>
kubectl logs <pod>
nvidia-smi
kubectl top pod <pod>

If the pod is killed by Kubernetes, increasing GPU memory won’t help unless CPU memory pressure is also fixed.

---

2) Reduce GPU memory use

A. Lower batch size / concurrency

This is the most common fix.

• Reduce per-request batch size
• Reduce number of simultaneous in-flight requests
• Add request queueing/backpressure

For LLMs, also reduce:

• max input length
• max output length
• max total tokens per request

B. Use smaller / more efficient precision

Use lower precision if your model/runtime supports it:

• FP16/BF16 instead of FP32
• INT8 or INT4 quantization if acceptable

Examples:

• TensorRT-LLM / vLLM / TGI often support better memory efficiency than vanilla Python serving
• Quantization can dramatically cut weights memory, but sometimes increases latency slightly

C. Limit KV cache growth

For transformer inference, KV cache is a frequent culprit.

Control:

• max sequence length
• max tokens generated
• max concurrent sequences
• page/block size if your serving stack supports it

If you run an LLM server, make sure the scheduler isn’t allowing too many long-context requests at once.

D. Enable memory-efficient attention / serving backend

Use kernels and runtimes designed for inference:

• FlashAttention / SDPA
• TensorRT
• vLLM paged attention
• optimized batching in Triton/TGI/TensorRT-LLM

These reduce fragmentation and cache overhead.

E. Avoid loading extra copies of the model

Common issue:

• one process loads the model multiple times
• worker forking duplicates memory
• separate Python workers each hold a full model copy

Fix:

• run a single model process per GPU
• avoid --workers > 1 unless the runtime is explicitly built for it
• use model sharding if needed instead of replication

---

3) Reduce CPU memory use too

A lot of “GPU OOM” incidents are actually CPU-side OOMs that destabilize the pod.

Common causes

• huge request payloads
• large intermediate tensors on CPU
• tokenization buffers
• loading the model twice
• too many dataloader/preprocessing workers

Fixes

• cap request size
• stream large inputs if possible
• reduce preprocessing worker count
• move preprocessing closer to the client or into a separate service
• set realistic pod memory requests/limits

---

4) Tune Kubernetes resources correctly

Make sure the pod has enough memory

If the container limit is too low, the pod dies even if GPU has space.

Example:

resources:
  requests:
    cpu: "2"
    memory: "8Gi"
    nvidia.com/gpu: 1
  limits:
    cpu: "4"
    memory: "16Gi"
    nvidia.com/gpu: 1

Don’t set CPU/memory limits too tightly

Inference workloads can have bursts:

• tokenization spikes
• startup model load spike
• request batching spikes

If possible:

• give enough headroom
• set requests close to average usage
• set limits above peak usage

Watch for startup OOMs

Model loading can require much more memory than steady-state inference.

If startup is the issue:

• use smaller init image / fewer startup workers
• increase memory limit during startup, or keep a safe buffer
• consider lazy loading only if supported

---

5) Prevent fragmentation and allocator issues

GPU OOM can happen even when total free memory looks sufficient due to fragmentation.

Helpful tactics

• restart long-running pods periodically if fragmentation builds up
• use allocator tuning if applicable:
  • PyTorch: PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128
  • sometimes expandable_segments:True helps in newer versions
• avoid repeated load/unload cycles in the same process
• pre-allocate or use fixed-size batching if your stack supports it

---

6) Control request patterns

Even a well-sized model will OOM if traffic is spiky.

Add safeguards:

• max request tokens
• max batch size
• max concurrent requests
• queue depth limit
• timeout on very long requests
• rate limiting per tenant/user

For LLMs, one or two enormous prompts can evict memory needed by everyone else.

---

7) Scale horizontally instead of vertically

If a single pod is too close to the limit, add replicas and load balance:

• fewer requests per pod
• lower tail latency
• less memory contention

If the model is too large for one GPU:

• use tensor parallelism / model sharding
• use a larger GPU
• use model quantization

---

8) Practical debugging steps

Measure peak memory usage under realistic load

• GPU:

  watch -n 0.5 nvidia-smi
  

• PyTorch:

  torch.cuda.max_memory_allocated()
  torch.cuda.max_memory_reserved()
  

Reproduce with the worst-case request

Test:

• longest prompt
• largest batch
• highest concurrency
• startup and steady-state separately

Look at pod events

kubectl get events --sort-by=.lastTimestamp

---

9) If you’re serving an LLM, the biggest wins are usually

1. Reduce max context / max output tokens
2. Limit concurrency
3. Use batching carefully
4. Use quantization
5. Use vLLM / TensorRT-LLM / TGI instead of a naive server
6. Give the pod enough CPU memory for startup and tokenization
7. Set sane Kubernetes memory limits

---

Quick rule of thumb

If you want the fastest path to stability:

• cut batch size/concurrency by 2–4x
• reduce max tokens
• switch to FP16/BF16 or quantized weights
• increase pod memory limit
• cap request sizes and queue excess traffic

---

If you want, I can help you diagnose a specific setup. If you paste:

• your model/runtime
• pod resources YAML
• an OOM log snippet
• whether it’s GPU OOM or OOMKilled

…I can suggest the exact fix.