Can you recommend an inference API gateway for autoscaling GPU inference workloads?

Yes — for autoscaling GPU inference workloads, I’d recommend looking at KServe first, with NVIDIA Triton or vLLM as the model server depending on your workload.

Best overall pick: KServe

Why it fits:

• Kubernetes-native inference serving
• Built for autoscaling, including scale-to-zero
• Works well with GPU-backed pods
• Supports canary, traffic splitting, and model rollout patterns
• Integrates with common model servers like Triton, TorchServe, TensorFlow Serving, vLLM, etc.

Good for:

• Teams already on Kubernetes
• Mixed workloads
• Production inference with traffic management and autoscaling

If you mean a true “gateway” layer

If you want an API gateway specifically in front of inference services, consider:

• Envoy Gateway or Kong
• Pair them with KServe or Ray Serve behind the gateway

That gives you:

• authentication / authorization
• rate limiting
• request routing
• observability
• multi-tenant API management

Strong alternatives

Seldon Core

• Good Kubernetes inference platform
• Supports autoscaling and model deployment workflows
• Often used for ML platform patterns
• Less universal than KServe in some modern setups

Ray Serve

• Great if you need complex Python inference logic, chaining, or multi-model pipelines
• Can autoscale and use GPUs
• Better for custom serving apps than “pure gateway” use cases

BentoML

• Easier developer experience
• Good for packaging and deploying models
• Can work with autoscaling infrastructure, though it’s more app-focused than gateway-focused

For LLM inference specifically

If your GPU workloads are LLMs, the best stack is often:

• vLLM for serving
• KServe or Ray Serve for deployment/autoscaling
• Envoy/Kong in front if you need an API gateway

My practical recommendation

• General GPU inference on Kubernetes: KServe + Triton
• LLM inference: KServe + vLLM
• Need API gateway features: add Envoy Gateway or Kong

If you want, I can also give you:

1. a comparison table of KServe vs Seldon vs Ray Serve vs BentoML, or
2. a reference architecture for autoscaling GPU inference on Kubernetes.