How do I set up a compute autoscaling tool for real-time inference workloads on Kubernetes?

For real-time inference on Kubernetes, you usually want autoscaling that reacts to actual demand quickly without causing cold-start delays or thrashing. The best setup is typically:

• HPA for scaling pods based on load
• Cluster Autoscaler or Karpenter for adding/removing nodes
• Optional: a queue/requests-based scaler like KEDA if your inference is event-driven
• For inference specifically, consider GPU-aware scheduling and model warmup

Below is a practical setup pattern.

---

1) Choose the right autoscaling layers

A. Pod autoscaling: HPA

Use Horizontal Pod Autoscaler to scale your inference deployment based on:

• CPU/memory
• custom metrics like:
  • requests per second
  • queue length
  • in-flight requests
  • p95 latency
  • GPU utilization

For real-time inference, custom metrics are usually better than CPU alone.

B. Node autoscaling: Cluster Autoscaler or Karpenter

When HPA adds pods but nodes are full, you need node scaling:

• Cluster Autoscaler: standard choice, works with node groups
• Karpenter: more dynamic and often faster, especially useful for inference/GPU workloads

C. Event/queue autoscaling: KEDA

If requests come through a queue or stream, KEDA can scale pods from:

• Kafka
• RabbitMQ
• SQS
• Redis
• Prometheus metrics

For synchronous inference APIs, HPA is more common.

---

2) Make your inference service autoscaling-friendly

Your model server should:

• start quickly
• expose health/readiness endpoints
• handle graceful shutdown
• support warmup/loading before receiving traffic
• keep models in memory if possible
• avoid long startup times on each pod

If startup is slow, scale-up will still be sluggish even with autoscaling.

---

3) Expose the right metrics

For real-time inference, CPU alone often misses the real bottleneck. Better metrics:

• requests per second
• concurrent requests
• p95/p99 latency
• queue depth
• GPU memory / utilization
• batch queue size if using dynamic batching

Common ways to surface metrics:

• Prometheus + Prometheus Adapter
• OpenTelemetry metrics pipeline
• Managed service metrics if using cloud provider tooling

---

4) Example: HPA with CPU and custom metrics

Deployment

Make sure requests/limits are set:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: inference-service
  template:
    metadata:
      labels:
        app: inference-service
    spec:
      containers:
      - name: model-server
        image: myrepo/inference:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5

HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

If using custom metrics, you’d add another metric block for latency or request rate.

---

5) Add a node autoscaler

If using Cluster Autoscaler

Install Cluster Autoscaler and ensure:

• node groups are tagged correctly
• pod resource requests are realistic
• unschedulable pods trigger scale-up

If using Karpenter

Karpenter is often better for inference because it can:

• provision nodes faster
• choose instance types dynamically
• handle GPU/spot/on-demand more flexibly

Example idea:

• CPU inference on general-purpose nodes
• GPU inference on GPU node pools
• separate node classes for latency-sensitive workloads

---

6) Use GPU-aware scheduling if needed

If your model uses GPUs:

• request GPUs in pod spec:

  resources:
    limits:
      nvidia.com/gpu: 1
  

• add node selectors or affinity for GPU nodes
• use tolerations if GPU nodes are tainted
• ensure the NVIDIA device plugin is installed

Be careful: GPU scale-up is slower, so keep some warm capacity if latency matters.

---

7) Tune for real-time latency

Real-time inference often needs different settings than batch workloads:

• set minReplicas > 0 to avoid cold starts
• keep a few warm pods always running
• use pre-warmed model pods or model caches
• consider dynamic batching only if it doesn’t hurt latency
• set conservative scale-down delays to avoid churn

Useful HPA behavior tuning:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - type: Percent
      value: 100
      periodSeconds: 15
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 20
      periodSeconds: 60

This makes scale-up fast and scale-down slower.

---

8) If you need request-based scaling, use KEDA or custom metrics

For example, with Prometheus metrics, scale based on request backlog or latency.

A common pattern:

• Ingress/API gateway sends requests to service
• Service emits metrics
• Prometheus scrapes metrics
• HPA uses Prometheus Adapter custom metrics

If requests are queued, KEDA can scale directly from queue length.

---

9) Production best practices

• Set requests and limits correctly
• Use pod anti-affinity so replicas spread across nodes
• Add PodDisruptionBudgets
• Use readiness probes to avoid routing traffic too early
• Implement graceful termination
• Keep model loading outside request path
• Monitor:
  • p95 latency
  • error rate
  • queue length
  • autoscaling events
  • node provisioning time

---

10) Recommended setup by workload

CPU inference, HTTP API

• HPA on CPU + request rate
• Cluster Autoscaler or Karpenter
• minReplicas = 2 or more

GPU inference, low latency

• HPA on custom metrics
• Karpenter preferred
• warm spare GPU capacity
• readiness/warmup logic

Event-driven inference

• KEDA on queue length
• HPA/KEDA for pods
• node autoscaler for nodes

---

Simple recommended stack

If you want a solid default:

• Kubernetes Deployment
• HPA with custom metrics
• Prometheus + Prometheus Adapter
• Karpenter for node scaling
• readiness probes + model warmup
• minReplicas > 0

---

If you want, I can give you:

1. a full YAML example for HPA + Prometheus metrics,
2. a Karpenter setup for GPU inference, or
3. a production architecture diagram for real-time model serving on Kubernetes.