How do I set up model serving platform infrastructure for multi-GPU batch inference jobs?

A practical way to set up model-serving platform infrastructure for multi-GPU batch inference is to design it as a distributed batch system, not classic low-latency online serving. The core idea is:

• API/control plane submits batch jobs
• Scheduler allocates GPUs and nodes
• Workers load model shards or replicas
• Storage/data plane feeds input and collects output
• Observability tracks throughput, latency, GPU utilization, failures

Below is a solid reference architecture and setup plan.

---

1) Start with the workload shape

Before choosing tools, define:

• Model type: LLM, vision, embedding, reranker, multimodal, etc.
• Batch pattern:
  • many small independent requests
  • large files / datasets
  • streaming micro-batches
• Serving requirement:
  • throughput-first
  • cost-first
  • deadline-based
• Parallelism needs:
  • data parallel across requests
  • tensor parallel / pipeline parallel for one large model
  • model replicas for horizontal scale

For multi-GPU batch inference, the most common patterns are:

1. One job per model replica using multiple GPUs for model sharding
2. Many jobs distributed across GPUs for independent sample inference
3. Hybrid: one multi-GPU replica + queue of batches

---

2) Recommended high-level architecture

Control plane

• Job submission API: REST/gRPC/UI
• Job metadata store: Postgres, MySQL, or DynamoDB
• Queue: Kafka, Redis, RabbitMQ, SQS, Pub/Sub
• Scheduler: Kubernetes scheduler, Kueue, Volcano, Ray, or custom
• Artifact registry:
  • model weights in S3/GCS/Azure Blob/NFS
  • container images in a registry

Data plane

• GPU worker pods/nodes
• Inference runtime:
  • vLLM
  • Triton Inference Server
  • TensorRT-LLM
  • Ray Serve
  • Hugging Face TGI
  • custom PyTorch distributed inference
• Shared storage for inputs/outputs
• Result sink: object storage, database, event stream

Observability

• Prometheus + Grafana
• centralized logs
• tracing if needed
• GPU metrics via DCGM exporter

---

3) Choose an orchestration layer

Best default: Kubernetes

Use Kubernetes if you want:

• GPU scheduling
• autoscaling
• job isolation
• standard ops and observability
• repeatable deployments

Useful components:

• NVIDIA GPU Operator for drivers/runtime/device plugin
• Kueue for batch queueing and fair sharing
• Volcano if you need gang scheduling and batch-centric scheduling
• Argo Workflows or Kubeflow Pipelines for complex pipelines
• Ray on Kubernetes if you want distributed Python-native execution

Alternatives

• Ray cluster: very good for Python batch inference and task parallelism
• Slurm: common in HPC environments
• Nomad: simpler, less common for GPU ML infra
• Bare metal + custom scheduler: only if you have very specialized needs

If you’re building a platform, a common choice is:

Kubernetes + GPU Operator + Kueue/Volcano + vLLM/Triton/Ray

---

4) Decide how to use multi-GPU

There are two main modes.

A. Data parallel inference

Use when:

• each request/sample is independent
• model fits on one GPU
• you want max throughput

Approach:

• replicate the model across GPUs
• shard batches across replicas
• each worker handles a subset of inputs

Pros:

• simple
• scales well
• easy failure isolation

Cons:

• duplicates model memory

B. Model parallel inference

Use when:

• model is too large for one GPU
• you need tensor parallelism or pipeline parallelism

Approach:

• one inference instance spans multiple GPUs
• use NCCL for inter-GPU comms
• run one worker per node or across nodes depending on runtime

Pros:

• supports very large models
• avoids fitting constraints

Cons:

• more complex
• sensitive to network and topology
• harder scheduling

Common rule of thumb

• If the model fits on one GPU: replicate
• If it does not: tensor parallel / sharded serving

---

5) Pick an inference runtime

For LLM batch inference

• vLLM: excellent throughput, paged attention, easy batching
• TensorRT-LLM: highest performance on NVIDIA, more setup
• TGI: easy Hugging Face ecosystem integration
• Triton: great for mixed workloads and model management

For general deep learning inference

• Triton Inference Server
• TorchServe if legacy PyTorch-centric, though less popular now
• custom PyTorch + multiprocessing/DDP
• ONNX Runtime / TensorRT for optimized inference

For flexible distributed batch jobs

• Ray Serve / Ray Data
• Spark + UDF + GPU scheduling for ETL-style inference
• Dask in some environments

---

6) Design the job execution model

A good batch inference job spec should include:

• model name/version
• input source URI
• output destination URI
• GPU count
• CPU/memory request
• batch size / microbatch size
• timeout
• retry policy
• priority
• placement constraints
• parallelism strategy

Example conceptual job flow:

1. User submits a batch job with input data in S3
2. Scheduler reserves 8 GPUs on 2 nodes
3. Runtime loads the model weights from shared storage/cache
4. Input data is split into chunks
5. Workers process chunks in parallel
6. Outputs are written to S3/DB
7. Job status updates are emitted throughout

---

7) Make scheduling topology-aware

For multi-GPU jobs, topology matters.

If using 1 node, 8 GPUs

• easiest and fastest
• NVLink/NVSwitch gives better performance
• ideal for tensor parallel inference

If spanning multiple nodes

• use only when necessary
• ensure:
  • fast interconnect like InfiniBand or RoCE
  • NCCL configured properly
  • correct pod/node affinity
  • gang scheduling so all GPUs are allocated together

Kubernetes scheduling best practices

• use node labels for GPU type
• use taints/tolerations to reserve GPU nodes
• use pod affinity/anti-affinity for placement
• use topology spread constraints if running replicas
• use gang scheduling for multi-GPU distributed jobs

---

8) Handle model and data distribution efficiently

Model distribution

Avoid downloading weights from the internet on every job start.

Use:

• object storage + local cache
• pre-baked container images for small models
• shared read-only volume for larger models
• node-local cache or daemonset warmers

Input data distribution

• store input files in object storage
• chunk large datasets into shards
• use manifest files
• stream records via queue or table scan if needed

Output handling

• write outputs partitioned by job/chunk
• use idempotent writes
• include job_id, shard_id, record_id for traceability

---

9) Build batching and throughput controls

Batch inference performance often depends on batch management more than raw GPU count.

Key tuning knobs:

• static batch size
• dynamic batching window
• microbatch size
• max sequence length
• concurrency per GPU
• prefill/decode batching
• token limits per batch

For LLMs:

• use request-level batching plus token-based scheduling
• separate prompt processing from generation when supported
• limit max tokens to avoid tail latency and OOMs

For vision or embeddings:

• pack records into batches by shape
• group similar tensor sizes to reduce padding waste

---

10) Add fault tolerance

Batch platforms need resilient retries.

Implement:

• job-level retries
• shard-level retries
• checkpointing for long jobs
• idempotent output writes
• lease/heartbeat mechanism for workers
• timeout and preemption handling
• poison input quarantine

If a worker fails:

• requeue the shard
• avoid recomputing successful shards
• record partial progress

If a node fails:

• scheduler resubmits the job or shard
• distributed jobs should detect rank loss cleanly

---

11) Security and tenancy

If multiple teams share the platform:

• separate namespaces or accounts
• RBAC for model and data access
• network policies
• secrets management via Vault/KMS/Secrets Manager
• encryption at rest and in transit
• per-tenant quotas
• audit logs

Also consider:

• limiting egress from inference pods
• scanning model artifacts and containers
• restricting who can deploy custom code

---

12) Observability you actually need

Track:

• job submission rate
• queue wait time
• job completion time
• GPU utilization
• memory utilization
• batch size distribution
• tokens/sec or samples/sec
• error rate
• retry count
• OOMs and NCCL failures
• model load time
• input/output throughput

For GPU monitoring:

• NVIDIA DCGM exporter
• node exporter
• container logs with structured fields

Useful alerts:

• GPU utilization below threshold
• queue backlog too high
• repeated job failures
• model load errors
• node pressure / memory exhaustion

---

13) Autoscaling strategy

You’ll usually want two types of autoscaling:

Worker autoscaling

Scale GPU worker pods/nodes based on:

• queue depth
• pending jobs
• GPU utilization
• SLA deadlines

Node autoscaling

Add/remove GPU nodes when cluster demand changes.

In Kubernetes:

• Cluster Autoscaler or Karpenter
• KEDA for event-driven scaling
• custom metrics autoscaling

For batch workloads, scaling should be based more on queue backlog and job SLA than on CPU.

---

14) A concrete reference stack

Here’s a practical stack that works well:

Small to medium platform

• Kubernetes
• NVIDIA GPU Operator
• Kueue or Volcano
• vLLM or Triton
• S3-compatible object storage
• Postgres for metadata
• Prometheus + Grafana
• Karpenter/Cluster Autoscaler

Larger platform

• Kubernetes multi-cluster
• Kueue/Volcano
• Ray for distributed tasks
• TensorRT-LLM / vLLM / Triton depending on model type
• Dedicated model registry and artifact cache
• Kafka for events
• Central observability stack
• Node-local SSD cache for weights

---

15) Example deployment pattern for multi-GPU LLM batch jobs

If model fits on 1 GPU

• deploy 1 replica per GPU
• each replica uses dynamic batching
• scheduler assigns jobs to least loaded worker

If model needs 4 GPUs

• schedule one pod requesting 4 GPUs
• run tensor parallel inference inside the pod
• each job gets one pod
• queue jobs until 4 GPUs are available

If you have many independent requests

• split input into shards
• run multiple pods in parallel
• each pod processes a subset
• aggregate results at the end

---

16) Common pitfalls

• downloading model weights every job start
• ignoring topology for multi-GPU jobs
• using batch sizes that OOM the GPU
• not separating control plane from data plane
• no job retries or checkpointing
• no GPU metrics
• mixing online traffic with batch traffic on the same nodes without safeguards
• underestimating NCCL/network issues for multi-node jobs

---

17) Suggested implementation path

If you’re starting from scratch:

Phase 1

• Kubernetes cluster with GPU nodes
• GPU Operator
• one inference runtime (vLLM or Triton)
• job submission API
• object storage for input/output
• basic logging and metrics

Phase 2

• batch queueing with Kueue/Volcano
• shard-based execution
• retries and job status tracking
• model cache and artifact registry

Phase 3

• autoscaling
• topology-aware scheduling
• per-tenant quotas
• advanced observability
• workload-specific optimizations

---

18) Practical architecture recommendation

If you want a safe default recommendation:

• Kubernetes for orchestration
• NVIDIA GPU Operator
• Kueue for batch queueing
• vLLM for LLM batch inference, Triton for general workloads
• S3/GCS for inputs, outputs, and model artifacts
• Postgres for job metadata
• Prometheus/Grafana for metrics
• Karpenter or Cluster Autoscaler for node scaling

This gives you a platform that can support both:

• multi-GPU single-job inference
• many independent batch jobs across the fleet

---

If you want, I can also give you:

1. a Kubernetes reference architecture diagram in text form,
2. a sample Helm/K8s manifest set, or
3. a design for a batch inference control plane API.