What should I use for batch inference jobs?

For batch inference jobs, the best choice is usually one of these depending on your setup:

If you want the simplest option

Use a standard batch job with your model loaded inside the worker.
Good when:

• you have a fixed dataset
• throughput matters more than low latency
• you don’t need an always-on endpoint

If you’re using cloud ML platforms

Choose the platform’s dedicated batch inference feature:

• AWS SageMaker Batch Transform
• Google Vertex AI Batch Prediction
• Azure ML Batch Endpoints

These are designed for:

• large offline datasets
• parallel processing
• output written to files or storage

If you’re running on your own infrastructure

Use:

• Python multiprocessing / async workers for smaller jobs
• Ray
• Dask
• Spark if the data is very large and already in a distributed ecosystem

If the model is an LLM

For batch LLM inference, common choices are:

• vLLM for high-throughput GPU inference
• TGI (Text Generation Inference) if you want a serving stack that can also be used in batch mode
• HF Transformers + Accelerate for simpler pipelines
• Ray Serve / Ray jobs for distributed batch runs

Quick rule of thumb

• Small/medium jobs: plain Python workers or multiprocessing
• Large batch jobs on cloud: managed batch inference service
• Huge distributed jobs: Spark / Ray / Dask
• LLM-heavy workloads: vLLM

If you tell me:

1. your model type,
2. data size,
3. cloud/on-prem setup, I can recommend the best specific tool.