Do I need inference infrastructure for batch jobs?

Usually yes—but not always in the “online inference” sense.

For batch jobs, you still need a place to run the model and manage compute, but it can be much simpler than real-time inference infrastructure.

You need some inference infrastructure if:

• You’re running a model to score lots of records, generate embeddings, classify documents, etc.
• The job needs GPUs/CPUs, model loading, autoscaling, retries, monitoring, or checkpointing.
• You want repeatable, scheduled processing at scale.

In that case, batch inference infrastructure might be:

• a batch worker
• a job queue + worker pool
• a Spark / Ray / Databricks / Kubernetes job
• a containerized script that loads the model and processes files

You may not need dedicated inference infrastructure if:

• The job is tiny and runs occasionally
• You can just call the model once from a notebook or script
• Latency doesn’t matter and scale is trivial
• You’re using an external API directly and don’t manage the model yourself

Rule of thumb

• Online inference = low-latency, always-on serving
• Batch inference = scheduled or ad hoc, high-throughput processing
• Both are “inference,” but batch usually uses lighter, cheaper infrastructure

If you want, I can help you decide for your specific setup (e.g. ML model, LLM, embeddings, data size, frequency).