I'm building a batch scoring workflow, what should I use for model serving?

For a batch scoring workflow, you usually want offline/batch inference, not low-latency online serving.

What to use

Choose a serving approach that can:

• read inputs from files / tables / object storage,
• process many records at once,
• write predictions back to storage,
• run on a schedule or as a job.

Common options:

1. Batch inference jobs

   • Best fit for batch scoring.
   • Examples: scheduled Spark jobs, Kubernetes jobs, cloud batch jobs, Airflow/Argo pipelines.

2. Model serving frameworks with batch support

   • If you already use a serving stack, pick one that supports bulk requests or asynchronous jobs.
   • Examples: TensorFlow Serving batch APIs, TorchServe, KServe batch inference patterns, BentoML batch endpoints, Ray Serve for distributed batch work.

3. Data/ML platform batch inference

   • Managed services can simplify operations.
   • Examples depend on cloud provider.

Rule of thumb

• Need real-time predictions? Use online serving.
• Need scoring thousands/millions of records periodically? Use batch processing infrastructure, not a traditional REST serving endpoint.

Good default recommendation

If you’re starting fresh, use:

• A batch job runner like Spark / Ray / Kubernetes Jobs
• plus object storage or a warehouse for input/output
• plus MLflow/BentoML/KServe only if you need packaging/versioning around the model.

If you tell me your stack (cloud, language, model type, data volume), I can recommend a specific serving setup.