What should I use if I need to support both batch and real-time inference?

If you need to support both batch and real-time inference, the usual choice is to use a single model serving layer that can handle both online and offline requests.

A few good patterns:

1. Use the same model, deployed in two modes

• Real-time inference: low-latency API service
• Batch inference: scheduled jobs over large datasets
• Best when you want consistency between both paths

2. Use a serving framework that supports both

Examples:

• TensorFlow Serving for online serving, paired with batch jobs that load the same model
• TorchServe or BentoML for both API serving and batch execution
• MLflow Models with separate batch and online consumers
• KServe / Seldon in Kubernetes setups

3. If you want one platform for both

A common architecture is:

• Model registry stores the trained model
• Online endpoint for real-time predictions
• Batch pipeline reads from the same registry and writes results to storage

Practical recommendation

If you’re deciding broadly:

• For simple setups: use BentoML or MLflow
• For Kubernetes/cloud-native setups: use KServe or Seldon
• For pure TensorFlow or PyTorch workflows: use the native serving tool plus a batch job framework

If you want, I can recommend the best option based on your stack, like AWS/GCP/Azure, Python/Java, or Kubernetes/non-Kubernetes.