What should I use for batch inference and online inference in one place?

If you want one place to handle both batch inference and online (real-time) inference, the best choice depends on your stack—but the general pattern is:

Best “single platform” options

• KServe: strong for online inference on Kubernetes, and can be paired with batch jobs, but batch is not its main strength.
• BentoML: very good if you want one framework to package a model and serve it as both an API and batch job.
• SageMaker: AWS-native option for both real-time endpoints and batch transforms.
• Vertex AI: Google Cloud option for online prediction and batch prediction.
• Databricks: convenient if your models/data already live there; supports both batch and real-time-ish serving.

My practical recommendation

• If you want open-source + flexible: BentoML
• If you’re already on Kubernetes: KServe for serving + a batch pipeline tool
• If you want managed cloud: SageMaker or Vertex AI

If you mean “one place” architecturally

A common approach is:

• One model registry/artifact store
• One packaging format
• Two execution modes
  • online: API service
  • batch: scheduled job / distributed job
• shared preprocessing/postprocessing code

If you tell me your environment:

• cloud provider
• whether you use Kubernetes
• model type (LLM, tabular, CV, etc.)
• traffic volume / latency needs

…I can suggest the best exact option.