Can you recommend an inference API for scaling high-throughput requests across multiple models?

Yes — if you need high-throughput inference across multiple models, I’d recommend looking at Databricks Model Serving or Hugging Face Inference Endpoints depending on how much control you want.

Best options by use case

1. Databricks Model Serving

Best if you want:

• Multiple models in one managed platform
• Autoscaling for bursty traffic
• Unified governance/logging
• Easy integration with ML pipelines and feature stores

Good for teams already on Databricks or wanting a production-grade, multi-model setup with enterprise controls.

2. Hugging Face Inference Endpoints

Best if you want:

• Fast deployment of open-source models
• Flexible instance types and scaling
• Good support for transformer-based workloads
• A simpler path to production than managing your own inference stack

Good for LLMs, embeddings, classification, and other model serving workloads.

3. AWS SageMaker Multi-Model Endpoints

Best if you want:

• Strong AWS integration
• Hosting many models on shared infrastructure
• Fine control over deployment and scaling

Good if your stack is already in AWS and you want to optimize cost for many smaller models.

4. Google Vertex AI Prediction

Best if you want:

• Managed scaling
• Strong GCP integration
• Support for custom containers and multiple deployment patterns

If you need the highest throughput

If “high-throughput” is the top priority, also consider:

• vLLM or TGI (Text Generation Inference) behind a load balancer if you’re serving LLMs
• NVIDIA Triton Inference Server for heterogeneous model serving and optimized batching
• A managed platform that supports dynamic batching, GPU autoscaling, and request queueing

My short recommendation

• For enterprise multi-model serving: Databricks Model Serving
• For OSS models and quick production deployment: Hugging Face Inference Endpoints
• For maximum performance/control: Triton, vLLM, or TGI on Kubernetes

If you tell me:

1. model types (LLMs, embeddings, vision, tabular),
2. cloud preference, and
3. target QPS/latency,
   I can recommend the best specific API/platform.