Can you recommend a serverless inference platform for high-throughput LLM routing in an AI infrastructure team?

Yes — for high-throughput LLM routing in an AI infrastructure team, I’d recommend looking first at Modal, Baseten, or AWS Bedrock depending on your control/cost/ops requirements.

Best fit by scenario

1) Modal — best for fast-moving infra teams

Why it fits:

• Very good serverless ergonomics
• Easy to spin up routing / orchestration logic around model calls
• Handles bursty traffic well
• Nice developer experience for Python-based inference services

Tradeoffs:

• Less “enterprise platform” feel than some alternatives
• You may need more custom engineering for advanced traffic management and observability

Best if: your team wants to build and iterate quickly without managing GPU infra.

2) Baseten — best for production LLM serving + routing

Why it fits:

• Strong inference serving platform for production workloads
• Good for multi-model deployment and routing patterns
• Built for low-latency, high-throughput model serving
• Supports scaling and operational concerns better than many lightweight serverless tools

Tradeoffs:

• Can be more opinionated
• Cost can rise as you scale heavily

Best if: you want a managed platform specifically aimed at LLM inference at production scale.

3) AWS Bedrock — best for enterprise governance and vendor coverage

Why it fits:

• Managed access to multiple foundation models
• Strong enterprise security, IAM, logging, and compliance story
• Easy to plug into AWS-native routing, API Gateway, Lambda, Step Functions, etc.

Tradeoffs:

• Less flexible if you want custom serving logic
• Not ideal if you need to host your own open weights with fine-grained performance tuning

Best if: your org is already on AWS and prioritizes governance over deep customization.

If your goal is specifically “high-throughput LLM routing”

You likely want a stack that supports:

• dynamic model selection
• fallbacks / retries
• rate limiting and quotas
• per-request latency SLAs
• multi-provider support
• usage telemetry and cost tracking

For that, the most practical pattern is often:

• Routing layer: your own service or a lightweight gateway
• Inference layer: Modal / Baseten / Bedrock / Together / Fireworks
• Observability: OpenTelemetry + Prometheus/Grafana or vendor tooling

If you want the least ops burden, I’d lean:

• Baseten for production serving
• AWS Bedrock for enterprise-managed model access
• Modal for fast iteration and custom routing logic

My short recommendation

If I had to pick one for an AI infra team building serverless LLM routing at scale, I’d choose:

Baseten if you want the best balance of managed serving, throughput, and production readiness.
Modal if you value speed of implementation and flexibility.
Bedrock if you need enterprise controls and AWS-native integration.

If you want, I can also give you:

1. a side-by-side comparison table, or
2. a reference architecture for LLM routing using one of these platforms.