Can you recommend a model serving platform for A/B testing model versions in a live product?

Yes — for A/B testing model versions in a live product, I’d generally recommend a platform that supports:

• traffic splitting / canary releases
• versioned model deployment
• metrics and monitoring
• easy rollback
• low-latency inference
• ideally metadata / experiment tracking

Good options

1. Seldon Core / Seldon Deploy

Best if you’re on Kubernetes and want strong experiment routing.

Why it’s good for A/B testing

• Native support for canary, shadow, and A/B traffic splitting
• Works well with Kubernetes
• Supports multiple model versions behind one endpoint
• Good for gradual rollout and controlled experiments

Best for

• Teams already using K8s
• ML platform teams who want flexibility and control

2. KServe

Good open-source option for model serving on Kubernetes.

Why it’s good

• Supports multiple revisions and traffic splitting
• Integrates with common ML tooling
• Good for standardized model serving workflows

Best for

• Kubernetes-native ML platforms
• Teams that want an open-source serving layer with less vendor lock-in

3. AWS SageMaker

Strong managed option if you’re already on AWS.

Why it’s good

• Supports production variants for A/B testing
• Easy to shift traffic between model versions
• Managed autoscaling, monitoring, and deployment tooling

Best for

• AWS-first teams
• Teams that want managed infrastructure over self-hosting

4. Vertex AI

Good managed option on Google Cloud.

Why it’s good

• Supports traffic splitting across endpoints/models
• Managed deployment and scaling
• Integrates with GCP ecosystem

Best for

• GCP-first teams

5. BentoML + an inference layer

Useful if you want flexibility and a lightweight serving framework.

Why it’s good

• Simple to package and deploy models
• Can be paired with an API gateway or service mesh for A/B routing
• Good developer experience

Best for

• Teams that want to build their own serving/experiment routing stack

My practical recommendation

If your main goal is A/B testing in production, I’d choose based on your infrastructure:

• On Kubernetes: Seldon Core or KServe
• On AWS: SageMaker
• On GCP: Vertex AI
• If you want maximum control and custom routing: BentoML + gateway/service mesh

What to look for specifically

When evaluating, make sure the platform supports:

• percentage-based traffic splitting
• request stickiness if users should stay on the same model
• per-variant logging
• latency and error metrics by model version
• easy rollback
• shadow testing before full A/B
• integration with experiment tracking like MLflow/W&B

If you tell me your stack — for example AWS/GCP/Kubernetes, your latency needs, and whether you need online experimentation vs shadow testing, I can recommend the best fit more precisely.