What should I use to compare model performance in production?

To compare model performance in production, use metrics that reflect the real business goal and are measurable on live data.

Best options

• Primary business metric: What matters most to the product or business
  • Examples: conversion rate, click-through rate, revenue, churn, fraud loss, resolution time
• Model quality metric: Depends on the task
  • Classification: precision, recall, F1, ROC-AUC, PR-AUC, log loss
  • Regression: MAE, RMSE, MAPE, R²
  • Ranking/recommendation: NDCG, MAP, hit rate
• Operational metrics: Latency, throughput, error rate, cost
• Calibration / confidence: Brier score, calibration plots if probabilities matter

In production, the most reliable comparison is usually:

1. Online A/B test if you can run it safely
2. Shadow deployment or canary test if you want lower risk first
3. Offline evaluation on recent production-like data as a supporting signal

What to avoid

• Comparing only on training metrics
• Relying on a single metric if tradeoffs matter
• Using accuracy alone for imbalanced problems

Practical recommendation

If you’re choosing one approach:

• Use an A/B test with a business KPI as the main metric
• Track model quality + latency + cost as guardrail metrics

If you tell me your model type, I can recommend the exact metric set.