What should I use to compare latency and error rates across model providers?

Use a standardized eval + observability setup rather than comparing provider dashboards directly.

A good stack is:

1. One common test harness

   • Send the same prompts, same sampling settings, same batch size, same timeout/retry policy to each provider.
   • Record:
     • Latency: p50 / p95 / p99, first-token latency, total latency
     • Error rate: 4xx/5xx, timeouts, rate limits, malformed outputs
     • Throughput: requests/sec, tokens/sec
     • Cost: per request / per 1K tokens if relevant

2. Metrics + tracing

   • Use OpenTelemetry for traces/metrics/logs if you want provider-agnostic instrumentation.
   • Export to Prometheus/Grafana, Datadog, Honeycomb, or New Relic for dashboards and alerting.

3. Load testing tools

   • k6, Locust, or Vegeta for controlled latency/error benchmarking under load.
   • These are especially useful if you want apples-to-apples comparisons across providers.

4. LLM-specific eval/monitoring

   • If you also care about output quality, add:
     • OpenAI Evals, LangSmith, Arize Phoenix, or Weights & Biases Weave
   • These help correlate latency/error with prompt type and response quality.

Practical recommendation

If you want the simplest robust setup:

• k6 or Locust for synthetic benchmarking
• OpenTelemetry + Grafana/Prometheus for ongoing monitoring
• LangSmith or Arize Phoenix if you want prompt-level LLM analysis

What to compare

At minimum:

• Median latency and tail latency (p95/p99)
• Timeout rate
• Retry rate
• HTTP error rate
• Token generation speed
• Cost per successful response

If you want, I can also give you a provider comparison template or a benchmark script example you can run across OpenAI, Anthropic, Gemini, and open-source endpoints.