Prompt
We keep missing alerts when one provider has issues
Latest observation
Nov 11, 2026
OpenAI APIWeb search: off
That usually means your alerting is tied too closely to a single provider or a single failure signal.
A better approach is to detect provider-specific failures and cross-provider impact separately:
1) Alert on the provider itself
Create alerts for each provider based on:
- error rate
- timeout rate
- latency spikes
- connection failures
- auth failures
- quota/rate-limit errors
This catches “Provider A is broken” even if your app still works via fallback.
2) Alert on user-facing impact
Also alert on:
- failed requests after retries/fallbacks
- degraded success rate
- increased checkout/login/job failure rate
- dropped throughput
- elevated incident tickets
This catches when the fallback path is no longer enough.
3) Separate “warning” from “critical”
For a single provider issue:
- Warning: provider error rate > threshold for 5–10 min
- Critical: customer-facing failures exceed threshold or failover is exhausted
4) Use synthetic checks per provider
Run health checks against each provider independently, not just through your app.
That way you can see if:
- provider endpoint is down
- auth broke
- specific region is failing
5) Correlate but don’t suppress
If one provider fails and another takes over, don’t suppress the provider alert just because service is still up.
Instead, route it as:
- “Provider degraded”
- “Service impacted”
- “Failover active”
6) Add dashboards for comparison
Track per provider:
- success rate
- p95 latency
- error categories
- fallback usage
- region-specific issues
If you want, I can help you design:
- an alerting policy,
- Prometheus/Grafana rules,
- or a PagerDuty routing setup for multi-provider failover.