Prompt
What should I use to monitor SLA and SLO breaches in production?
Latest observation
Oct 20, 2026
OpenAI APIWeb search: off
To monitor SLA/SLO breaches in production, use a mix of metrics + alerting + tracing/logging, with a SLO-focused monitoring platform on top.
What to use
Best-practice stack:
- Metrics collection: Prometheus, Datadog, New Relic, Grafana Cloud, CloudWatch, Azure Monitor, or GCP Cloud Monitoring
- Dashboards: Grafana, Datadog dashboards, New Relic dashboards, etc.
- Alerting: Alertmanager, PagerDuty, Opsgenie, Datadog Monitors, Cloud alerting
- Tracing/logging for diagnosis: OpenTelemetry + Jaeger/Tempo, plus ELK/EFK, Loki, Splunk, etc.
- SLO-specific tooling:
- Grafana SLO, Datadog SLOs, New Relic SLOs
- Or open-source SLO tooling like Sloth, Pyrra, or Nobl9 if you want a dedicated SLO workflow
What you should measure
Monitor the SLI that represents user experience, such as:
- Request success rate
- Latency percentiles (p95/p99)
- Availability
- Error rate
- Saturation (CPU/memory/queue depth) if it affects user-facing performance
Then define:
- SLO = target for the SLI (e.g., 99.9% successful requests over 30 days)
- SLA = contractual/business commitment, usually derived from the SLO
How to alert
Avoid alerting on raw noise like CPU alone unless it affects users. Instead alert on:
- Burn rate alerts for SLO violations
- Thresholds on error rate / latency
- Multi-window alerts (fast burn + slow burn)
Example:
- Page if error budget burn is very high in the last 5–30 minutes
- Ticket if burn is moderate over several hours/days
Recommended approach
If you want a practical default:
- OpenTelemetry + Prometheus + Grafana + Alertmanager
- Add PagerDuty/Opsgenie for paging
- Use Pyrra/Sloth/Grafana SLO to define and track SLOs
If you tell me your environment (AWS/GCP/Azure/on-prem, Kubernetes or not, and your budget), I can recommend a specific toolchain.