Prompt
How can I monitor HTTP errors and latency in production without drowning in alerts?
Latest observation
To monitor HTTP errors and latency in production without alert fatigue, the key is to separate observability from alerting:
1) Track the right signals
Use a small set of core metrics for every service/API:
- Request rate: requests/sec
- Error rate: 4xx and 5xx, but especially 5xx
- Latency: p50, p95, p99
- Saturation: CPU, memory, connection pool, queue depth
A simple way to structure this is the RED method for request-driven services:
- Rate
- Errors
- Duration
2) Alert on symptoms, not every error
Don’t alert on single failures or short spikes. Alert when the service is actually user-impacting.
Good alert patterns:
- High 5xx error rate for a sustained window
- Latency above threshold for p95/p99 over several minutes
- Error budget burn rate (best for mature teams)
Bad alert patterns:
- Every 500 response
- Every timeout
- Thresholds that fire during normal traffic noise
3) Use alert windows and burn rates
Instead of “>1% errors right now,” use a rolling window:
- Fire if error rate is high for 5–10 minutes
- Prefer multi-window burn rate alerts:
- fast burn: catches major incidents quickly
- slow burn: catches smaller but sustained issues
Example:
- Page if 5xxs are consuming error budget too fast over 5 minutes
- Ticket if burn is elevated over 1 hour
This reduces false positives dramatically.
4) Separate paging alerts from informational alerts
Create two levels:
Paging alerts
Only for urgent, user-visible incidents:
- API down
- 5xx rate massively elevated
- latency severe enough to break SLOs
Non-paging notifications
Send to Slack/email for:
- moderate latency increases
- elevated 4xx rates
- retries increasing
- saturation trends
This keeps humans from being interrupted unnecessarily.
5) Use SLOs and error budgets
Define service-level objectives like:
- 99.9% of requests succeed
- 99% of requests under 300ms
Then alert on burning through the error budget, not raw metrics alone.
This aligns alerts with user impact.
6) Add filtering and grouping
If you’re using an alerting system, make sure it supports:
- deduplication
- grouping
- suppression during deployments
- routing by service/team
- alert cooldowns
Also group by:
- endpoint
- customer tier
- region
- error class
This helps avoid dozens of alerts for one root cause.
7) Watch percentiles carefully
Latency averages hide problems. Use:
- p95 for general user experience
- p99 for tail latency
- optionally histogram-based alerts
Alert if:
- p95 is above threshold for N minutes
- p99 is exploding while traffic is stable
8) Correlate errors with deployments
Many production incidents come from releases. Add:
- deployment markers
- version labels
- canary analysis
This helps distinguish real incidents from normal variation.
9) Include enough context in alerts
Every alert should answer:
- What service?
- What endpoint?
- What changed?
- How severe is it?
- Is it affecting all users or just one region?
Without context, even good alerts become noisy because they require manual investigation.
10) Practical setup example
For a typical HTTP service:
- Dashboard: request rate, 5xx rate, p95 latency, p99 latency, saturation
- Page alert:
- 5xx rate > 2% for 5 minutes
- p95 latency > 1s for 10 minutes
- error budget burn rate > threshold
- Warn alert:
- 4xx spike
- p95 latency trending upward
- queue depth increasing
Recommended philosophy
A good rule is:
Alert only when a human needs to take action.
Everything else should go to dashboards, logs, or lower-priority notifications.
If you want, I can also give you:
- a Prometheus/Grafana alert rule example, or
- a minimal SLO-based alerting strategy for a web API.