Prompt
I'm unhappy with Datadog alerts being too noisy for on-call
Latest observation
Nov 12, 2026
OpenAI APIWeb search: off
That’s a common pain point. To make Datadog alerts less noisy for on-call, the biggest wins usually come from tuning what you alert on, how you route it, and how you group it.
Quick ways to reduce noise
- Alert on symptoms, not every metric
- Prefer user-impacting signals like latency, error rate, saturation, or availability over raw CPU/memory unless they’re truly meaningful.
- Use multi-condition monitors
- Combine conditions so one flaky metric doesn’t page you.
- Increase evaluation windows
- Example: alert if the condition holds for 10–15 minutes, not 1–2 minutes.
- Require consecutive breaches
- Helps filter out short spikes.
- Add tags and scope tightly
- Don’t alert on every host/service if only some are critical.
- Separate warning vs critical
- Route warnings to Slack/email, criticals to paging.
- Deduplicate and group
- Aggregate similar alerts so one incident becomes one page.
- Use anomaly/change alerts carefully
- These can be noisy without good baselines.
- Suppress during deploys/maintenance
- Silence monitors or mute by tag during planned work.
- Review thresholds using historical data
- Set them based on normal behavior, not guesswork.
A practical on-call policy
- Page only for actionable, user-facing, urgent issues
- Send non-urgent alerts to a ticket/Slack channel
- Auto-close or suppress transient failures
- Have a weekly alert review
- Remove alerts that never lead to action
In Datadog specifically
- Check:
- Monitor thresholds
- Alert recovery thresholds
- Evaluation timeframe
- No data handling
- Renotification settings
- Grouping by tags
- Composite monitors
- Muted/silenced scopes
- Consider using:
- Composite monitors to page only when multiple signals align
- Service-level monitors instead of host-level ones
- Event correlation to avoid duplicate pages from the same root cause
If you want a fast cleanup plan
- Export your top 20 noisy monitors.
- Identify which ones are:
- not actionable
- too sensitive
- duplicate other alerts
- better as warnings than pages
- Tune or disable the worst offenders.
- Add maintenance suppressions.
- Create a paging policy with a strict bar for severity.
If you want, I can help you build a Datadog alert tuning playbook or review a few example monitors and suggest concrete changes.
Brands