Prompt
I'm building a tool for DNS and CDN outage detection
Latest observation
Nov 19, 2026
OpenAI APIWeb search: off
Nice use case. For DNS/CDN outage detection, the most useful design is usually multi-layered: don’t rely on a single signal, because DNS failures and CDN issues can look similar from one vantage point.
Here’s a practical framework you can build around:
1) What to detect
You’ll want to distinguish these cases:
- Authoritative DNS failure
- zone doesn’t answer
- SERVFAIL / NXDOMAIN / timeout
- broken delegation / lame NS
- Recursive resolver issue
- public resolvers disagree
- resolution works from one region but not another
- CDN edge outage / degradation
- DNS resolves, but HTTP/TLS fails or is slow
- some POPs fail, others work
- specific geography affected
- Origin outage masked by CDN
- CDN is up, origin is down, causing 5xx or stale content issues
- Path/network issue
- BGP / peering / regional ISP problem
- DNS and CDN both appear down from certain probes only
2) Core signals to collect
DNS checks
For each hostname, query multiple resolver types:
- authoritative NS directly
- public resolvers:
- 1.1.1.1
- 8.8.8.8
- 9.9.9.9
- optionally local recursive resolver
- query types:
- A / AAAA
- CNAME chain
- NS / SOA for zone health
- DNSSEC-related checks if enabled
Track:
- success/failure
- latency
- response code
- TTL consistency
- answer consistency across resolvers
CDN / HTTP checks
From multiple regions:
- HTTP GET/HEAD to the origin-facing hostname
- measure:
- connect time
- TLS handshake time
- TTFB
- total response time
- HTTP status codes
- verify:
- certificate validity
- hostname mismatch
- redirect loops
- expected content hash / marker string
Network/route checks
Useful if you want to reduce false positives:
- traceroute/mtr or lightweight TCP path metrics
- DNS query path from multiple PoPs
- BGP visibility via external feeds if you can integrate them
3) Detection logic
A good simple model:
DNS outage likely if:
- authoritative queries fail from most probes
- multiple recursive resolvers fail or return SERVFAIL
- failures persist beyond a short transient window
- at least one independent probe region agrees
CDN outage likely if:
- DNS is healthy but HTTP/TLS fails or degrades
- failures are regional and correlate with CDN edge locations
- DNS answers remain stable while HTTP status codes jump to 5xx or timeout
- latency spikes are concentrated in one or more regions
Confidence scoring
Instead of binary up/down, assign a score:
- DNS health score
- CDN health score
- Regional anomaly score
- Confidence based on number of agreeing probes
This helps avoid false alarms.
4) Multi-region probing is essential
If you only probe from one place, you’ll confuse:
- local ISP issues
- resolver problems
- CDN geo-routing quirks
Minimum recommended setup:
- 3–5 probe regions
- at least 2 recursive resolvers
- authoritative query from direct NS when possible
5) Alerting strategy
Alert on:
- sustained failure across multiple probes
- consensus failure for DNS
- divergence between regions
- sudden shift in response codes or latency
Avoid alerting on:
- one-off packet loss
- one resolver only
- a single region only
Use:
- rolling windows
- quorum-based thresholds
- hysteresis to avoid flapping
6) Data model suggestion
A simple event schema:
targettimestampprobe_regioncheck_type(dns_auth,dns_recursive,http,tls)resolverstatusrcodelatency_msip_answercdn_popif identifiableerror_categoryconfidence
7) Nice features to add
- DNS answer diffing
- compare answers across resolvers/regions
- CNAME chain inspection
- helps identify CDN provider changes or failures
- ASN / POP mapping
- infer which CDN edge/region is impacted
- Historical baselines
- detect anomalies versus normal latency patterns
- Synthetic content validation
- fetch a known endpoint and verify body checksum
8) Common pitfalls
- Treating NXDOMAIN and SERVFAIL the same
- Assuming one resolver represents the internet
- Using only ping/ICMP
- Not checking authoritative NS directly
- Ignoring IPv6
- Not accounting for cached DNS answers
9) Suggested architecture
- Probe agents in multiple regions
- Central aggregator
- Metrics store for time series
- Rules engine for incident classification
- Alerting layer for paging/webhooks
- Optional machine learning later for anomaly detection
10) Simple MVP
If you want a first version quickly:
- Pick target domains
- Query A/AAAA via 3 public resolvers + authoritative NS
- Fetch HTTPS from 3 regions
- Compare response status, latency, and IP answers
- Alert if:
- DNS fails in 2+ resolvers for 5 minutes
- HTTP fails in 2+ regions while DNS is healthy
If you want, I can help you design:
- a system architecture
- a database schema
- a detection algorithm
- or a minimal Python/Go prototype.