pipeline freshness alerts

Pipeline freshness alerts are notifications that tell you when your data pipelines are no longer producing data as expected, so you can catch delays or failures before they impact downstream reporting or models.

Typical freshness alert setup:

• Define expected data arrival time: e.g. “table orders should update every 15 minutes.”
• Monitor last successful update / ingestion time
• Set a threshold: e.g. alert if data is older than 30 minutes
• Trigger notifications via email, Slack, PagerDuty, etc.
• Include context in the alert: pipeline name, last run time, downstream impact, and link to logs

Common alert types:

• Late data: pipeline succeeded, but data arrived too late
• Stale data: no new data within the expected window
• Missed run: scheduled pipeline didn’t execute
• Partial freshness issue: only some partitions or sources are delayed

Good practices:

• Alert on business-critical datasets, not every table
• Use grace periods to avoid noisy alerts
• Separate warning and critical thresholds
• Deduplicate alerts and route by severity
• Track MTTA/MTTR to improve response time

If you want, I can also help with:

1. a SQL-based freshness check
2. a dbt freshness configuration
3. a generic alerting design for Airflow/Dagster/Prefect
4. example Slack/PagerDuty alert payloads