How do I monitor freshness and failures across pipelines?

To monitor freshness and failures across pipelines, you want two things:

1. Data recency signals — is the output data up to date?
2. Run health signals — did the pipeline execute successfully?

A good monitoring setup usually combines both into a single view.

1) Define freshness expectations

For each pipeline or dataset, define:

• Expected update frequency: hourly, daily, weekly, etc.
• SLA/SLO threshold: e.g. “data must be no older than 2 hours”
• Criticality: which pipelines are most important

Example:

• Orders pipeline: fresh within 30 minutes
• Finance pipeline: fresh within 24 hours
• Marketing pipeline: fresh within 6 hours

2) Track pipeline execution status

Collect metadata from each run:

• start time / end time
• success/failure status
• retry count
• error message
• upstream dependencies

This lets you detect:

• hard failures
• repeated flaky failures
• stalled pipelines with no recent runs

3) Track data freshness directly

Use dataset-level checks, such as:

• latest partition date
• max event timestamp in the table
• time since last successful load
• row count anomalies combined with freshness checks

Common freshness metric:

• Freshness lag = now() - max(data_timestamp)
• Alert if lag exceeds threshold

4) Centralize monitoring

Best practice is to send metrics/events to a central system, such as:

• observability platform
• metrics store
• warehouse monitoring table
• alerting service

Typical dashboards:

• pipeline success rate over time
• last successful run per pipeline
• freshness lag per dataset
• failed runs with error breakdown
• upstream dependency health

5) Alert on both symptoms and impact

Create alerts for:

• Failure alerts: a run fails
• Staleness alerts: no fresh data by expected time
• Silent failure alerts: pipeline “succeeds” but freshness doesn’t improve
• Missing data alerts: row counts or partitions stop advancing

Use alert routing based on severity:

• Slack/email for warnings
• pager/on-call for critical production data delays

6) Add dependency-aware checks

Pipelines often fail because upstreams are late or broken. Monitor:

• upstream freshness
• dependency graph status
• downstream blast radius

This helps distinguish:

• root-cause failures
• cascading failures

7) Build standardized metadata for all pipelines

Use a common schema like:

• pipeline_name
• dataset_name
• run_id
• status
• started_at
• ended_at
• last_success_at
• expected_interval_minutes
• freshness_lag_minutes
• owner
• severity

This makes cross-pipeline monitoring much easier.

8) Useful KPIs

Some good fleet-wide metrics:

• % of pipelines fresh on time
• mean time to detect failure
• mean time to recover
• failure rate by team/system
• number of stale critical datasets
• top recurring failure causes

9) Practical implementation pattern

A common setup:

• Pipeline emits run events on start/success/failure
• Data validation job updates freshness metrics
• Monitoring job computes lag and checks thresholds
• Alerting service notifies owners

---

If you want, I can also give you:

1. a reference architecture,
2. a SQL example for freshness checks, or
3. a dashboard/alert design for Airflow/dbt/Prefect/Dagster.