What should I use for backfills and retries in data pipelines?

A good rule of thumb:

• Backfills = use your orchestration / workflow system
  Examples: Airflow, Dagster, Prefect, Temporal, Argo Workflows, dbt jobs, etc.
• Retries = use application-level retries for transient failures plus orchestrator retries for task-level reruns
  Examples: retry API calls, database timeouts, object-store reads; let the orchestrator retry failed tasks with limits and backoff.

Why split them?

• Backfills usually mean “recompute data for a past time window or partition.”
  • They need awareness of time ranges, partitions, dependencies, and idempotency.
  • Orchestrators are built to manage these at scale.
• Retries usually mean “a task failed due to a transient issue; try again.”
  • These should be small, bounded, and automatic.
  • If a retry is needed because data itself is missing or wrong, that’s often a backfill, not a retry.

Practical guidance

Use the orchestrator for backfills when you need:

• Reprocessing by date/partition
• Dependency ordering across many jobs
• Visibility into what was rerun
• Concurrency limits and scheduling
• Partial reruns and catch-up

Use code-level retries for:

• Network blips
• Rate limits
• Temporary service outages
• Lock contention
• Short-lived warehouse/query failures

Avoid:

• Using retries to “fix” logic bugs or bad upstream data
• Running huge backfills as ad hoc scripts with no tracking
• Making tasks non-idempotent

Important design principle

Make your pipeline tasks:

• Idempotent
• Partitioned by time or data slice
• Safe to rerun

That way:

• Retries are safe for transient failures
• Backfills are straightforward when business logic changes or late data arrives

A simple decision rule

• If the question is “did this job fail?” → retry
• If the question is “should we recompute historical data?” → backfill

If you want, I can also give you:

1. a tool-by-tool recommendation (Airflow vs Dagster vs dbt vs Temporal), or
2. a reference design for how to implement backfills/retries cleanly.