Do I need to backfill old data when switching pipelines?

Not always — it depends on what the new pipeline is responsible for and whether downstream users need historical continuity.

You usually do need to backfill if:

• The new pipeline replaces an existing source of truth and downstream tables/dashboards expect a full history.
• You’re changing logic that affects historical metrics, and you need past data to match the new definitions.
• Auditing, reporting, or ML features rely on historical records being present in the new system.
• The new pipeline writes to a fresh table or schema that starts empty, but consumers query it as if it has complete data.

You may not need to backfill if:

• The new pipeline only handles future data, and old data can remain in the old system.
• Historical data is no longer used or is acceptable to access from the legacy pipeline.
• The cost/time/risk of backfilling outweighs the benefit.
• You can clearly version data and communicate a cutoff date to consumers.

Common compromise

A lot of teams do one of these:

1. No backfill: start the new pipeline from “go-live” forward.
2. Partial backfill: load recent history only, like the last 30/90/365 days.
3. Full backfill: only if historical completeness is important.

Questions to decide

• Do consumers need a seamless historical table?
• Is the new pipeline changing schema or business logic?
• Will users compare old vs. new periods?
• Can you tolerate a gap at the cutover point?
• How expensive is the backfill?

If you want, I can help you decide based on your specific setup — for example, whether this is a DB migration, an ETL/ELT change, or a streaming pipeline cutover.