Do I need to deduplicate records before loading to the warehouse?

Usually, yes — but it depends on your warehouse design and loading pattern.

When you should deduplicate before loading

You should deduplicate upstream if:

• The warehouse tables are meant to represent unique business entities or one row per key
• Your warehouse/load process is append-only
• Downstream users expect clean, analytics-ready data
• Duplicates would create problems in joins, aggregates, or BI dashboards

When you might not need to deduplicate first

You may skip pre-deduplication if:

• You intentionally store raw, immutable landing data
• You use the warehouse to keep history/events, where repeated records are valid
• Your ELT process handles deduplication later with:
  • MERGE
  • ROW_NUMBER() OVER (...)
  • unique constraints / surrogate keys
  • dbt models or similar transformations

Common practice

A common pattern is:

1. Load raw data as-is into a staging/landing layer
2. Deduplicate and transform into curated warehouse tables
3. Keep both:
   • raw data for audit/replay
   • clean data for reporting

Important caveat

If you don’t deduplicate anywhere, duplicate rows can:

• inflate counts and revenue
• break slowly changing dimension logic
• cause duplicate dimension/member records
• make incremental loads messy

If you want, I can help you decide based on your specific source, warehouse, and load type.