Why are there so many duplicate records in this data feed?

Duplicate records in a data feed usually come from one of a few common issues:

1. Source system re-sending the same event

   • The upstream system may retry failed deliveries without idempotency, so the same record gets sent more than once.

2. Batch reprocessing

   • A job may be rerun after a failure, and prior output wasn’t cleared or deduplicated.

3. Join/query logic creating multiplicative rows

   • If the feed is built from SQL joins, one-to-many joins can unintentionally duplicate records.

4. Window overlap in incremental loads

   • If the feed extracts “new records since last run” using timestamps, overlapping time windows can re-include already sent records.

5. Consumer-side deduplication missing

   • The feed may be designed to allow duplicates, expecting the downstream system to dedupe by key + timestamp/version.

6. Late updates or CDC events

   • Change data capture feeds can emit multiple rows for the same business entity as it changes over time; these may look like duplicates if you’re expecting only the latest state.

7. Key mismatch

   • Records may look identical except for hidden fields like load timestamp, source batch ID, or sequence number.

If you want, I can help you narrow it down by looking at:

• a few example duplicate rows,
• the feed format,
• the primary key/business key,
• and how the feed is generated (SQL, ETL tool, CDC, API, etc.).