Prompt
Why do my app syncs keep duplicating records
Latest observation
Oct 19, 2026
OpenAI APIWeb search: off
Duplicate records during app syncs usually come from one of a few root causes:
Common reasons
-
No stable unique ID
- If each sync creates records based on a name/email/title instead of a true unique identifier, the same item can be inserted again.
- Fix: use a permanent ID from the source system.
-
Sync is not idempotent
- The same payload gets processed more than once, but your app treats every run as “new.”
- Fix: make the sync safe to repeat by checking whether the record already exists before inserting.
-
Race conditions / concurrent syncs
- Two sync jobs run at the same time and both decide the record doesn’t exist yet.
- Fix: lock the sync process or use database-level unique constraints.
-
Pagination or cursor bugs
- The sync may be re-reading the same page/batch repeatedly because the cursor/offset isn’t updated correctly.
- Fix: verify checkpoint handling and pagination logic.
-
Retry logic without deduping
- If a request times out, the app retries, but the original request actually succeeded.
- Fix: use request IDs, idempotency keys, or dedupe keys.
-
Partial updates treated as inserts
- Missing fields during incremental sync cause the system to think it’s a new entity.
- Fix: merge by ID and handle upserts rather than blind inserts.
-
Webhook + polling both creating records
- One source event arrives via webhook, and the polling sync creates it again.
- Fix: ensure both paths share the same deduplication logic.
Best fixes
- Add a unique constraint in the database on the source identifier.
- Use upsert instead of insert.
- Store a last synced cursor/timestamp correctly.
- Add deduplication checks before write.
- Prevent multiple sync jobs from running simultaneously.
- Log sync runs with record IDs so you can see where duplicates start.
Quick diagnostic questions
- Are duplicates exact copies or slightly different?
- Do they happen after retries, manual re-syncs, or scheduled syncs?
- Is there a stable external ID available?
- Do duplicates come from one integration or multiple sources?
If you want, I can help you debug this systematically if you share:
- your sync flow,
- the database schema,
- and a sample of one duplicated record pair.