Why do my pipeline retries create duplicate events?

Pipeline retries usually create duplicate events because the retry is re-running a step that is not idempotent.

What’s happening

If a job:

1. reads an input event,
2. publishes/records a new event,
3. then fails before the pipeline marks the step as complete,

a retry will run the same code again and publish the same event a second time.

This is especially common when:

• the pipeline retry happens at the job/task level,
• the output event is sent to a queue/topic/log before the step is committed,
• there’s no deduplication key or idempotency check,
• external systems are involved and the pipeline can’t “roll back” the first attempt.

Why retries are tricky

Retries are designed for at-least-once execution, not exactly-once execution.
That means the system guarantees the work will be attempted again, but not that side effects happen only once.

Common causes

• Task timeout after success: the event was already emitted, but the worker didn’t report success in time.
• Network error on ack/commit: the event was published, but the pipeline didn’t receive confirmation.
• Partial failure: one part of the job completed, then an error triggered retry.
• No deduplication: each retry generates a fresh event ID, so downstream sees them as distinct.

How to prevent duplicates

• Make the step idempotent.
• Use a stable deduplication key or event ID based on the original input.
• Store processed event IDs in a database/cache before emitting.
• Use a transactional outbox pattern if publishing to an external broker.
• Have downstream consumers ignore duplicates using event IDs/versioning.
• If your platform supports it, enable exactly-once semantics, though those are often limited.

Rule of thumb

If a retry can safely happen, then the step must be written so repeating it doesn’t create new side effects.

If you want, I can help you diagnose this for a specific system like Airflow, Kafka, Temporal, AWS Step Functions, or a CI/CD pipeline.