Prompt
my worker is processing the same job twice, what am I missing?
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
A worker processing the same job twice usually means the job is not being acknowledged/removed atomically or the queue system is allowing at-least-once delivery. Common causes:
Most common reasons
-
No ack / delete after success
- The job gets fetched, processed, but never marked complete.
- On timeout / reconnect / restart, it becomes visible again.
-
Visibility timeout too short
- The job takes longer than the queue’s invisibility window.
- Another worker can receive it again before the first finishes.
-
Worker crash or restart
- If the worker dies after processing but before acking, the job is retried.
-
Multiple consumers with no dedupe
- Queue guarantees are often “at least once,” not exactly once.
- You need idempotent handlers or deduplication keys.
-
Ack happens too early
- The worker removes/acks the job before the actual side effect completes, then fails mid-way.
-
Job gets re-enqueued by retry logic
- Automatic retries, timeouts, or exception handling may be duplicating work.
What to check
- Does your queue use ack/delete semantics?
- Is there a visibility timeout / lease / lock duration?
- Are you retrying on errors or timeouts?
- Does the handler take longer than the queue’s lock?
- Are you running multiple instances of the worker?
- Do you have a unique job ID and dedupe store?
Practical fix
- Make the job handler idempotent.
- Ensure the job is acked only after successful completion.
- Increase the visibility timeout/lease to exceed max processing time.
- Add a distributed lock or dedupe check keyed by job ID.
- Log the lifecycle:
fetched -> started -> finished -> acked.
If you tell me which queue system you’re using (e.g. SQS, RabbitMQ, Celery, Sidekiq, BullMQ, Bull, Redis, Hangfire, etc.) and show the worker code, I can point to the exact cause.