Prompt
How do I set up retries and dead-letter handling for background jobs?
Latest observation
A solid retry + dead-letter setup for background jobs usually has 4 parts:
- Transient failure retries
- A maximum retry limit
- Dead-letter queue / table for poison jobs
- Visibility and alerting
Here’s a practical pattern you can use.
1) Define which failures are retryable
Retry only failures that are likely temporary:
- network timeouts
- rate limits / 429s
- database connection hiccups
- upstream service 5xx errors
Do not retry things like:
- invalid input
- missing required data
- permission denied
- schema/logic errors that will never succeed without code/data changes
A good rule: if retrying won’t change the outcome, don’t retry.
2) Use exponential backoff with jitter
Avoid retrying immediately and overloading the system.
Example schedule:
- attempt 1: immediately
- attempt 2: after 1s
- attempt 3: after 5s
- attempt 4: after 30s
- attempt 5: after 2m
Add jitter so many failed jobs don’t retry at the same moment.
Typical formula:
delay = base * 2^attempt + random(0, jitter)
3) Track attempt count and job state
Store metadata with each job:
job_idpayloadattempt_countmax_attemptsstatus(queued,running,retrying,failed,dead_lettered,completed)last_errornext_run_at- timestamps
This can live in:
- a queue system that supports retries
- a database table
- both, if you need auditability
4) Move poison jobs to a dead-letter queue
If a job exceeds max_attempts, move it to a dead-letter queue/table instead of retrying forever.
Dead-letter record should include:
- original payload
- failure reason
- stack trace / error class
- number of attempts
- timestamps
- worker/job version
- any correlation IDs
This lets you:
- inspect why it failed
- reprocess manually after fixing the issue
- avoid blocking the queue
5) Separate retryable and non-retryable failures
In your worker, classify failures:
- Non-retryable → dead-letter immediately
- Retryable → reschedule with backoff
- Unknown → often retry a few times, then dead-letter
Pseudo-logic:
processJob(job):
try:
run job
mark completed
catch error:
if not retryable(error):
move to dead-letter
else if job.attempts >= job.max_attempts:
move to dead-letter
else:
job.attempts += 1
job.next_run_at = now + backoff(job.attempts)
requeue job
6) Use idempotency
Retries can cause duplicate execution, so make job handlers idempotent.
Examples:
- use unique operation IDs
- check if an order was already charged before charging again
- use upserts instead of inserts where appropriate
- store “already processed” markers
This is essential if a worker crashes after doing the work but before marking the job complete.
7) Add visibility and alerting
Monitor:
- retry rate
- dead-letter count
- age of oldest queued job
- jobs failing repeatedly
- worker crashes
- queue depth
Alert when:
- dead-letter queue grows
- a specific job type spikes in failures
- jobs exceed expected processing time
8) Consider circuit breakers for upstream dependencies
If many jobs fail because one external service is down, retries can make things worse.
Use a circuit breaker or pause job intake temporarily when:
- downstream service is unavailable
- error rate crosses a threshold
This prevents retry storms.
9) Reprocessing dead-lettered jobs
Provide a controlled way to replay dead-letter jobs:
- after fixing the bug
- after correcting bad data
- after the dependency recovers
Best practice:
- replay manually or in small batches
- keep original dead-letter history
- ensure idempotency before replaying
10) Example implementation pattern
If you’re building this yourself, a simple database-backed setup is:
Jobs table
idpayloadstatusattemptsmax_attemptsnext_run_atlast_errorcreated_atupdated_at
Dead-letter table
- same fields as jobs
dead_letter_reasonfailed_at
Worker loop:
- fetch due jobs
- lock job
- process
- on success, mark completed
- on retryable failure, increment attempts and set
next_run_at - on max attempts or non-retryable failure, insert into dead-letter table
If you’re using a queue library or platform
Many systems already support this:
- AWS SQS: visibility timeout + redrive policy to DLQ
- RabbitMQ: dead-letter exchanges
- Kafka: retry topics + DLQ topic
- Sidekiq: retries + dead set
- Celery: retries + reject / dead-letter patterns
- BullMQ: attempts + backoff + failed jobs handling
If you tell me your stack, I can show the exact setup for it.
Recommended baseline policy
A good general default:
- retry on transient errors only
- exponential backoff with jitter
- max 5 attempts
- dead-letter after final failure
- idempotent job handlers
- alerts on DLQ growth
If you want, I can also give you:
- a sample implementation in Node.js / Python / Go / .NET, or
- the best-practice setup for a specific queue system like SQS, RabbitMQ, Kafka, Celery, or BullMQ.