Prompt

How do I set up retries and dead-letter handling for background jobs?

Technology / Cloud Infrastructure1 observationLast seen Oct 24, 2026

Latest observation

Oct 24, 2026OpenAI APIWeb search: off

A solid retry + dead-letter setup for background jobs usually has 4 parts:

  1. Transient failure retries
  2. A maximum retry limit
  3. Dead-letter queue / table for poison jobs
  4. Visibility and alerting

Here’s a practical pattern you can use.


1) Define which failures are retryable

Retry only failures that are likely temporary:

  • network timeouts
  • rate limits / 429s
  • database connection hiccups
  • upstream service 5xx errors

Do not retry things like:

  • invalid input
  • missing required data
  • permission denied
  • schema/logic errors that will never succeed without code/data changes

A good rule: if retrying won’t change the outcome, don’t retry.


2) Use exponential backoff with jitter

Avoid retrying immediately and overloading the system.

Example schedule:

  • attempt 1: immediately
  • attempt 2: after 1s
  • attempt 3: after 5s
  • attempt 4: after 30s
  • attempt 5: after 2m

Add jitter so many failed jobs don’t retry at the same moment.

Typical formula:

  • delay = base * 2^attempt + random(0, jitter)

3) Track attempt count and job state

Store metadata with each job:

  • job_id
  • payload
  • attempt_count
  • max_attempts
  • status (queued, running, retrying, failed, dead_lettered, completed)
  • last_error
  • next_run_at
  • timestamps

This can live in:

  • a queue system that supports retries
  • a database table
  • both, if you need auditability

4) Move poison jobs to a dead-letter queue

If a job exceeds max_attempts, move it to a dead-letter queue/table instead of retrying forever.

Dead-letter record should include:

  • original payload
  • failure reason
  • stack trace / error class
  • number of attempts
  • timestamps
  • worker/job version
  • any correlation IDs

This lets you:

  • inspect why it failed
  • reprocess manually after fixing the issue
  • avoid blocking the queue

5) Separate retryable and non-retryable failures

In your worker, classify failures:

  • Non-retryable → dead-letter immediately
  • Retryable → reschedule with backoff
  • Unknown → often retry a few times, then dead-letter

Pseudo-logic:

processJob(job):
  try:
    run job
    mark completed
  catch error:
    if not retryable(error):
      move to dead-letter
    else if job.attempts >= job.max_attempts:
      move to dead-letter
    else:
      job.attempts += 1
      job.next_run_at = now + backoff(job.attempts)
      requeue job

6) Use idempotency

Retries can cause duplicate execution, so make job handlers idempotent.

Examples:

  • use unique operation IDs
  • check if an order was already charged before charging again
  • use upserts instead of inserts where appropriate
  • store “already processed” markers

This is essential if a worker crashes after doing the work but before marking the job complete.


7) Add visibility and alerting

Monitor:

  • retry rate
  • dead-letter count
  • age of oldest queued job
  • jobs failing repeatedly
  • worker crashes
  • queue depth

Alert when:

  • dead-letter queue grows
  • a specific job type spikes in failures
  • jobs exceed expected processing time

8) Consider circuit breakers for upstream dependencies

If many jobs fail because one external service is down, retries can make things worse.

Use a circuit breaker or pause job intake temporarily when:

  • downstream service is unavailable
  • error rate crosses a threshold

This prevents retry storms.


9) Reprocessing dead-lettered jobs

Provide a controlled way to replay dead-letter jobs:

  • after fixing the bug
  • after correcting bad data
  • after the dependency recovers

Best practice:

  • replay manually or in small batches
  • keep original dead-letter history
  • ensure idempotency before replaying

10) Example implementation pattern

If you’re building this yourself, a simple database-backed setup is:

Jobs table

  • id
  • payload
  • status
  • attempts
  • max_attempts
  • next_run_at
  • last_error
  • created_at
  • updated_at

Dead-letter table

  • same fields as jobs
  • dead_letter_reason
  • failed_at

Worker loop:

  1. fetch due jobs
  2. lock job
  3. process
  4. on success, mark completed
  5. on retryable failure, increment attempts and set next_run_at
  6. on max attempts or non-retryable failure, insert into dead-letter table

If you’re using a queue library or platform

Many systems already support this:

  • AWS SQS: visibility timeout + redrive policy to DLQ
  • RabbitMQ: dead-letter exchanges
  • Kafka: retry topics + DLQ topic
  • Sidekiq: retries + dead set
  • Celery: retries + reject / dead-letter patterns
  • BullMQ: attempts + backoff + failed jobs handling

If you tell me your stack, I can show the exact setup for it.


Recommended baseline policy

A good general default:

  • retry on transient errors only
  • exponential backoff with jitter
  • max 5 attempts
  • dead-letter after final failure
  • idempotent job handlers
  • alerts on DLQ growth

If you want, I can also give you:

  1. a sample implementation in Node.js / Python / Go / .NET, or
  2. the best-practice setup for a specific queue system like SQS, RabbitMQ, Kafka, Celery, or BullMQ.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.