Prompt
I need a durable job system with retries, scheduling, and dead-letter queues, what are my options?
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
If you want a durable job system with retries, scheduling, and dead-letter queues (DLQs), you generally have four broad options:
1) Use a managed queue / job service
Best if you want less operational burden.
Examples
- AWS
- SQS + Lambda/ECS workers for retries and durability
- EventBridge Scheduler or Step Functions for scheduling
- DLQ support built into SQS
- GCP
- Cloud Tasks for retries, scheduling, rate limiting
- Pub/Sub + dead-letter topics
- Azure
- Service Bus queues/topics with DLQ
- Azure Functions for workers/scheduling
Pros
- Durable and managed
- Built-in retries / DLQ
- Scales well
- Less infrastructure to run
Cons
- Scheduling semantics can be limited depending on service
- More “queue + worker” than full job orchestration
- Vendor-specific
2) Use a workflow / job orchestration engine
Best if jobs have multiple steps, dependencies, or long-running processes.
Examples
- Temporal
- AWS Step Functions
- Azure Durable Functions
- Google Workflows
- Argo Workflows (Kubernetes)
Pros
- Strong durability and retry semantics
- Great for complex workflows
- Visibility into state, progress, failures
- Timers/scheduling often built in
Cons
- More conceptual overhead
- Can be heavier than a simple job queue
- Sometimes overkill for single-step jobs
3) Use a queue library / framework on top of Redis/Postgres
Best if you want to run your own app-managed job system.
Examples
- BullMQ (Node.js, Redis)
- Sidekiq (Ruby, Redis)
- Celery (Python, usually Redis/RabbitMQ; can use DB-backed setups)
- Hangfire (.NET, SQL Server/Redis)
- RQ (Python, Redis)
- Quartz (Java)
Pros
- Good developer ergonomics
- Retries, delayed jobs, scheduling often supported
- Easy to integrate into your app
- Can be self-hosted
Cons
- You own reliability/ops
- Durability depends on backend and configuration
- DLQ support varies; sometimes implemented manually
- Horizontal scaling and failover need care
4) Build it on a durable datastore directly
Best if you need a custom domain-specific system and are willing to build more.
Common approach
- Store jobs in Postgres or MySQL
- Use:
- status fields
- leasing/locking
- attempt counters
- next-run timestamps
- retry/backoff logic
- failed jobs table as DLQ
Pros
- Maximum flexibility
- Strong consistency if built carefully
- No extra queue infrastructure required
Cons
- Most engineering effort
- Easy to get subtle race conditions wrong
- Needs careful design for scale and recovery
How to think about the features
Retries
You’ll want:
- exponential backoff
- max attempts
- retryable vs non-retryable errors
- jitter to avoid thundering herds
Scheduling
Typical needs:
- one-time delayed jobs
- recurring jobs
- cron-like scheduling
- timezone handling
- missed-run recovery
Dead-letter queues
You want:
- capture permanently failed jobs
- preserve error history / attempts
- ability to inspect and replay
- manual or automated reprocessing
Practical recommendations
Choose managed services if:
- you want to move fast
- ops should be minimal
- your workload is straightforward
Choose Temporal / Step Functions if:
- jobs are multi-step or long-running
- you need strong visibility and retries
- you care about workflow correctness more than simplicity
Choose BullMQ / Sidekiq / Celery if:
- you’re building a product app
- you want familiar app-level job processing
- you’re okay running Redis or RabbitMQ
Build your own on Postgres if:
- you need custom semantics
- you already rely heavily on the DB
- you have engineering time for correctness and scaling
My quick shortlist
If you want a solid default:
- Temporal for complex durable workflows
- BullMQ / Sidekiq / Celery for app background jobs
- SQS + DLQ or Cloud Tasks if you want managed queueing
- Postgres-based queue if you want fewer moving parts and can engineer it carefully
If you want, I can also give you:
- a comparison table,
- recommendations by language/platform,
- or a reference architecture for your specific use case.