Prompt
What should I use for durable background jobs in production?
Latest observation
For durable background jobs in production, you generally want a system that gives you:
- persistence: jobs survive restarts/crashes
- retries: automatic retry on failure
- idempotency support: safe re-processing
- observability: easy to inspect status/failures
- scalability: workers can run horizontally
Good choices
1. Managed job queue / workflow service
Best if you want reliability with less ops.
Examples:
- Temporal — excellent for long-running, durable workflows
- AWS SQS + worker fleet — simple and robust
- Google Cloud Tasks
- Azure Queue Storage / Service Bus
- Cloudflare Queues (depending on workload)
Use this if: you want production-grade durability and fewer self-managed moving parts.
2. Message queue + worker system
Classic and widely used.
Examples:
- RabbitMQ
- Kafka (more for event streaming than job queues, but can work)
- Redis-based queues like BullMQ / RQ / Sidekiq-style systems
Use this if: you want control, already run the infra, and your jobs are relatively straightforward.
Note: Redis-based queues can be durable enough in many cases, but durability depends heavily on Redis persistence/config and operational discipline.
3. Database-backed job queue
Good for simpler systems and fewer dependencies.
Examples:
- PostgreSQL queue implementations
SELECT ... FOR UPDATE SKIP LOCKEDjob tables
Use this if: you already rely on Postgres and want a simple, reliable solution for moderate scale.
Tradeoff: not as feature-rich or scalable as dedicated queue/workflow systems.
My practical recommendation
If you want the safest default:
- Temporal for complex workflows, retries, timers, orchestration
- SQS + workers for simple background jobs
If you’re on a small/medium app:
- Postgres-backed job queue is often a great pragmatic choice
- Or Redis + a proven library if you accept slightly more operational risk
Avoid relying on:
- in-memory queues
- cron-only approaches for critical jobs
- “fire-and-forget” background threads in web servers
Those are not durable enough for production.
Rule of thumb
- Long-running, multi-step, retry-heavy jobs → Temporal
- Simple async task processing → SQS / RabbitMQ / Redis queue
- Already have Postgres and want minimal infra → DB-backed queue
If you tell me your stack (language/framework/cloud) and the kind of jobs you need to run, I can recommend a specific setup.