Prompt
unhappy with Redis queues for durable background jobs
Latest observation
If you’re unhappy with Redis-backed queues for durable background jobs, that’s a common pain point. Redis is great for speed and simple buffering, but it’s often the wrong choice when you need durability, strong delivery guarantees, auditing, retries, and operational confidence.
Why Redis queues often disappoint
Common issues:
- Data loss risk if Redis restarts, evicts keys, or persistence isn’t configured perfectly
- Weak durability semantics compared to a real job store
- At-least-once delivery quirks can still lead to duplicate processing
- Operational fragility when queues, locks, retries, and visibility timeouts are homegrown
- Harder recovery after crashes or worker failures
Better options for durable background jobs
1) Database-backed job tables
If you already use PostgreSQL or MySQL, this is often the simplest durable alternative.
How it works:
- Store jobs in a table
- Workers poll or use
SELECT ... FOR UPDATE SKIP LOCKED - Update status, retries, next run time, and result metadata
Pros:
- Strong durability
- Easy auditing/debugging
- No extra infrastructure if you already have a DB
- Good fit for moderate throughput
Cons:
- Can become noisy under very high job volume
- Polling design needs care
- More app logic than a managed queue
Best for: most business apps, especially if correctness matters more than raw throughput.
2) Dedicated durable queue systems
If you want a real queue but not Redis, look at systems built for persistence.
RabbitMQ
- Mature message broker
- Durable queues, acknowledgements, dead-letter exchanges
- Good for task distribution
Pros: robust messaging semantics, flexible routing
Cons: more operational complexity than Redis, not a job store by default
Apache Kafka
- Great for event streams and replayability
- Very durable and scalable
Pros: excellent durability and throughput
Cons: not naturally a “job queue” unless you build that layer; retries and per-job state are more complex
Best for: high-throughput event-driven systems, not simple background tasks.
3) Workflow engines / job orchestration platforms
If your background jobs are actually multi-step workflows, use a workflow engine.
Temporal
- Designed for durable execution
- Built-in retries, timers, state, resumability
- Excellent for long-running business processes
Pros: strongest model for durable background work
Cons: different programming model; learning curve
Sidekiq alternatives with persistent backends
Depending on language/ecosystem:
- Celery with RabbitMQ/DB-backed transports
- Resque-like systems with more durable stores
- Native DB job runners in your framework
Best for: business processes, sagas, long-running tasks, retries with state.
4) Managed cloud queues
If you want durability without running your own broker:
- AWS SQS
- Google Cloud Tasks / Pub/Sub
- Azure Service Bus
Pros:
- Highly durable
- Managed scaling
- Built-in retries and DLQs
Cons:
- Vendor-specific semantics
- Sometimes less ergonomic than a local queue
Best for: teams that prefer managed infrastructure.
If you want the simplest “better than Redis” option
For many teams, the best upgrade is:
PostgreSQL job table + worker process + SKIP LOCKED + retry metadata
This gives:
- durable storage
- easy inspection
- transactional job creation
- reasonable reliability
- minimal infrastructure changes
Important design features for durable jobs
No matter what system you choose, make sure you have:
- Idempotency keys to handle duplicates
- Retries with exponential backoff
- Dead-lettering for poison jobs
- Visibility timeouts / lease renewal or equivalent
- Job state transitions: pending, running, succeeded, failed, retrying
- Observability: metrics, logs, traces
- Recovery logic for stuck or orphaned jobs
Practical recommendation
- If you have moderate volume and want durability: use your primary database
- If you need messaging at scale: use RabbitMQ or managed queues
- If you need reliable workflows: use Temporal
- If you need event replay and streaming: use Kafka
If you want, I can help you choose based on your stack and constraints, for example:
- language/framework
- job volume
- retry requirements
- need for scheduling / cron
- whether tasks are idempotent
- whether you need exactly-once-ish behavior