Prompt
unhappy with Redis queue jobs disappearing after restart
Latest observation
If Redis-backed queue jobs are “disappearing after restart,” that usually means the jobs were only in memory or were acknowledged/removed before they were actually completed. The fix depends on which queue system you’re using, but here are the common causes and remedies.
Common reasons
1. Redis is not persisting data
If Redis restarts and loses queued jobs, persistence may be off.
Check:
- RDB snapshots enabled
- AOF enabled for stronger durability
Redis config examples:
save 900 1
save 300 10
save 60 10000
appendonly yes
If you’re using Redis as a queue store, AOF is usually the safer option.
2. Your queue library removes jobs too early
Some libraries mark jobs as “done” or remove them from the queue before the worker fully finishes, especially if they use:
- at-most-once semantics
- non-atomic pop patterns
- no retry/visibility timeout handling
If the worker crashes after popping a job, it may be lost.
Look for settings like:
removeOnCompleteremoveOnFailacknowledgeautocleanupdelete_after_success
3. Using the wrong Redis data structure
A plain Redis LIST with LPOP is fragile if workers crash after popping.
Safer patterns:
- Reliable queue pattern using
BRPOPLPUSH/RPOPLPUSH - Use a queue library with:
- retries
- job state tracking
- dead-letter queue
- delayed jobs support
4. Container restart is wiping Redis volume
If Redis runs in Docker/Kubernetes and no persistent volume is mounted, a restart may create a fresh instance.
Check:
- Docker volume attached to
/data - Kubernetes PVC mounted to Redis data directory
5. Job metadata is stored elsewhere
Some systems store jobs in Redis but results/state in another DB. After restart, it may look like jobs disappeared when only the processing state was lost.
What to do
If you manage Redis directly
- Enable persistence (
appendonly yes) - Mount persistent storage
- Avoid using plain
LPOPfor critical jobs - Use acknowledgments and requeue logic
If you use a job library
Tell me which one, and I can give exact settings. Common ones:
- Bull / BullMQ
- RQ
- Sidekiq
- Celery
- Resque
- Hangfire
- Beanstalkd-like wrappers
Quick diagnosis checklist
- Does Redis keep data after restart?
- Is
appendonlyenabled? - Is Redis running in Docker without a volume?
- Are jobs removed on enqueue or on completion?
- Can workers crash without requeuing jobs?
- Does the library support retries and durability?
If you want, I can help you diagnose this for your exact setup. Send:
- the queue library name
- Redis deployment details
- relevant config
- what “disappearing” means in your case: lost after Redis restart, worker restart, or app restart