Prompt
How do I run delayed jobs and scheduled tasks with a queue?
Latest observation
To run delayed jobs and scheduled tasks with a queue, you usually need three pieces:
-
A queue backend
Examples: Redis, RabbitMQ, SQS, database-backed queue, etc. -
A worker process
This is a long-running process that pulls jobs off the queue and executes them. -
A scheduler or delay mechanism
Used to enqueue jobs at a later time or at fixed intervals.
Common patterns
1) Delayed jobs
A delayed job is simply a job that should not be executed immediately.
Typical flow:
- Create a job
- Attach a scheduled execution time or delay
- Push it to the queue
- Worker ignores it until it becomes due
Example conceptually:
send_email(user_id)with a delay of 10 minutes- Queue stores it with
run_at = now + 10min - Worker picks it up when due
Implementation options:
- Queue supports native delayed delivery
- Store delayed jobs in a separate “scheduled” structure and move them to the main queue when due
- Use a scheduler that periodically enqueues delayed jobs
2) Scheduled tasks
Scheduled tasks are recurring jobs, like:
- every minute
- every hour
- every day at midnight
Typical flow:
- A scheduler runs on a timer
- It enqueues a job when the schedule matches
- Workers execute the job from the queue
Example:
- Every day at 2:00 AM, enqueue
rebuild_search_index - Worker processes it like any other job
Architecture options
Option A: Queue has built-in delayed/scheduled support
Some systems support this directly:
- Sidekiq
- Celery
- BullMQ
- Hangfire
- Quartz-like schedulers
You define:
- delay time, or
- cron expression / interval
Then the queue system handles timing.
Option B: Separate scheduler service
If the queue does not support timing natively:
- A cron-like scheduler checks every N seconds/minutes
- It inserts due jobs into the queue
- Workers consume them normally
This is very common and scales well.
Option C: Database polling
- Store jobs in a table with
run_at - Worker polls for due jobs
- Lock rows to avoid duplicate execution
This is simpler but can be less efficient at scale.
Key things to get right
- Idempotency: jobs may run more than once, so make job handlers safe to retry
- Retries: support transient failures
- Visibility/locking: prevent multiple workers from executing the same job
- Time zones: especially for scheduled tasks
- Persistence: make sure delayed jobs survive restarts
- Backpressure: avoid flooding workers if many jobs become due at once
Simple mental model
- Queue = “what to do”
- Worker = “does it”
- Scheduler/delay system = “when to do it”
If you want, I can show you:
- a language-specific example (Node, Python, Go, etc.)
- a Redis-based design
- or a minimal implementation of delayed + scheduled jobs.