Prompt
unhappy with Resque for long-running background work
Latest observation
If you’re unhappy with Resque for long-running background work, that’s pretty common. Resque is simple and reliable, but it has a few drawbacks for long-running jobs:
- One job per worker process: long jobs can tie up workers and reduce throughput.
- Fork-per-job overhead: can be heavier than other options.
- Less flexible scheduling/retries compared with newer tools.
- Operational pain if jobs run for a long time or need better concurrency controls.
Good alternatives to consider
1. Sidekiq
Best if you’re in the Ruby ecosystem and want high throughput.
- Uses threads instead of forked processes
- Backed by Redis
- Great for lots of short-to-medium jobs
- Mature retry/dead job support
Caveat: long-running jobs can still be problematic if they block a thread too long, so it’s better for jobs that are not excessively long or can be broken into smaller tasks.
2. GoodJob
If you’re using Rails and want Postgres-backed jobs.
- Uses PostgreSQL
- Easier ops if you already use Postgres
- Good for reliable background processing
- Supports scheduling and cron-like jobs
Caveat: may not be ideal for extremely high throughput compared with Redis-based systems.
3. Que
Another Postgres-backed Ruby job queue.
- Simple and robust
- Good integration with PostgreSQL
- Suitable for persistent background jobs
4. Shoryuken
If your work can fit AWS well.
- Uses SQS
- Scales nicely
- Good for distributed workloads
5. Delayed Job
Simple and familiar, but older.
- Uses your database
- Easy to set up
- Fine for modest workloads
Caveat: not the best for large scale or very long-running tasks.
If the issue is truly “long-running work”
Sometimes the best answer is not just “switch job frameworks,” but to change the job shape:
- Break long tasks into smaller chunks
- Make jobs idempotent
- Persist progress/checkpoints
- Use a workflow/orchestration system if the task has multiple stages
- Consider external workers or event-driven processing for truly heavy work
Quick recommendation
- If you want the most common Ruby replacement: Sidekiq
- If you prefer Postgres and simpler ops: GoodJob
- If jobs are very long and complex: consider chunking + orchestration, not just a queue swap
If you want, I can help you compare Resque vs Sidekiq vs GoodJob specifically for your app and workload.