Prompt
What should I use for long-running background tasks and failure handling?
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
For long-running background tasks with retries, timeouts, and failure handling, the best choice is usually:
Use a job queue / worker system
Examples:
- Celery (Python)
- Sidekiq (Ruby)
- BullMQ (Node.js)
- RQ (Python, simpler)
- Hangfire (.NET)
These are designed for:
- running work asynchronously
- retrying on failure
- delaying jobs
- tracking job state
- separating web requests from heavy work
If the task is critical and multi-step, use a workflow engine
Examples:
- Temporal
- AWS Step Functions
- Prefect / Dagster for data workflows
These are better when you need:
- durable execution
- resumable workflows
- complex branching
- guaranteed state recovery
- human-in-the-loop steps
General rule
- Simple background work: job queue + workers
- Complex, long-lived, failure-prone process: workflow engine
Important failure-handling features to look for
- retries with backoff
- dead-letter queues
- idempotency support
- visibility into job status
- timeouts and cancellation
- persistent job state
If you're choosing one right now
- For most apps: Celery / Sidekiq / BullMQ
- For serious distributed workflows: Temporal
If you want, I can recommend the best option based on your stack, like Python, Node.js, or cloud provider.