Prompt
I'm building something that needs scheduled jobs and long-running background work, should I use a queue or workflow engine?
Latest observation
Use a workflow engine if you need to model multi-step processes that can wait, retry, branch, or span long periods of time. Use a queue if you mainly need to run independent background jobs asynchronously and at scale.
Quick rule of thumb
Choose a queue when you need:
- simple async work
- fire-and-forget tasks
- retrying failed jobs
- rate limiting / worker pools
- high throughput, low coordination
Examples:
- send an email
- resize an image
- generate a report
- process webhook events
Choose a workflow engine when you need:
- multi-step business processes
- durable state across long waits
- human approval steps
- branching logic and compensation
- visibility into “where is this process now?”
- orchestration across services
Examples:
- order fulfillment
- loan application processing
- subscription onboarding
- “wait 3 days, then check again”
- refund/rollback flows
Scheduled jobs
If by “scheduled jobs” you mean:
- run something at a certain time → queue + scheduler is often enough
- coordinate a schedule-dependent process over time → workflow engine is better
Long-running background work
If the work is just one long task, a queue can still work well, as long as:
- the worker can safely run for a long time
- you handle retries/idempotency
- you don’t need the process state persisted across crashes in a rich way
If the work is long-running because it pauses, waits, or has multiple steps, a workflow engine is usually the right fit.
A practical way to decide
Ask:
-
Is this just a job, or a process?
- job → queue
- process → workflow engine
-
Do I need to survive restarts without re-implementing state management?
- yes → workflow engine
-
Do I need visibility into progress, branching, retries, and compensation?
- yes → workflow engine
-
Do I just need durable execution of independent tasks?
- yes → queue
Common hybrid pattern
A lot of systems use both:
- workflow engine for orchestration
- queue for heavy leaf tasks / fan-out work
Recommendation
- Start with a queue if your needs are mostly “background jobs + scheduling.”
- Reach for a workflow engine if you already know you need durable orchestration and long-lived stateful processes.
If you want, tell me your exact use case and I can recommend a concrete architecture.