Prompt
Can you help me decide between a queue, stream, or workflow engine for async work?
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
Yes — a good way to choose is to ask what kind of “async work” you mean, because queues, streams, and workflow engines solve different problems.
Quick rule of thumb
- Queue: use when you want to hand off a task to be processed once, eventually.
- Stream: use when you want to observe a sequence of events and let multiple consumers react.
- Workflow engine: use when the work is a multi-step business process with retries, branching, timeouts, and state.
1) Queue: “Do this task later”
Best for:
- background jobs
- email sending
- image/video processing
- webhook delivery
- simple retries
- offloading slow work from a request path
Characteristics:
- message is typically processed by one consumer
- usually “at least once” delivery
- simple mental model
- good for decoupling and smoothing spikes
Choose a queue if:
- each job is independent
- you don’t need to track a long-lived process
- one worker doing the job is enough
- you mainly need buffering and retry
Examples:
- user signs up → enqueue “send welcome email”
- upload photo → enqueue “generate thumbnail”
2) Stream: “Record events and let consumers subscribe”
Best for:
- event-driven architectures
- analytics
- audit trails
- CDC/change data capture
- real-time processing
- multiple downstream consumers
- replaying historical events
Characteristics:
- events are appended to a log
- multiple consumers can read the same events independently
- replay is often important
- ordering matters within partitions/keys
- more about event flow than task completion
Choose a stream if:
- you care about the history of events
- multiple systems need the same data
- you want consumers to process at their own pace
- you may want to reprocess old events later
Examples:
- order-created events feed billing, inventory, analytics
- clickstream data for dashboards and ML pipelines
3) Workflow engine: “Orchestrate a process”
Best for:
- multi-step business processes
- long-running jobs
- human approval steps
- sagas/orchestration
- compensating actions
- complex retries and branching logic
Characteristics:
- explicitly models state and progress
- step-by-step execution with durable state
- handles retries, timeouts, timers, branching
- often includes visibility into execution
- can survive crashes and continue later
Choose a workflow engine if:
- the process spans multiple steps/services
- you need to know exactly where the process is
- you need durable orchestration, not just message passing
- failures require compensation or manual intervention
Examples:
- order fulfillment: payment → inventory → shipping → notification
- loan application approval with manual review
- insurance claim processing
A practical decision tree
Ask these questions:
- Is this just one task to run later?
- Yes → Queue
- Is this a sequence of events that many systems may consume, with replay/history?
- Yes → Stream
- Is this a business process with multiple steps, state, and complex failure handling?
- Yes → Workflow engine
Common pitfalls
- Using a workflow engine for simple background jobs: often too much overhead.
- Using a queue when you really need event history and multiple consumers: you may lose useful event semantics.
- Using a stream when you really need orchestration/state: consumers become responsible for business process coordination, which gets messy.
A hybrid approach is common
Many systems use all three:
- Queue for low-level background tasks
- Stream for emitting domain events
- Workflow engine for coordinating multi-step business processes
If you want a very short answer
- Queue = task dispatch
- Stream = event distribution
- Workflow engine = process orchestration
If you tell me:
- what the job is,
- whether it’s one-off or multi-step,
- whether multiple systems need the output,
- and your failure/retry requirements,
I can recommend the best fit for your case.