Prompt
How do I orchestrate a multi-step background workflow?
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
To orchestrate a multi-step background workflow, you usually want three things:
- A way to queue work
- A way to track state/progress
- A way to handle retries/failures reliably
Common patterns
1) Use a workflow engine
Best when the workflow has multiple dependent steps, retries, timers, or long-running jobs.
Examples:
- Temporal
- AWS Step Functions
- Azure Durable Functions
- Google Cloud Workflows
- Cadence
Typical flow:
- API starts a workflow
- Workflow engine persists state
- Each step runs asynchronously
- Failures are retried or compensated
- You can query status anytime
Good for: complex orchestration, reliability, observability.
2) Use a job queue + worker state machine
Good for simpler systems.
Example components:
- API/server enqueues a “workflow start” job
- Worker(s) process one step at a time
- Workflow state stored in DB/Redis
- Each step enqueues the next step
- A scheduler or retry mechanism handles delayed/retried work
Good for: moderate complexity, lower infrastructure overhead.
3) Use event-driven choreography
Each step emits events; other services react.
Example:
- Step A completes → emits
ACompleted - Step B listens and starts
- Step B completes → emits
BCompleted
Good for: distributed systems with clear ownership boundaries.
Tradeoff: harder to understand and debug than centralized orchestration.
Practical design
Store workflow state
Keep a record like:
{
"workflow_id": "wf_123",
"status": "running",
"current_step": "transcode_video",
"steps": {
"upload": "done",
"transcode_video": "in_progress",
"notify_user": "pending"
},
"attempts": {
"transcode_video": 2
}
}
Make steps idempotent
Each step should be safe to retry without causing duplicate side effects.
Example:
- Use idempotency keys
- Check whether output already exists
- Deduplicate messages/jobs
Use a step transition model
Each step:
- reads workflow state
- verifies prerequisites
- performs action
- persists result
- schedules next step
Handle failures
Decide what happens on error:
- retry with backoff
- mark failed
- compensation action
- send to dead-letter queue
- alert humans
Include observability
Track:
- workflow id
- step name
- status
- retries
- duration
- error details
Simple implementation sketch
- Client starts workflow:
POST /workflows - API creates DB row:
status = pending - API enqueues
start_workflow(workflow_id) - Worker loads workflow, executes step 1
- On success, worker updates DB and enqueues step 2
- Repeat until complete
- On error, retry or mark failed
When to choose what
- Simple 2–3 step async task: queue + worker
- Many steps, long-running, retries, branching: workflow engine
- Multiple services each owning steps: event-driven choreography
If you want, I can show:
- a Python example with Celery
- a Node.js example with BullMQ
- a Temporal workflow example
- or a database-driven state machine design