Prompt
How do I orchestrate multi-step workflows with APIs and webhooks?
Latest observation
To orchestrate multi-step workflows with APIs and webhooks, think of the system as a state machine: each step triggers the next, while your workflow engine or backend keeps track of progress, retries, failures, and timeouts.
Core pattern
-
Trigger starts the workflow
- A user action, cron job, API call, or webhook event starts execution.
- Example: “Order created” webhook.
-
Persist workflow state
- Store:
- workflow ID
- current step
- input/output data
- status (
pending,running,waiting,failed,completed) - retry count and timestamps
- This lets you resume after failures.
- Store:
-
Execute step 1 via API
- Call the first service API.
- Validate response.
- Save output.
-
Decide next step
- If the next action depends on an external event, set workflow status to
waitingand stop. - If it’s immediate, continue to step 2.
- If the next action depends on an external event, set workflow status to
-
Handle webhooks for asynchronous completion
- External systems send webhooks when work finishes.
- Verify webhook signatures.
- Match webhook payload to the correct workflow instance.
- Resume from the saved state and continue processing.
-
Use retries and compensation
- Retry transient failures with exponential backoff.
- For irreversible actions, define compensating steps.
- Example: if payment succeeds but shipping fails, refund the payment or mark for manual review.
Recommended architecture
Option A: Workflow engine
Use a workflow tool if the process is non-trivial:
- Temporal
- AWS Step Functions
- Google Cloud Workflows
- Azure Durable Functions
- Camunda / Zeebe
These help with:
- retries
- timers
- long-running tasks
- durable state
- human approval steps
- observability
Option B: Custom orchestrator
Build your own if the flow is simple:
- API server receives triggers
- queue system dispatches tasks
- worker processes execute steps
- database stores state
- webhook endpoint resumes workflows
Common components:
- API gateway / webhook receiver
- Queue: SQS, RabbitMQ, Kafka, Pub/Sub
- Worker service
- Database: Postgres, DynamoDB, Redis (for coordination)
- Monitoring/logging
Design principles
1. Make steps idempotent
Each step should be safe to run more than once.
- Use idempotency keys
- Deduplicate webhook events
- Track processed event IDs
2. Correlate everything
Include a workflow/run ID in:
- outbound API calls
- webhook payloads if possible
- logs and tracing spans
3. Separate sync and async steps
- Synchronous: step completes during the request
- Asynchronous: step starts now, completes later via webhook or polling
4. Use a queue for reliability
Don’t chain API calls directly in request threads if the workflow may be long.
- enqueue task
- worker executes
- on completion, enqueue next task
5. Handle failures explicitly
Define:
- retryable errors
- permanent errors
- dead-letter queue behavior
- rollback/compensation actions
Example flow
Use case: Create a customer, charge payment, then provision account.
- User submits signup form.
- Backend creates workflow record.
- Backend calls CRM API to create customer.
- Backend calls payment API to charge card.
- Payment provider sends webhook
payment_succeeded. - Workflow resumes and calls provisioning API.
- Provisioning API returns success.
- Workflow marked
completed.
If payment fails:
- mark workflow
failed - notify user
- optionally retry or request new payment method
Practical implementation tips
- Store workflow steps as records in a database table.
- Use a unique workflow ID for each run.
- Use webhook signature verification to avoid spoofing.
- Return 200 quickly from webhook endpoints, then process asynchronously.
- Avoid holding open HTTP connections waiting for long tasks.
- Keep step logic small and composable.
Minimal pseudocode
def start_workflow(input):
wf = db.create_workflow(status="running", step="create_customer", input=input)
enqueue(wf.id)
def worker(workflow_id):
wf = db.get_workflow(workflow_id)
if wf.step == "create_customer":
result = crm.create_customer(wf.input)
db.save_step_output(wf.id, result)
wf.step = "charge_payment"
db.update(wf)
elif wf.step == "charge_payment":
payment = payments.charge(wf.input)
if payment.status == "pending":
wf.status = "waiting"
wf.step = "wait_for_payment_webhook"
db.update(wf)
else:
wf.step = "provision_account"
db.update(wf)
enqueue(wf.id)
def payment_webhook(event):
if not verify_signature(event):
return 401
wf = db.find_workflow_by_payment_id(event.payment_id)
if wf and wf.status == "waiting":
wf.step = "provision_account"
wf.status = "running"
db.update(wf)
enqueue(wf.id)
return 200
When to choose what
- Few steps, low complexity → custom orchestrator
- Long-running, critical, multi-service workflows → workflow engine
- Need human approvals or timers → workflow engine strongly preferred
If you want, I can also give you:
- a reference architecture diagram,
- a Node.js/Python example, or
- a Step Functions / Temporal implementation pattern.