Prompt
How do I add retries and rollbacks to serverless workflows?
Latest observation
Adding retries and rollbacks to serverless workflows is mostly about making each step reliable, idempotent, and compensatable. Because serverless functions are often stateless and distributed, you usually don’t “roll back” like a database transaction; instead, you design a saga: if a later step fails, you run compensating actions for earlier successful steps.
1) Add retries the right way
Use built-in retry support where available
Most serverless orchestrators support retries directly:
- AWS Step Functions:
Retrywith exponential backoff, max attempts, error filters - Azure Durable Functions: retry options
- Google Cloud Workflows / Cloud Tasks: retry policies
- Temporal (not purely serverless, but common for workflows): activity retries
Example pattern:
- Retry only on transient errors:
- timeouts
- rate limiting
- network failures
- 5xx responses
- Don’t retry on permanent errors:
- validation failures
- authentication issues
- duplicate state transitions
Use exponential backoff with jitter
This prevents retry storms.
Typical policy:
- 3–5 attempts
- exponential delay: 1s, 2s, 4s, 8s...
- add jitter (randomness)
Make functions idempotent
A retry can run the same step more than once, so your function must safely handle duplicates.
Common techniques:
- Use idempotency keys
- Store processed request IDs
- Use conditional writes / optimistic locking
- Check status before doing work
Example:
- If “charge payment” is retried, ensure you don’t charge twice by using a payment intent/transaction ID.
2) Add rollback using compensating actions
In serverless, rollback usually means:
- Step A succeeds
- Step B fails
- Run compensation for Step A
Examples of compensating actions
- Created user account → delete user account
- Reserved inventory → release inventory
- Charged customer → issue refund
- Created cloud resource → delete resource
Design each step with a reverse action
For every side effect, define:
- the forward action
- the compensation action
This is the core of the Saga pattern.
3) Orchestrate the workflow explicitly
Don’t let functions “just call each other” if you need robust retries/rollbacks. Use a workflow engine or orchestrator.
Good orchestration options
- AWS Step Functions
- Azure Durable Functions
- Google Cloud Workflows
- Temporal
- Prefect / Dagster for some use cases
Why:
- Central state tracking
- Retry policies per step
- Error handling branches
- Compensation logic on failure
- Visibility into workflow execution
4) Example workflow pattern
Forward flow
- Validate input
- Reserve inventory
- Charge payment
- Create order
- Send confirmation
Compensation flow if step 3 or 4 fails
- Release inventory
- Void/refund payment if needed
- Mark order as failed
Pseudo-structure:
try:
reserve_inventory()
charge_payment()
create_order()
send_confirmation()
except:
if payment_charged:
refund_payment()
if inventory_reserved:
release_inventory()
mark_failed()
raise
In a real workflow engine, that logic is usually modeled as:
- step states
- retry rules
- catch handlers
- compensation branches
5) Handle partial failures carefully
Some actions are hard to fully undo:
- sending emails
- pushing messages to external systems
- posting to Slack
- calling third-party APIs
For these:
- prefer “do once” semantics when possible
- use deduplication
- send notifications only after the core transaction is committed
- record emitted events so you don’t resend duplicates
6) Use dead-letter queues and failure stores
For repeated failures:
- send events/messages to a DLQ
- store workflow state in a database
- alert ops/engineering
- allow manual reprocessing
This is useful when rollback can’t fully repair the state automatically.
7) Practical checklist
Retries
- Retry only transient failures
- Exponential backoff + jitter
- Set max attempts
- Use timeouts
- Make operations idempotent
Rollbacks
- Define compensation for every side effect
- Persist workflow state
- Use a workflow engine/orchestrator
- Handle compensation failures too
- Avoid irreversible side effects early in the workflow
Observability
- Correlation IDs
- Structured logs
- Traces across steps
- Alerts on retry exhaustion / compensation failure
8) Rule of thumb
If the workflow involves:
- multiple services
- payments/orders/inventory
- external APIs
- anything business-critical
then use:
- orchestrated workflow
- retries for transient errors
- compensating actions for rollback
- idempotency everywhere
If you want, I can show you how to implement this specifically in:
- AWS Step Functions
- Azure Durable Functions
- Google Cloud Workflows
- or with a custom Node.js/Python serverless pattern.