Prompt
How do I handle out-of-order webhook events?
Latest observation
Out-of-order webhooks are normal: delivery systems usually guarantee at-least-once, not in-order. The safest approach is to make your webhook processing idempotent, state-aware, and replay-tolerant.
1) Don’t assume arrival order
Treat every webhook as an independent event. A later event may arrive before an earlier one, and duplicates may arrive too.
2) Use a durable event identifier
Store each event’s unique ID and ignore duplicates.
- If the payload has
event_id, save it in a table with a unique constraint. - If you see the same
event_idagain, return200 OKand do nothing.
3) Track the resource’s latest known state
For each entity (order, subscription, ticket, etc.), store:
- current state
- latest processed version / timestamp / sequence number
When a webhook arrives:
- compare its version/timestamp/sequence to the latest stored one
- if it’s older, discard it or log it
- if it’s newer, apply it
Best case: monotonic version/sequence
If the provider includes a strictly increasing version or sequence, use that.
If only timestamps exist
Use timestamps cautiously:
- clock skew can happen
- equal timestamps can be ambiguous
If timestamps are the only option, prefer:
- “last writer wins” using provider timestamp
- and a fallback fetch of the authoritative object if uncertain
4) Make updates conditional
When writing to your database, use optimistic concurrency:
- update only if
incoming_version > stored_version - or
WHERE version = expected_version
This prevents older events from overwriting newer data.
5) Reconcile with the source of truth
If an event seems stale or missing context:
- fetch the full current object from the provider’s API
- reconcile your local state with the authoritative state
This is especially useful for:
- state transitions
- deleted/created races
- partial webhooks
6) Design handlers to be idempotent
Each handler should safely run multiple times:
- creating a record: use upsert / unique key
- sending an email: record that it was already sent
- provisioning resources: check if already provisioned
7) Queue processing if order matters internally
If you need ordered processing per entity:
- enqueue webhook events
- partition by entity ID
- process each partition sequentially
This preserves order within one entity, while still allowing parallelism across entities.
8) Keep a dead-letter / retry path
If an event can’t be applied because prerequisites aren’t present:
- store it for retry
- or re-fetch current state later
- don’t fail permanently on first attempt
Practical pattern
A common flow is:
- Receive webhook
- Verify signature
- Check
event_idfor dedupe - Load current entity state
- If event is older than current version, ignore
- Apply event or fetch authoritative state
- Persist updated state and processed event record atomically
Example rule
If you get:
subscription.updatedversion 5- then
subscription.createdversion 1
You should not revert to “created” state. Keep version 5 and ignore version 1.
Short answer
Handle out-of-order webhooks by:
- deduplicating events
- storing per-entity version/state
- applying only newer events
- making handlers idempotent
- reconciling with the provider when needed
If you want, I can show a concrete implementation pattern in Python, Node.js, or SQL.