How do I handle provider webhook retries?

To handle provider webhook retries well, design your webhook receiver to be idempotent, fast, and durable.

Core approach

1. Verify the webhook

   • Check signature/authentication from the provider.
   • Reject invalid or malformed requests.

2. Acknowledge quickly

   • Return a 2xx response as soon as you’ve safely recorded the event.
   • Don’t do heavy processing inline if you can avoid it.

3. Make processing idempotent

   • Providers often retry the same event multiple times.
   • Store a unique event ID and ignore duplicates.
   • Ensure your downstream side effects can also be safely repeated.

4. Persist the event before processing

   • Write the incoming webhook payload to durable storage or a queue first.
   • Then process asynchronously.
   • This prevents losing events if your app crashes after receiving them.

5. Use retry-safe side effects

   • If the event triggers email, billing, order fulfillment, etc., guard each action with deduplication.
   • Example: create a “processed_events” table with a unique constraint on provider event ID.

6. Handle out-of-order delivery

   • Retries are not the only issue; providers may also send events out of order.
   • Base your logic on current resource state when possible, not just event sequence.

7. Monitor failures

   • Log event IDs, retry counts, signatures, and processing status.
   • Alert on repeated failures or backlog growth.

---

A simple pattern

• Receive webhook
• Verify signature
• Store raw event + event ID
• Enqueue job
• Return 200 OK
• Worker processes job with dedupe and transactional updates

---

Example deduplication strategy

Use a database table like:

• provider_event_id (unique)
• received_at
• processed_at
• status
• payload

When a webhook comes in:

• Try to insert the event ID.
• If insert succeeds: process it.
• If insert fails due to uniqueness: it’s a retry, so ignore or return 200.

---

Important pitfalls

• Don’t rely on provider retry timing; it varies.
• Don’t assume one delivery equals one action.
• Don’t process long-running work before responding.
• Don’t use non-atomic dedupe checks like “look up then insert” without a unique constraint.

---

If you want a practical rule of thumb

Treat every webhook as “at least once” delivery, and build your handler so duplicate deliveries are harmless.

If you want, I can also show:

• a Node/Express example,
• a Python/FastAPI example,
• or a database schema + transaction flow for reliable webhook handling.