Prompt
I'm building a SaaS platform and need webhook-driven integrations with retries
Latest observation
A solid webhook integration design for a SaaS platform usually needs four things:
- Reliable delivery
- Idempotency
- Retry/backoff
- Observability + replay
Here’s a practical pattern you can use.
1) Recommended webhook flow
Outbound event lifecycle
- Your app creates an event, e.g.
invoice.paid - Save it in an
eventstable/collection - Enqueue a delivery job for each subscribed webhook endpoint
- A worker attempts delivery
- On success:
- mark delivery as
delivered
- mark delivery as
- On failure:
- schedule retry with exponential backoff
- stop after max attempts
- Expose delivery logs and manual replay
2) Webhook payload design
Include enough metadata for consumers to safely process duplicates:
{
"id": "evt_123",
"type": "invoice.paid",
"created_at": "2026-10-27T12:00:00Z",
"data": {
"invoice_id": "inv_456",
"amount": 1200,
"currency": "usd"
}
}
Important headers
Send something like:
Webhook-Id: evt_123Webhook-Type: invoice.paidWebhook-Timestamp: 1730000000Webhook-Signature: t=...,v1=...
3) Security
Sign requests
Use HMAC with a shared secret per tenant/webhook.
Example approach:
- Signing string:
timestamp.body - Signature:
HMAC_SHA256(secret, signing_string)
Then consumers verify:
- signature matches
- timestamp is recent
- event ID hasn’t been processed before
Why this matters
- Prevents spoofed requests
- Helps detect tampering
- Protects against replay attacks
4) Retry strategy
A good default:
- Retry on network errors, timeouts, and
5xx - Do not retry on most
4xxresponses except maybe429 - Use exponential backoff with jitter
Example schedule:
- attempt 1: immediately
- attempt 2: after 1 minute
- attempt 3: after 5 minutes
- attempt 4: after 30 minutes
- attempt 5: after 2 hours
- attempt 6: after 12 hours
- attempt 7: after 24 hours
Retry rules
2xx: success408,429,5xx: retry400,401,403,404: usually fail permanently- timeout / DNS / connection reset: retry
5) Idempotency
Because retries can duplicate deliveries, the consumer must handle duplicates.
Best practice
Consumers should store Webhook-Id and ignore already-processed IDs.
You can also help by:
- making
evt_123globally unique - keeping a delivery log per endpoint
- offering an idempotency key header
6) Data model
A simple relational model:
webhook_endpoints
idtenant_idurlsecretstatuscreated_at
webhook_events
idtenant_idtypepayloadcreated_at
webhook_deliveries
idevent_idendpoint_idattempt_countstatus(pending,delivered,failed)next_attempt_atlast_response_codelast_errordelivered_at
7) Worker implementation pattern
Use a job queue like:
- Sidekiq / Redis
- Celery / Redis or RabbitMQ
- BullMQ
- SQS + worker fleet
- Kafka if you already run it
Delivery worker steps
- Load next due delivery job
- Build signed HTTP request
- POST to subscriber endpoint
- Measure response time
- If success:
- mark delivered
- If retryable failure:
- increment attempt count
- compute next retry time
- If permanent failure:
- mark failed
8) Timeout and response expectations
Set a short timeout, like:
- connect timeout: 3–5s
- total timeout: 10–15s
Consumers should return 2xx quickly and process asynchronously if needed.
Rule of thumb
Don’t let webhook endpoints block for long business logic. Acknowledge fast, process later.
9) Observability
Make webhook delivery debuggable.
Track:
- delivery attempts
- latency
- response code
- error message
- last successful delivery
- retry count
- per-endpoint success rate
Expose:
- dashboard
- logs
- manual retry button
- event replay
10) Operational safeguards
Rate limiting
If a consumer is failing repeatedly:
- back off more aggressively
- cap concurrent deliveries per endpoint
Dead letter queue
After max retries:
- move to dead-letter state
- allow manual replay
Circuit breaker
If endpoint is consistently failing:
- pause deliveries temporarily
- notify the customer
11) Example retry policy
A practical policy:
- Max attempts: 8
- Backoff: exponential with jitter
- Retryable status codes:
408,429,500–599 - Hard fail on:
400,401,403,404,422 - Max delivery age: 7 days
12) Example signing logic
Pseudo-code:
timestamp = str(int(time.time()))
body = json.dumps(payload)
message = f"{timestamp}.{body}"
signature = hmac_sha256(secret, message)
headers = {
"Webhook-Id": event_id,
"Webhook-Timestamp": timestamp,
"Webhook-Signature": signature,
"Content-Type": "application/json"
}
13) What I’d recommend for a SaaS MVP
For an MVP, implement:
- event table
- endpoint table
- delivery attempts table
- signed POST requests
- retry queue
- exponential backoff
- dashboard with logs
- manual replay
That gets you a very production-usable system without overengineering.
If you want, I can also give you:
- a database schema
- a retry algorithm
- a reference implementation in Node.js, Python, or Go
- a full webhook architecture diagram