Prompt
I'm building an integration layer for webhook events with retries and filtering — what architecture makes sense?
Latest observation
A good default architecture is:
Webhook receiver → durable event store / queue → filter/router → retry workers → delivery log / dead-letter queue
This gives you reliability, replayability, and clean separation of concerns.
Recommended flow
-
Ingress/API layer
- Accept webhook events from upstream systems.
- Validate signature/authentication.
- Normalize payload into an internal event format.
- Return fast
2xxonce persisted.
-
Durable persistence
- Store the raw event immediately in a database or append-only log.
- Assign an internal event ID and idempotency key.
- This is your source of truth for replay, auditing, and debugging.
-
Filtering / routing
- Apply subscription rules:
- event type
- tenant/customer
- resource predicates
- version
- Decide which downstream destinations should receive the event.
- Keep filter evaluation deterministic and side-effect free.
- Apply subscription rules:
-
Dispatch queue
- Enqueue delivery jobs per destination.
- Each job should contain:
- event ID
- destination ID
- attempt count
- next retry time
- delivery policy
-
Retry worker / delivery engine
- Worker picks jobs due for delivery.
- Sends webhook with timeout and exponential backoff.
- On failure:
- retry for transient errors/timeouts
- stop on permanent failures (e.g. 4xx other than 429, depending on your policy)
- Use jitter to avoid thundering herds.
-
Delivery tracking
- Persist every attempt:
- request timestamp
- response code
- latency
- response body snippet if safe
- error category
- Expose status per delivery and per event.
- Persist every attempt:
-
Dead-letter handling
- After max retries, move job to DLQ / failed state.
- Support manual replay after fixing the destination or filters.
Core design principles
1. Idempotency everywhere
Webhooks are inherently at-least-once. Assume duplicates will happen.
- Include a stable
event_id - Encourage consumers to dedupe by
event_id - Internally dedupe inbound events with a unique key
- Make dispatch jobs idempotent as well
2. Separate ingestion from delivery
Do not call downstream endpoints synchronously in the request path. Persist first, deliver later.
This avoids:
- slow upstream requests
- cascading failures
- losing events during downtime
3. Make filtering declarative
Store subscriptions as data, not code:
- JSON rules
- SQL-like predicates
- simple expression DSL
This makes it easier to:
- re-evaluate on replay
- version rules
- audit who receives what
4. Use a state machine for delivery
Typical states:
receivedmatchedqueueddeliveringsucceededretry_scheduledfaileddead_lettered
This improves observability and retry logic.
5. Backpressure and rate limiting
Protect your system and your customers:
- per-destination concurrency limits
- per-tenant limits
- circuit breakers for consistently failing destinations
- adaptive retry delays on 429/503
Suggested component split
Minimal viable architecture
If you want something simple and solid:
- API service: receives webhook events
- Postgres: stores events, subscriptions, delivery attempts
- Queue: Redis Streams / SQS / RabbitMQ
- Worker service: performs filtering and delivery
- Admin UI: inspect deliveries, retry, replay
More scalable architecture
If traffic grows:
- Event log: Kafka / Pulsar / Kinesis
- Materialized subscription index
- Dispatcher service: creates delivery jobs
- Worker fleet: horizontally scaled
- Observability pipeline: metrics + traces + logs
- DLQ + replay service
Data model sketch
events
event_idsourcetypetenant_idpayloadcreated_atdedupe_key
subscriptions
subscription_idtenant_iddestination_urlsecretfilter_ruleenabledretry_policy
deliveries
delivery_idevent_idsubscription_idstateattempt_countnext_attempt_atlast_status_codelast_errorcreated_atupdated_at
delivery_attempts
attempt_iddelivery_idattempt_norequest_atresponse_atstatus_codeerror_typelatency_ms
Retry policy recommendations
- Retry only transient failures:
- timeouts
- network errors
408,429,500,502,503,504
- Exponential backoff with jitter:
- e.g. 1m, 5m, 15m, 1h, 6h
- Cap max attempts:
- e.g. 10–15
- Add circuit breaking for repeated endpoint failures
Security considerations
- Sign every webhook request with HMAC
- Include timestamp to prevent replay attacks
- Rotate secrets per subscription
- TLS only
- Validate destination URLs to prevent SSRF
- Consider mTLS for high-trust environments
Operational features worth having
- replay from event ID or date range
- filter dry-run/testing
- dead-letter inspection
- per-destination health metrics
- audit log of subscription changes
- schema versioning for payloads
- webhook simulator for customers
A pragmatic recommendation
If you’re early-stage, start with:
- Postgres for persistence
- SQS/Redis queue for delivery jobs
- worker pool for retries
- subscription filters stored in DB
- delivery attempts table for observability
That is often enough until scale forces a move to a log-based architecture.
If you want, I can also sketch:
- a concrete system diagram,
- a Postgres schema,
- or a retry worker pseudocode example.