Prompt

I'm building an integration layer for webhook events with retries and filtering — what architecture makes sense?

Technology / API Platforms2 observationsLast seen Oct 26, 2026

Latest observation

Oct 26, 2026OpenAI APIWeb search: off

A good default architecture is:

Webhook receiver → durable event store / queue → filter/router → retry workers → delivery log / dead-letter queue

This gives you reliability, replayability, and clean separation of concerns.

Recommended flow

  1. Ingress/API layer

    • Accept webhook events from upstream systems.
    • Validate signature/authentication.
    • Normalize payload into an internal event format.
    • Return fast 2xx once persisted.
  2. Durable persistence

    • Store the raw event immediately in a database or append-only log.
    • Assign an internal event ID and idempotency key.
    • This is your source of truth for replay, auditing, and debugging.
  3. Filtering / routing

    • Apply subscription rules:
      • event type
      • tenant/customer
      • resource predicates
      • version
    • Decide which downstream destinations should receive the event.
    • Keep filter evaluation deterministic and side-effect free.
  4. Dispatch queue

    • Enqueue delivery jobs per destination.
    • Each job should contain:
      • event ID
      • destination ID
      • attempt count
      • next retry time
      • delivery policy
  5. Retry worker / delivery engine

    • Worker picks jobs due for delivery.
    • Sends webhook with timeout and exponential backoff.
    • On failure:
      • retry for transient errors/timeouts
      • stop on permanent failures (e.g. 4xx other than 429, depending on your policy)
    • Use jitter to avoid thundering herds.
  6. Delivery tracking

    • Persist every attempt:
      • request timestamp
      • response code
      • latency
      • response body snippet if safe
      • error category
    • Expose status per delivery and per event.
  7. Dead-letter handling

    • After max retries, move job to DLQ / failed state.
    • Support manual replay after fixing the destination or filters.

Core design principles

1. Idempotency everywhere

Webhooks are inherently at-least-once. Assume duplicates will happen.

  • Include a stable event_id
  • Encourage consumers to dedupe by event_id
  • Internally dedupe inbound events with a unique key
  • Make dispatch jobs idempotent as well

2. Separate ingestion from delivery

Do not call downstream endpoints synchronously in the request path. Persist first, deliver later.

This avoids:

  • slow upstream requests
  • cascading failures
  • losing events during downtime

3. Make filtering declarative

Store subscriptions as data, not code:

  • JSON rules
  • SQL-like predicates
  • simple expression DSL

This makes it easier to:

  • re-evaluate on replay
  • version rules
  • audit who receives what

4. Use a state machine for delivery

Typical states:

  • received
  • matched
  • queued
  • delivering
  • succeeded
  • retry_scheduled
  • failed
  • dead_lettered

This improves observability and retry logic.

5. Backpressure and rate limiting

Protect your system and your customers:

  • per-destination concurrency limits
  • per-tenant limits
  • circuit breakers for consistently failing destinations
  • adaptive retry delays on 429/503

Suggested component split

Minimal viable architecture

If you want something simple and solid:

  • API service: receives webhook events
  • Postgres: stores events, subscriptions, delivery attempts
  • Queue: Redis Streams / SQS / RabbitMQ
  • Worker service: performs filtering and delivery
  • Admin UI: inspect deliveries, retry, replay

More scalable architecture

If traffic grows:

  • Event log: Kafka / Pulsar / Kinesis
  • Materialized subscription index
  • Dispatcher service: creates delivery jobs
  • Worker fleet: horizontally scaled
  • Observability pipeline: metrics + traces + logs
  • DLQ + replay service

Data model sketch

events

  • event_id
  • source
  • type
  • tenant_id
  • payload
  • created_at
  • dedupe_key

subscriptions

  • subscription_id
  • tenant_id
  • destination_url
  • secret
  • filter_rule
  • enabled
  • retry_policy

deliveries

  • delivery_id
  • event_id
  • subscription_id
  • state
  • attempt_count
  • next_attempt_at
  • last_status_code
  • last_error
  • created_at
  • updated_at

delivery_attempts

  • attempt_id
  • delivery_id
  • attempt_no
  • request_at
  • response_at
  • status_code
  • error_type
  • latency_ms

Retry policy recommendations

  • Retry only transient failures:
    • timeouts
    • network errors
    • 408, 429, 500, 502, 503, 504
  • Exponential backoff with jitter:
    • e.g. 1m, 5m, 15m, 1h, 6h
  • Cap max attempts:
    • e.g. 10–15
  • Add circuit breaking for repeated endpoint failures

Security considerations

  • Sign every webhook request with HMAC
  • Include timestamp to prevent replay attacks
  • Rotate secrets per subscription
  • TLS only
  • Validate destination URLs to prevent SSRF
  • Consider mTLS for high-trust environments

Operational features worth having

  • replay from event ID or date range
  • filter dry-run/testing
  • dead-letter inspection
  • per-destination health metrics
  • audit log of subscription changes
  • schema versioning for payloads
  • webhook simulator for customers

A pragmatic recommendation

If you’re early-stage, start with:

  • Postgres for persistence
  • SQS/Redis queue for delivery jobs
  • worker pool for retries
  • subscription filters stored in DB
  • delivery attempts table for observability

That is often enough until scale forces a move to a log-based architecture.

If you want, I can also sketch:

  1. a concrete system diagram,
  2. a Postgres schema,
  3. or a retry worker pseudocode example.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.