incoming webhook deduplication

To deduplicate incoming webhooks, the standard pattern is:

1. Identify a stable unique event key

   • Best: the sender provides an event_id, delivery_id, or idempotency_key
   • Good fallback: hash of immutable fields like:
     • provider name
     • event type
     • object/resource ID
     • event timestamp
     • payload digest

2. Store processed keys atomically

   • Use a database table / Redis set / KV store
   • Enforce a unique constraint on the event key
   • On webhook receipt:
     • try to insert the key
     • if insert succeeds → process
     • if duplicate key error → ignore

3. Make processing idempotent

   • Even with dedupe, retries/races happen
   • Use upserts / compare-and-set / status transitions where possible

4. Acknowledge quickly

   • Return 2xx after the event is safely recorded
   • Do heavy work asynchronously if possible

5. Handle retry windows / TTL

   • If dedupe storage is temporary, keep keys for longer than the sender’s retry period
   • Common TTL: 24h–7d depending on provider

Example approaches

PostgreSQL

CREATE TABLE webhook_events (
  event_key TEXT PRIMARY KEY,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  payload JSONB NOT NULL
);

Then:

INSERT INTO webhook_events (event_key, payload)
VALUES ($1, $2)
ON CONFLICT DO NOTHING;

If rows affected = 0, it’s a duplicate.

Redis

SET webhook:event_key 1 NX EX 604800

• If it returns OK, process it
• If nil, it’s a duplicate

Important caveats

• Do not rely only on timestamps; they may collide.
• Do not dedupe on full payload alone unless payloads are guaranteed identical and stable.
• If the provider signs payloads, use their signature for authenticity, not dedupe.
• Some providers send:
  • same event multiple times
  • different events for the same object
  • out-of-order deliveries
    So dedupe should be per event, not per object, unless that matches your business logic.

If you want, I can give you:

• a production-ready webhook dedupe design
• code examples in Node/Python/Go
• a Redis vs Postgres comparison