How do I create a dead-letter queue for failed webhook deliveries?

A dead-letter queue (DLQ) for failed webhook deliveries is usually just a separate place to store events that couldn’t be delivered after retries, so they can be inspected, replayed, or alerted on later.

Typical setup

1) Record every outgoing webhook attempt

Store:

• event ID
• destination URL
• request body / payload
• headers
• attempt count
• last error / HTTP status
• timestamps
• delivery status: pending, retrying, delivered, failed, dead-lettered

2) Retry transient failures

Retry on things like:

• timeouts
• DNS/network errors
• 5xx responses
• 429 rate limits

Use an exponential backoff strategy, for example:

• 1 min
• 5 min
• 15 min
• 1 hour
• 6 hours

Set a max retry count or max retry window.

3) Move permanent failures to the DLQ

After retries are exhausted, or for non-retriable errors, mark the delivery as dead-lettered and store it in a DLQ table/queue.

Examples of non-retriable failures:

• 400 Bad Request due to invalid payload
• 401/403 if credentials are invalid and won’t recover automatically
• 404 if endpoint is gone
• schema validation failure before sending

4) Expose tooling for replay and inspection

A DLQ is useful only if you can:

• inspect the payload and failure reason
• replay after fixing the issue
• manually delete or archive old failed deliveries

---

Implementation patterns

Option A: Database-backed DLQ

Simple and common.

Tables:

• webhook_deliveries
• webhook_delivery_attempts
• webhook_dead_letters

When a delivery exceeds retry limits:

INSERT INTO webhook_dead_letters (...)
SELECT ... FROM webhook_deliveries WHERE id = ...
UPDATE webhook_deliveries SET status = 'dead-lettered' ...

Good for:

• smaller systems
• easy debugging
• admin UI replay

Option B: Message queue DLQ

If you use a queue system:

• main queue holds delivery jobs
• failed jobs after max retries go to a DLQ queue

Examples:

• AWS SQS main queue + DLQ
• RabbitMQ dead-letter exchange
• Kafka retry topics + parking-lot topic

Good for:

• high volume
• decoupled worker architecture

---

Best practices

• Sign webhook payloads so recipients can verify authenticity.
• Use idempotency keys so replaying doesn’t create duplicates.
• Separate transient vs permanent errors.
• Store failure context: response body, status code, exception, stack trace.
• Alert on DLQ growth so failures are noticed quickly.
• Set retention rules for old dead letters.
• Support manual replay after configuration fixes.

---

Simple state flow

pending -> sending -> delivered
                -> retrying -> sending ...
                -> dead-lettered

---

Example rule of thumb

• Retry up to 5 times over 24 hours
• Dead-letter if:
  • 4xx permanent failure
  • or 5 retries exhausted
  • or payload validation fails before first send

---

If you want, I can also give you:

1. a database schema for webhook DLQ,
2. an AWS SQS/RabbitMQ example,
3. or code for implementing retries + DLQ in Node, Python, or Go.