notification API retries fallback channels

A good notification API retry + fallback channel design usually has 3 goals:

1. Transient failures are retried
2. Permanent failures stop quickly
3. If a channel keeps failing, try another channel
   • e.g. push → SMS → email

Recommended approach

1) Model notifications as a workflow

Each notification request should include:

• recipient
• message payload
• ordered channel preferences
• retry policy per channel
• fallback policy

Example:

{
  "user_id": "123",
  "message": "Your OTP is 482931",
  "channels": ["push", "sms", "email"],
  "retry": {
    "max_attempts": 3,
    "backoff": "exponential",
    "base_delay_ms": 1000
  }
}

---

2) Retry only on retryable errors

Retry:

• network timeouts
• 5xx from provider
• rate limits with Retry-After
• connection errors

Do not retry:

• invalid phone/email/token
• user unsubscribed
• malformed request
• provider says “permanent failure”

Use error classification:

• retryable
• non_retryable
• needs_fallback

---

3) Use channel fallback logic

Example behavior:

1. Try push
2. If push fails permanently or exhausts retries, try sms
3. If sms fails, try email

This can be:

• sequential fallback: one channel after another
• parallel fallback with cancellation: send to multiple channels and accept first success
• policy-based: different flows for OTP vs marketing vs alerts

Sequential fallback is usually best for transactional notifications.

---

4) Add idempotency

Retries can duplicate sends unless you make them safe.

Use:

• idempotency_key per notification request
• provider message dedup keys if supported
• delivery record tracking

Example:

{
  "idempotency_key": "notif_abc123"
}

---

5) Persist delivery attempts

Store:

• notification id
• channel
• attempt number
• provider response
• status
• timestamps

This helps with:

• auditing
• replay
• analytics
• debugging

---

6) Use a queue + worker for reliability

API flow:

1. API accepts notification request
2. Persist job
3. Enqueue work
4. Worker sends
5. Worker retries/falls back
6. Update status

This avoids long-running HTTP requests and improves resilience.

---

Example retry/fallback strategy

For transactional OTP

• push retry 2 times
• if still failing, fallback to sms
• if SMS invalid/unavailable, fallback to email
• stop after all channels exhausted

For marketing

• maybe only email
• no retries beyond 1–2
• no fallback to SMS unless user opted in

---

Simple state machine

PENDING -> ATTEMPTING(push)
         -> RETRYING(push)
         -> FALLBACK(sms)
         -> RETRYING(sms)
         -> FALLBACK(email)
         -> DELIVERED / FAILED

---

Practical API response

Return quickly from the API:

{
  "notification_id": "notif_abc123",
  "status": "queued"
}

Then expose a status endpoint:

GET /notifications/notif_abc123

Response:

{
  "notification_id": "notif_abc123",
  "status": "delivered",
  "channel": "sms"
}

---

Best practices

• exponential backoff with jitter
• circuit breaker per provider
• DLQ for poison messages
• per-channel rate limiting
• user preference + consent checks
• observability: logs, metrics, tracing

---

If you want, I can also give you:

1. a concrete API design,
2. a database schema,
3. or sample code in Node.js / Python / Go for retries and fallback.