I'm building a customer alerts system and need reliable push delivery with retries

A good way to build this is to treat push delivery as an asynchronous, retriable job pipeline, not as a direct request/response from your app.

Recommended approach

1. Store alerts durably first

When an alert is created:

• write it to your database
• mark it as pending
• enqueue a delivery job

This ensures alerts aren’t lost if your app crashes after creation.

2. Use a queue for delivery

Put delivery jobs onto a queue such as:

• SQS
• RabbitMQ
• Kafka
• Redis-backed queue (BullMQ, Sidekiq, Celery, etc.)

Each job should include:

• alert ID
• recipient/device token
• push provider
• attempt count
• next retry time

3. Make delivery idempotent

Retries can cause duplicate sends unless you prevent them.

Use an idempotency key like:

• alert_id + recipient_id + channel

Before sending, check whether that alert has already been successfully delivered for that recipient.

4. Retry with backoff

On transient failures, retry automatically with exponential backoff:

• 1 min
• 5 min
• 15 min
• 1 hr
• 6 hr

Add jitter so retries don’t all happen at once.

Only retry for:

• timeouts
• 5xx provider errors
• rate limiting
• temporary network issues

Do not retry indefinitely on:

• invalid token
• revoked permission
• malformed payload

5. Handle dead-letter cases

After N failed attempts, move the job to a dead-letter queue or failure table for review.

This helps you:

• inspect poison messages
• alert operators
• decide whether to reprocess later

6. Update delivery state

Track per-recipient state, for example:

• pending
• sent
• delivered
• failed
• expired

Store:

• timestamps
• provider message IDs
• error codes
• attempt counts

7. Separate send from receipt confirmation

For push systems, “sent to provider” is not always “delivered to device.”

If your push provider supports delivery callbacks, process those asynchronously too.

Suggested architecture

• API service: creates alerts
• Database: stores alert and delivery records
• Queue: buffers send jobs
• Worker service: sends pushes and retries failures
• Dead-letter queue: stores exhausted jobs
• Metrics/logging: monitors success, failure, latency, retry counts

Reliability tips

• Use transaction + outbox pattern to avoid losing jobs between DB write and queue publish
• Set queue visibility timeout longer than your send operation
• Deduplicate jobs by alert/device key
• Rate limit per provider/device to avoid throttling
• Refresh invalid device tokens proactively
• Keep payloads small and validated

Minimal retry policy example

• Try immediately
• Retry after 1 minute
• Retry after 5 minutes
• Retry after 15 minutes
• Retry after 1 hour
• Then fail permanently

What I’d recommend most

If you want something robust:

• DB + outbox table
• message queue
• worker with exponential backoff
• idempotency keys
• dead-letter queue
• delivery status tracking

If you want, I can also provide:

1. a reference architecture diagram,
2. a database schema, or
3. sample code for Node.js / Python / Go.