Prompt
I'm building internal tools that send a lot of automated emails when jobs finish or fail. I need retries and good observability because support will blame us when messages don't go out. What stack would you recommend?
Latest observation
For internal job-notification email, I’d optimize for:
- durability so messages aren’t lost,
- retries with backoff for transient failures,
- idempotency/deduping so you don’t spam people,
- traceability so support can answer “what happened to this email?”
Recommended stack
1) Queue + worker architecture
Use a durable queue between your job system and the email sender.
Good options:
- AWS SQS + worker service if you’re already on AWS
- RabbitMQ if you want richer routing/ack control
- Kafka only if you already run Kafka and need high-throughput event streams
- Google Pub/Sub / Azure Service Bus for their clouds
For most internal tools, I’d pick:
- SQS for simplicity and reliability
- or RabbitMQ if you need flexible routing and more control
2) Email provider with bounce/delivery webhooks
Use a provider that gives:
- SMTP/API sending
- delivery/bounce/complaint webhooks
- message IDs
- suppression handling
Good choices:
- Postmark if you want excellent deliverability and observability
- SendGrid if you need broader platform features
- Amazon SES if you want low cost and can build more yourself
My default recommendation:
- Postmark for operationally clean internal notifications
- SES if cost is the main concern
3) Worker framework with retries
Have a stateless worker that:
- pulls jobs from the queue
- renders the email
- sends via provider API
- records result
- retries on transient failures with exponential backoff + jitter
Frameworks:
- Sidekiq (Ruby)
- Celery (Python)
- BullMQ (Node.js)
- Resque or RQ if simpler Python/Ruby needs
- Or just custom workers using SQS/RabbitMQ + your app language
4) Observability stack
You want three layers:
Metrics
Track:
- queued emails
- send attempts
- success/failure rates
- retry counts
- provider latency
- bounce rates
- time from job completion → email sent
Use:
- Prometheus + Grafana
- or Datadog if you want managed observability
Logs
Structured logs with:
- internal notification ID
- job ID
- recipient
- template name
- provider message ID
- retry count
- final status
- error code
Use:
- JSON logs to CloudWatch / ELK / Datadog Logs
Traces
Propagate a correlation ID from the originating job through queue → worker → email provider call.
Use:
- OpenTelemetry
- export to Jaeger / Grafana Tempo / Datadog APM
Data model I’d use
Have a small “notification” table or document store record:
notification_idjob_idrecipienttemplatepayloadstatus= pending/sent/failed/dead_letteredattempt_countnext_retry_atprovider_message_idlast_errorcreated_at,updated_at
This makes support lookups easy.
Retry strategy
Use:
- retry only on transient errors: timeouts, 429s, 5xx
- exponential backoff with jitter
- max attempts, then dead-letter
- no retry on permanent errors: invalid recipient, rejected content, suppressed address
Also:
- use idempotency keys to prevent duplicates if a worker crashes after sending but before recording success
- if provider supports it, use their idempotency feature
- otherwise store a unique send key in your DB and check before resend
Dead-letter handling
Anything that exhausts retries should go to:
- a dead-letter queue
- and create an alert/event for manual review
Support-facing tooling
Build a simple internal dashboard where support can search by:
- job ID
- user email
- notification ID
- date range
Show:
- status timeline
- attempts
- last error
- provider response
- whether it bounced or was suppressed
This is what saves you when support asks “did we send it?”
My default “good” stack
If I were starting today on AWS:
- SQS for queueing
- Worker service in your app language
- Postmark or SES for sending
- PostgreSQL for notification records
- OpenTelemetry for traces
- Prometheus/Grafana or Datadog for metrics
- CloudWatch/ELK for logs
- Dead-letter queue for failed messages
If you want the simplest practical setup
- Queue: SQS
- Sender: SES
- Retries: worker-side exponential backoff
- Observability: structured logs + CloudWatch + basic metrics
- Database: PostgreSQL notification table
That’s the least moving parts while still being supportable.
If you want, I can also give you:
- a reference architecture diagram,
- a provider comparison table (SES vs SendGrid vs Postmark),
- or a sample notification schema + retry pseudocode.