Prompt
I need to handle duplicate delivery and retries safely, what queue design should I use?
Latest observation
Oct 27, 2026
OpenAI APIWeb search: off
Use an at-least-once queue with idempotent consumers, not an “exactly-once” queue.
Recommended design
-
Persist each message with a unique message ID
- Producer generates a stable
message_idoridempotency_key. - Store it with the payload and metadata.
- Producer generates a stable
-
Ack only after successful processing
- Consumer reads message.
- Processes it.
- Commits side effects.
- Then sends ack.
- If the consumer crashes before ack, the message may be redelivered.
-
Make processing idempotent
- Before applying side effects, check whether
message_idhas already been handled. - Use a dedup store / processed-message table with a unique constraint on
message_id. - If already seen, skip safely.
- Before applying side effects, check whether
-
Use retry + dead-letter handling
- Retries for transient failures with exponential backoff.
- After max attempts, move to a dead-letter queue for inspection.
Best-practice queue pattern
- Broker semantics: at-least-once delivery
- Consumer semantics: idempotent, deduplicated
- Storage semantics: transactional or atomic update + dedup record
Common implementation options
- RabbitMQ / SQS / Kafka can all work:
- SQS Standard: at-least-once, duplicates possible
- Kafka: typically at-least-once unless you add transactional producer/consumer logic
- RabbitMQ: at-least-once with manual ack
Practical dedup strategy
- Create a table like:
processed_messages(
message_id TEXT PRIMARY KEY,
processed_at TIMESTAMP
)
- In the same transaction as your business update:
- insert
message_id - perform the update
- insert
- If insert fails due to duplicate key, skip processing.
Rule of thumb
If retries or duplicate delivery can happen, assume they will. Design for:
- duplicate-safe consumers
- idempotent writes
- dead-letter queues
- unique message IDs
If you want, I can sketch a concrete queue architecture for AWS SQS, Kafka, or RabbitMQ.