How do I quarantine poison messages that keep failing?

To quarantine “poison” messages, the usual pattern is:

1. Limit retries

   • Process a message a small number of times.
   • Track retry count in message metadata or a side store.

2. Detect repeated failure

   • If the same message fails more than N times, stop reprocessing it.

3. Move it to a quarantine/dead-letter queue

   • Send the message to a separate queue or storage location for later inspection.
   • Preserve:
     • original payload
     • error details
     • stack trace / exception
     • retry count
     • timestamps / message ID

4. Alert and review

   • Notify ops/dev teams.
   • Build a tool or process to inspect and optionally replay after fixing the root cause.

5. Make processing idempotent

   • So replays don’t cause duplicate side effects.

A simple policy looks like:

• Try message
• On failure:
  • increment retry count
  • if retry count < 5: requeue
  • else: quarantine it

If you’re using a specific system, the exact implementation varies:

• AWS SQS: use a dead-letter queue (DLQ) with maxReceiveCount
• Azure Service Bus: messages can be moved to the dead-letter queue
• RabbitMQ: use dead-letter exchanges and TTL/requeue rules
• Kafka: commonly use a retry topic + dead-letter topic
• Celery: route failures to a failure queue or backend, depending on setup

If you tell me your queue/broker (SQS, RabbitMQ, Kafka, Azure, etc.), I can show the exact configuration or code.