Why am I getting duplicate requests after gateway retries?

Duplicate requests after gateway retries usually happen because the gateway is retrying a request that the upstream already partially processed, and the request is not idempotent or not safely deduplicated.

Common reasons:

1. Timeouts are ambiguous

   • The gateway times out waiting for a response.
   • It retries.
   • The upstream may have actually completed the original request, but the response was lost or delayed.
   • Result: same operation runs twice.

2. Retry policy on non-idempotent methods

   • Retries on POST, PATCH, or custom write endpoints can create duplicates.
   • GET is usually safe to retry, but write operations often are not unless designed for it.

3. No idempotency key

   • If the client/gateway doesn’t send an idempotency token, the server can’t tell the retry is a duplicate.
   • Each attempt looks like a brand-new request.

4. Retries happen at multiple layers

   • Client retries + gateway retries + service mesh retries can multiply requests.
   • Example: 3 retries at 2 layers can create many more actual calls than expected.

5. Retrying on connection failures after the server received the request

   • A connection reset doesn’t mean the upstream never got the request body.
   • The gateway may retry because it didn’t get a clean response, but the server may already have executed the action.

How to fix it:

• Use idempotency keys for write requests.
• Limit retries to idempotent operations or safe failure modes.
• Deduplicate on the server side using request IDs / transaction IDs.
• Avoid retrying after partial transmission of a request body.
• Tune timeouts so retries don’t fire too aggressively.
• Ensure only one layer owns retries to prevent retry storms.

If you want, I can help you diagnose this for a specific gateway like Kong, NGINX, Envoy, AWS API Gateway, or Apigee.