Troubleshoot API gateway retries causing duplicate requests

Here’s a practical way to troubleshoot API gateway retries causing duplicate requests.

1) Confirm where the duplicate is happening

Check whether the duplicate requests are coming from:

• The client retrying
• The API gateway retrying upstream calls
• A load balancer / service mesh retrying
• Your backend processing the same request more than once

Look at logs at each layer and compare:

• request timestamps
• request IDs / correlation IDs
• upstream retry counters
• response codes that triggered retries

2) Identify the retry trigger

Gateways often retry on:

• timeouts
• 5xx responses
• connection resets
• upstream unavailable / DNS failures
• read/write errors

If the backend is returning slow responses or transient 5xx errors, the gateway may resend the request.

3) Check whether retries are safe for the HTTP method

Retries on GET/HEAD/OPTIONS are usually okay. Retries on POST/PUT/PATCH/DELETE can create duplicates unless the request is idempotent.

If your gateway retries non-idempotent requests, that’s a common cause of duplication.

4) Add idempotency protection

For write operations, use one of these:

• Idempotency-Key header
• unique request ID stored server-side
• deduplication table/cache with TTL
• conditional writes / optimistic locking

Typical flow:

1. Client sends Idempotency-Key: abc123
2. Backend stores the first successful result for that key
3. Any repeated request with the same key returns the same result instead of creating a second record

5) Review gateway retry settings

Look for settings like:

• retry count
• retry on status codes
• retry on connect/read timeout
• retry budget
• per-route overrides
• retry only on safe methods

If possible:

• reduce retries for write endpoints
• disable retries for non-idempotent routes
• increase upstream timeout instead of retrying aggressively

6) Check timeout alignment

A common pattern is:

• gateway timeout is too short
• backend is still processing
• gateway retries
• backend eventually completes both attempts

Make sure these are aligned:

• client timeout
• gateway timeout
• upstream timeout
• backend processing time

Avoid a gateway timing out before the backend can reliably respond.

7) Inspect logs for duplicate IDs

Add or verify logging of:

• request ID
• idempotency key
• upstream attempt number
• gateway retry reason
• backend transaction/order ID

If the same request ID hits the backend multiple times, the gateway is likely retrying.
If different request IDs are generated, the client may be retrying.

8) Look for partial failures

Sometimes the backend completes the action but the response is lost:

• response timeout after commit
• gateway retries because it never saw the response
• duplicate side effect occurs

This is especially common with:

• payment creation
• order submission
• webhook delivery
• background job enqueue endpoints

9) Add deduplication at the business layer

Even with gateway retries, the safest design is for the backend to handle duplicates gracefully.

Examples:

• unique constraint on order reference
• “create once” semantics using an external request ID
• idempotent job submission queue keys
• transaction table with request fingerprint

10) Reproduce in a lower environment

Simulate:

• slow backend response
• 5xx response
• network timeout
• gateway retry policy

Then observe:

• number of backend invocations
• number of rows/records created
• whether retries are automatic

Quick checklist

• Is the gateway retrying non-idempotent methods?
• Are retries triggered by timeouts or 5xx?
• Are timeouts aligned across client/gateway/backend?
• Is there an idempotency key or dedupe mechanism?
• Do logs show same request ID multiple times?
• Can retries be disabled for write routes?

Best practice recommendation

For any endpoint that creates or changes state:

• disable automatic retries at the gateway, or limit them strictly
• implement idempotency on the server
• log request IDs and retry attempts
• make the backend safe against duplicates

If you want, I can help you troubleshoot this for a specific gateway such as AWS API Gateway, Kong, NGINX, Apigee, Envoy, Azure API Management, or GCP API Gateway.