How do I debug retries that cause duplicate LLM outputs?

Here’s a practical way to debug retries that lead to duplicate LLM outputs.

1) First determine where duplication is happening

There are usually three different failure modes:

1. Client retried the same request

   • Your app timed out, then resent the prompt.
   • The model/API may have produced the same or partially repeated output.

2. Your code appended the same response twice

   • Common with async handlers, callbacks, streaming events, or “finally” blocks.
   • Example: one path handles success, another handles “complete” event, both persist output.

3. The LLM itself repeated content

   • Models can sometimes restate prior text, especially with long context, low temperature, or malformed conversation state.

The fix depends on which one you have.

---

2) Add a request ID everywhere

Give every generation attempt a unique ID and log it end-to-end:

• request_id
• conversation_id
• attempt_number
• idempotency_key if your provider supports it
• timestamps for:
  • request sent
  • first token received
  • final token received
  • response persisted

If a duplicate appears, check:

• Was it the same request ID processed twice?
• Or two different request IDs produced similar output?

---

3) Log retry causes explicitly

For each retry, record the exact trigger:

• network timeout
• 5xx from provider
• rate limit
• JSON parse failure
• stream disconnected
• client-side cancellation
• manual retry

This helps distinguish “safe retry” from “accidental duplicate write.”

Example log fields:

{
  "request_id": "abc123",
  "attempt": 2,
  "retry_reason": "timeout_after_30s",
  "provider_status": null,
  "persisted": false
}

---

4) Check for non-idempotent write paths

A very common bug:

• request is sent
• response streams in
• you save partial output on each chunk
• retry kicks in
• original stream later completes
• both outputs get saved

Audit your code for:

• multiple save() calls
• event handlers registered twice
• “on token” and “on complete” both persisting
• retry wrappers around code that already retries internally
• background jobs reprocessing the same item after timeout

---

5) Distinguish output duplication from output repetition

These are different:

Output duplication

Same answer stored twice in your DB/UI.

Output repetition

The model generates repeated text like:

“Here’s the answer… Here’s the answer…”

To detect repetition:

• compare stored outputs
• check if the duplicated text is byte-identical
• inspect whether duplication occurs at the transport/app layer or inside generated text

---

6) Make retries idempotent

If your app can retry safely, ensure the operation is idempotent.

Options:

• Use an idempotency key per user action.
• Store generation state keyed by (conversation_id, turn_id, idempotency_key).
• Before persisting a new response, check whether that key already has a completed result.
• If streaming, only mark completion once.

Pseudo-logic:

if result_exists(idempotency_key):
    return existing_result

result = call_llm()
save_result(idempotency_key, result)
return result

For streaming:

• buffer chunks in memory
• persist only once at end, or
• persist incrementally with sequence numbers and dedupe on replay

---

7) Trace provider behavior

If using an LLM API, inspect:

• request timeout settings
• client retry policy
• server response headers
• stream reconnection behavior
• whether the provider supports request deduplication / idempotency

Watch for:

• retries happening automatically in the SDK
• your own app retrying on top of SDK retries
• load balancers closing idle connections mid-stream

---

8) Reproduce with retries forced

Create a test that intentionally injects failures:

• delay response beyond timeout
• drop the connection after first chunk
• return a 500 once, then succeed
• simulate slow stream

Then see whether your app:

• writes twice
• shows duplicate messages
• reuses stale state

This is the fastest way to pinpoint the bug.

---

9) Add deduplication safeguards

Even if you fix the root cause, defensive dedupe is useful:

• hash the final response
• store response_hash
• prevent identical writes for the same request key
• make UI rendering key off unique message IDs, not array index

For chat UIs, duplicates often come from state updates like:

• setMessages([...messages, newMessage]) with stale closure
• effect re-runs appending again
• websocket reconnection replaying the same event

---

10) Common root causes checklist

• SDK retries + app retries both enabled
• timeout occurs after provider already generated output
• same callback fires twice
• stream reconnection replays chunks
• response saved on both partial and final events
• UI state appended twice due to React re-render / stale closure
• job queue retries the same task without dedupe
• no idempotency key on generation request

---

11) A simple debugging workflow

1. Add request/attempt IDs.
2. Log every retry reason.
3. Check whether duplicates share the same idempotency key.
4. Verify whether duplication occurs before persistence or after rendering.
5. Force failures in staging to reproduce.
6. Add idempotent writes and dedupe guards.
7. Keep only one retry layer.

---

If you want, I can help you debug a specific stack, like:

• OpenAI API + Python
• OpenAI API + Node/TypeScript
• streaming chat UI in React
• job queue / webhook / background worker setup