I need to trace user complaints back to the exact prompt and model call

Yes — to trace a user complaint back to the exact prompt and model call, you want to log every LLM request with a unique trace ID and store the full request payload plus response metadata.

What to capture for each model call

Store these fields for every request:

• trace_id / request_id: unique identifier for the call
• user_id / session_id
• timestamp
• application route / feature name
• exact system prompt
• exact developer prompt
• exact user prompt
• all tool/function definitions
• retrieved context (RAG docs, memory, conversation history)
• model name / version
• all parameters:
  • temperature
  • top_p
  • max_tokens
  • presence/frequency penalties
  • seed, if used
• tool calls made by the model
• full model response
• finish reason
• token usage
• latency
• error details if the call failed

Best practice: log the “assembled prompt”

A lot of bugs happen after prompts are composed from multiple sources. So log:

1. Raw components

   • system prompt template
   • developer prompt template
   • user message
   • retrieved docs
   • prior chat history

2. Final assembled prompt/messages

   • the exact payload sent to the model API

That way you can compare:

• what the app intended to send
• what was actually sent

Recommended tracing pattern

Use a structured trace like:

{
  "trace_id": "abc123",
  "user_id": "u_456",
  "session_id": "s_789",
  "timestamp": "2026-10-09T12:34:56Z",
  "feature": "support_chat",
  "model": "gpt-4.1",
  "parameters": {
    "temperature": 0.2,
    "top_p": 1,
    "max_tokens": 800
  },
  "messages": [
    {"role": "system", "content": "..."},
    {"role": "developer", "content": "..."},
    {"role": "user", "content": "..."}
  ],
  "retrieval": {
    "docs": ["doc1", "doc2"]
  },
  "response": {
    "content": "...",
    "finish_reason": "stop"
  },
  "usage": {
    "prompt_tokens": 512,
    "completion_tokens": 184,
    "total_tokens": 696
  }
}

How to make complaints traceable

When a user complains, ask them for one of:

• the session ID
• the request ID / trace ID
• the timestamp
• the feature name

Then search your logs using that identifier and inspect:

• exact prompt content
• the model version
• the retrieval context
• the output

Important security/privacy note

If prompts can contain sensitive user data:

• redact PII where appropriate
• restrict log access
• encrypt logs at rest
• set retention limits
• consider storing hashes for sensitive blobs and keeping raw content in a secure audit store

If you want the exact cause of a bad answer

Also record:

• prompt template version
• retrieval index version
• tool versions
• application git commit / deployment version

This helps you determine whether the issue came from:

• prompt change
• model change
• retrieval change
• code change

If you want, I can give you:

1. a logging schema
2. a Python/TypeScript middleware example
3. a debugging workflow for tracing a complaint end-to-end