I'm building an AI chatbot and need tracing for prompts, model calls, and tool use

To trace prompts, model calls, and tool use in an AI chatbot, you’ll want structured observability around every step of the request lifecycle.

What to trace

Capture these events as separate spans/log records:

1. User input

   • session/user ID
   • conversation ID
   • message ID
   • timestamp
   • raw prompt text (or redacted form)

2. Prompt construction

   • system prompt
   • retrieved context
   • conversation history included
   • tool instructions
   • final assembled prompt
   • token counts

3. Model call

   • model name/version
   • temperature, max tokens, top_p, etc.
   • latency
   • input/output tokens
   • response text
   • finish reason
   • retries/errors

4. Tool calls

   • tool name
   • input arguments
   • execution start/end time
   • result or error
   • any side effects
   • correlation to the originating model call

5. Final response

   • returned answer
   • whether tools were used
   • citations / sources if applicable
   • safety filters or moderation decisions

Recommended structure

Use a trace tree with a root request span:

• chat_request
  • prompt_build
  • llm_call
  • tool_call.search
  • tool_call.database
  • response_assembly

Each span should include:

• trace_id
• span_id
• parent_span_id
• start_time
• end_time
• status
• attributes / metadata

Minimal schema example

{
  "trace_id": "trace_123",
  "span_id": "span_001",
  "parent_span_id": null,
  "name": "chat_request",
  "start_time": "2026-10-09T12:00:00Z",
  "end_time": "2026-10-09T12:00:02Z",
  "status": "ok",
  "attributes": {
    "conversation_id": "conv_456",
    "user_id": "user_789"
  }
}

Best practices

• Redact sensitive data before storing prompts or tool inputs.
• Store token counts and metadata even if you don’t store full text.
• Correlate all events with a single trace ID.
• Record retries and failures explicitly.
• Use sampling for high-volume traffic.
• Keep separate logs for PII/security-sensitive content.

Implementation options

You can implement this with:

• OpenTelemetry for spans/traces
• A logging system like ELK / OpenSearch / Datadog / LangSmith / Arize / Honeycomb
• A custom middleware/wrapper around:
  • prompt assembly
  • LLM SDK calls
  • tool execution functions

Example instrumentation flow

1. Request enters chatbot API
2. Create root trace/span
3. Build prompt and log inputs
4. Call model and record latency/token usage
5. If model requests a tool, create child span for that tool
6. Feed tool result back to model and trace the second model call
7. Emit final response span

If you want, I can give you:

• a Python example using OpenTelemetry
• a Node.js example
• a schema for storing traces in Postgres/JSON
• or a full tracing architecture for LLM apps