How do I monitor token usage and latency for LLM requests?

To monitor token usage and latency for LLM requests, track a few standard metrics at the request level and aggregate them over time.

1) What to measure

Token usage

Track:

• Input tokens: tokens in the prompt/messages
• Output tokens: tokens in the model response
• Total tokens: input + output
• Optionally:
  • Cached tokens / reasoning tokens if your provider exposes them
  • Tokens per user/session/model for cost attribution

Latency

Track:

• End-to-end latency: time from request start to final response
• Time to first token (TTFT): useful for streaming responses
• Model latency: if the provider exposes server-side timing
• Retry latency: if you retry failed requests

2) Basic instrumentation pattern

For every request, log:

• request id
• model name/version
• start timestamp
• end timestamp
• latency in ms
• input token count
• output token count
• total token count
• success/failure and error type
• user/session/tenant id
• cost estimate

Example pseudocode

import time

start = time.time()

response = client.chat.completions.create(
    model="your-model",
    messages=[{"role": "user", "content": "Hello"}],
)

end = time.time()

usage = response.usage  # if provided by your SDK/provider

log = {
    "model": "your-model",
    "latency_ms": round((end - start) * 1000),
    "input_tokens": usage.prompt_tokens if usage else None,
    "output_tokens": usage.completion_tokens if usage else None,
    "total_tokens": usage.total_tokens if usage else None,
}
print(log)

3) For streaming responses

Measure:

• TTFT: start time until first chunk arrives
• Full latency: start time until stream ends

Example approach:

• record t0 before the request
• on first streamed chunk, compute ttft_ms
• on final chunk, compute latency_ms

4) Where to store metrics

Common options:

• Logs: JSON logs to stdout / file / log platform
• Metrics system: Prometheus, Datadog, CloudWatch, OpenTelemetry
• Tracing: distributed traces with spans for each LLM call
• Analytics warehouse: for cost and usage reporting

A good setup is:

• Metrics for dashboards/alerts
• Logs for request-level debugging
• Traces for end-to-end latency

5) Useful dashboards

Build dashboards for:

• p50 / p95 / p99 latency
• TTFT
• token usage over time
• cost per day / per tenant / per feature
• error rate
• tokens per request
• throughput (requests/minute)

6) Alerts to set up

Alert on:

• latency spikes
• token usage spikes
• error rate increases
• cost anomalies
• unusually long outputs
• timeouts / rate limits

7) If your provider returns usage automatically

Many LLM APIs return token counts in the response. If so, use those values rather than estimating yourself. If not, you can estimate tokens with the provider’s tokenizer for the model.

8) Best practices

• Include a correlation/request id
• Separate metrics by model version
• Track prompt size vs response size
• Record retries and timeouts
• Be careful with PII in logs
• Sample logs if volume is high

If you want, I can give you:

1. a Python example with OpenTelemetry,
2. a Prometheus/Datadog setup, or
3. a streaming TTFT monitoring example.