LLM app latency token cost alerts

Here’s a practical way to set up latency and token-cost alerts for an LLM app.

What to monitor

Latency

Track:

• p50 / p95 / p99 end-to-end request latency
• LLM provider latency if available
• Time to first token (TTFT) for streaming apps
• Queue / retriever / tool-call latency separately from model time

Token usage / cost

Track:

• Prompt tokens
• Completion tokens
• Total tokens
• Cost per request
• Cost per user / org / route
• Daily / hourly burn rate

---

Good alert thresholds

Use a mix of absolute thresholds and relative changes.

Latency alerts

• p95 latency > X ms for 5–10 min
• p99 latency > Y ms for 5 min
• Latency increases by > 30–50% vs trailing 24h baseline
• TTFT > threshold for streaming endpoints

Example:

• Alert if p95 > 2s for 10 minutes
• Alert if p95 > 1.5x the last 24h average

Token / cost alerts

• Cost per minute/hour exceeds budget
• Tokens per request spike by > 25–50%
• Completion tokens increase unexpectedly
• Daily spend projects to exceed budget
• Single request cost above max expected
• A single user/org burns tokens unusually fast

Example:

• Alert if hourly spend > $50
• Alert if daily projection > $500
• Alert if avg tokens/request > 1.3x baseline

---

Recommended alert types

1) SLO-style alerts

For latency:

• “95% of requests under 2s”
• Page only when error budget burn is fast

Example:

• Alert when error budget burn rate > 2x over 1h
• Alert when burn rate > 5x over 5m

2) Anomaly alerts

For token usage:

• Detect spikes in:
  • prompts
  • completions
  • total cost
  • output length

Good for:

• prompt regressions
• runaway loops
• prompt injection causing verbose output
• tool recursion

3) Budget alerts

For finance / ops:

• 50%, 80%, 100% of daily/monthly budget
• projected overspend alerts

---

Metrics to instrument

Log these per request:

• request_id
• user_id / org_id
• route / feature
• model
• prompt_tokens
• completion_tokens
• total_tokens
• latency_ms
• ttft_ms
• tool_latency_ms
• retrieval_latency_ms
• error
• cost_usd

---

Example alert rules

Latency

• Warning: p95 latency > 1500 ms for 10 min
• Critical: p95 latency > 2500 ms for 5 min
• Critical: p99 latency > 5s for 5 min

Cost

• Warning: hourly cost > 80% of budget
• Critical: hourly cost > 100% of budget
• Critical: any request cost > $0.50
• Warning: tokens/request > baseline + 3σ

Usage spikes

• Warning: completion tokens up > 30% day-over-day
• Critical: total tokens up > 2x baseline for 15 min

---

Tools commonly used

Observability

• Datadog
• Grafana + Prometheus
• OpenTelemetry
• New Relic
• Honeycomb

LLM-specific tracking

• LangSmith
• Helicone
• OpenAI usage dashboards / billing exports
• Arize Phoenix
• PromptLayer

---

Practical best practice

Start with:

1. Per-request logging
2. p95 latency alerts
3. Hourly spend alerts
4. Tokens/request anomaly alerts
5. User/org-level caps

Also:

• Separate prod vs staging
• Alert by route/model/user
• Include baseline comparisons
• Add rate limits or kill switches for runaway spend

---

Simple dashboard layout

Show:

• Requests/min
• p50/p95/p99 latency
• TTFT
• Prompt/completion tokens
• Cost/hour and projected cost/day
• Top 10 expensive users/routes
• Error rate
• Retry rate
• Context length distribution

If you want, I can also give you:

1. Prometheus alert rules
2. Datadog monitor examples
3. A JSON schema for LLM request logging
4. A cost/latency alerting strategy for OpenAI or Anthropic apps