What should I use to monitor AI costs and latency across services?

For monitoring AI costs and latency across services, you generally want a combination of:

1. An observability platform
2. Request tracing / logging with token usage
3. Cost attribution by service/team/model
4. Latency metrics by endpoint/model/provider

Good options

If you want an all-in-one AI observability tool

• LangSmith — great if you use LangChain/LangGraph; tracks traces, latency, token usage, errors.
• Helicone — purpose-built for LLM observability; good for cost tracking, latency, and request logs across providers.
• Arize Phoenix — strong for tracing, evaluation, and debugging AI pipelines.
• WhyLabs — more enterprise-focused observability and monitoring.

If you already use general observability tools

• Datadog — solid for service-level latency, logs, distributed tracing, and custom cost metrics.
• Grafana + Prometheus + OpenTelemetry — best if you want flexible infra and custom dashboards.
• New Relic — similar to Datadog for app/service monitoring.

If you mainly need cloud/provider cost visibility

• AWS Cost Explorer / CUR
• GCP Billing + Cloud Monitoring
• Azure Cost Management

These are useful for infra spend, but they usually won’t give you good LLM-level breakdowns like:

• prompt vs completion tokens
• per-request model cost
• latency per chain/tool call
• retries / failures / cache hits

What I’d recommend

• For LLM apps: Helicone or LangSmith
• For broader platform monitoring: Datadog or Grafana/Prometheus with OpenTelemetry
• For finance attribution: add custom labels/tags for service, team, environment, and model

Key metrics to track

• Latency: p50 / p95 / p99
• Cost: per request, per 1k tokens, per service, per user
• Token usage: input/output tokens
• Error rates: timeouts, rate limits, provider errors
• Quality metrics: tool-call success, response acceptance, fallback rate

If you want, I can also give you:

• a recommended stack by company size
• a comparison table of Helicone vs LangSmith vs Datadog
• or a reference architecture for monitoring AI costs across microservices.