I'm building a way to compare model quality in production; what should I use to collect logs and evals?

If your goal is to compare model quality in production, you usually want a setup that captures two things:

1. Usage logs / traces of real requests and responses
2. Evaluation data that can be attached to those logs later

What to use

1) For collecting logs and traces

Use an observability/logging layer that records:

• prompt/input
• model name/version
• parameters
• output
• latency
• token usage / cost
• user feedback
• metadata (tenant, feature, route, experiment group)

Common options:

• OpenTelemetry if you want a vendor-neutral telemetry standard
• Langfuse, Helicone, WhyLabs, Arize, or TruLens if you want an LLM-focused product
• Your own structured logging pipeline if you already have infra

If you’re early, Langfuse or Helicone are often the fastest way to get started.

2) For evals

Use an eval framework that can run:

• offline batch evals on logged data
• online / continuous evals on production samples
• human review workflows
• LLM-as-judge or rubric-based scoring

Good choices:

• Langfuse Eval
• Arize Phoenix
• TruLens
• OpenAI Evals for custom harnesses
• Ragas for RAG-specific evaluation
• DeepEval for quick unit-style LLM tests

A practical setup

A common production architecture is:

• Instrument requests with OpenTelemetry or an LLM observability tool
• Store traces/logs centrally
• Sample production outputs for review
• Run eval jobs periodically on that sample set
• Compare model variants by slice:
  • task type
  • user segment
  • prompt template
  • region/language
  • failure category

Recommendation by maturity

Fastest path

• Langfuse + custom eval jobs
• Good if you want logging, tracing, and evals in one place

More analytics-heavy

• Arize Phoenix + warehouse
• Good if you want deep analysis and experimentation

If you already have observability

• OpenTelemetry + your data warehouse + eval framework
• Good if you want full control

What matters most

Make sure your logging captures:

• model version
• prompt version
• retrieval context
• output
• ground truth or proxy signal
• human feedback
• evaluation timestamps

Without versioning, it’s hard to compare models fairly.

If you want, I can suggest a recommended stack based on whether you’re using:

• OpenAI / Anthropic / self-hosted models
• RAG or non-RAG
• Python, JS, or both
• existing tools like Datadog, Snowflake, or BigQuery