I need LLM observability that works with multiple providers and custom tools

If you need LLM observability across multiple providers and custom tools, look for a platform that supports:

• OpenTelemetry or SDK-based tracing
• Vendor-neutral instrumentation
• Model/provider agnostic logging
• Custom tool / function-call traces
• Prompt, response, token, latency, error, and cost tracking
• Evaluation / feedback hooks
• PII redaction and access controls

Good options to consider

1. Langfuse

• Works with multiple LLM providers
• Strong support for traces, spans, scores, and evaluations
• Good fit for custom tools / workflows
• Can self-host
• Open-source and commonly used for product observability

2. LangSmith

• Strong tracing for chains/agents/tools
• Works well with LangChain-based apps
• Supports custom runs and metadata
• Best if your stack is already in the LangChain ecosystem

3. Helicone

• Easy proxy-based observability
• Supports multiple providers
• Tracks latency, usage, costs, and request/response data
• Good for quick setup and API-level observability

4. Arize Phoenix

• More focused on evaluation, debugging, and experimentation
• Useful for tracing LLM apps and analyzing quality issues
• Good if you want stronger analysis workflows

5. OpenTelemetry + your observability backend

• Best if you want full control and already use OTEL
• More setup effort, but very flexible
• Great for capturing custom tools, non-LLM spans, and app-level context

My quick recommendation

• Best all-around open-source choice: Langfuse
• Best if you want easiest API proxy setup: Helicone
• Best if you’re deeply in LangChain: LangSmith
• Best if you want maximum flexibility: OpenTelemetry

What to verify before choosing

Make sure the product supports:

• OpenAI, Anthropic, Gemini, Azure OpenAI, Bedrock, local models
• Custom spans for tools/functions
• Conversation/session grouping
• Prompt versioning
• Feedback and evals
• Self-hosting if needed
• Data retention and privacy controls

If you want, I can give you a side-by-side comparison table or recommend the best option for your stack if you tell me:

1. which providers you use,
2. whether you need self-hosting, and
3. whether you’re using LangChain/LlamaIndex/custom code.