Agent Observability: Langfuse, Phoenix, Opik

An agent observability platform ingests spans, runs evaluations, stores prompt versions, and surfaces regressions. OTel GenAI (Lesson 23) gives you the schema for the spans, but a schema doesn't store or score anything. You still need something on the receiving end, and the three main contenders each put their weight on a different part of the lifecycle.

Langfuse (MIT)

Langfuse is an open-source platform for tracing, prompt management, evaluation, and session replay, and it's the broadest of the three. It has 6M+ SDK installs a month and 19k+ GitHub stars.

• Tracing, prompt management with versioning and a playground, evaluations (LLM-as-judge, user feedback, custom), and session replays.
• In June 2025 the formerly commercial modules (LLM-as-a-judge, annotation queues, prompt experiments, Playground) were open-sourced under MIT.
• Strongest for end-to-end observability with a tight prompt-management loop.

Arize Phoenix (Elastic License 2.0)

Phoenix is an open-source evaluation and drift tool with native OpenInference auto-instrumentation. It goes deeper than the others on agent-specific evaluation: trace clustering, anomaly detection, and retrieval relevancy for RAG. It pairs with the managed Arize AX for production.

It has no prompt versioning. Treat it as a behavioral-regression tool that sits alongside a broader platform, not a replacement for one. Strongest for RAG relevancy, behavioral drift, and anomaly detection.

Comet Opik (Apache 2.0)

Opik is an open-source platform built around the optimization loop. It runs automated prompt optimization through A/B experiments, enforces guardrails (PII redaction, topical constraints), and detects hallucinations with an LLM judge.

Comet's own benchmark has Opik logging and evaluating in 23.44s against Langfuse's 327.15s, a gap of about 14x. That's a vendor measuring a rival, so read it as directional. Strongest for the optimization loop, automated experimentation, and guardrail enforcement.

Datadog LLM Observability

Datadog LLM Observability is the alternative for mixed ops and ML teams that already run Datadog. It sits outside the three open-source platforms above, and since all three export OTel, Datadog integration doesn't force a choice between them.

Industry data

Per Maxim's 2026 field analysis, 89% of organizations have agent observability in place, and quality issues are the top production barrier (32% of respondents cite them). One plausible reason adoption is that high: all three platforms are open source and all of them consume OTel spans, so getting traces flowing is cheap. Read together, the two numbers say that most teams can now see what their agents do, and the biggest barrier teams report is quality.

Picking one

Where this pattern goes wrong

• No eval strategy. Tracing without evaluation is just expensive logging.
• Self-rolled LLM-judge without grounding. The CRITIC pattern (Lesson 05) applies here: judges need external tools for factual verification.
• Prompt versions not tied to traces. When prod regresses, you can't bisect to the prompt that caused it.

code/main.py implements a stdlib trace collector and LLM-judge evaluator. It does four things:

• Ingests GenAI-shaped spans.
• Groups them by session and tags failed runs (guardrail trips, low-confidence evals).
• Scores agent responses against a rubric with a scripted LLM-judge.
• Prints a dashboard-style summary: failure rate, top failure reasons, eval score distribution.

Run it:

python3 code/main.py

The output is per-session eval scores and failure categorization, the same kind of view Langfuse, Phoenix, and Opik give you.

• Langfuse: self-hosted or cloud. Wire it in through OTel or their SDK.
• Arize Phoenix: self-hosted. Auto-instrument with OpenInference.
• Comet Opik: self-hosted or cloud. Use it when you want the automated optimization loop.
• Datadog LLM Observability: for mixed ops and ML teams that already run Datadog.

outputs/skill-obs-platform-wiring.md picks a platform and wires traces, evals, and prompt versions into an existing agent.

1. Export a week of OTel traces to Langfuse cloud (free tier). Which sessions failed? Why?
2. Write an LLM-judge rubric for your domain (factual correctness, tone, scope adherence). Test on 50 traces.
3. Compare Langfuse prompt versioning against Phoenix's trace clustering. Which tells you what broke faster?
4. Read Opik's guardrail docs. Wire a PII redaction guardrail to one of your agent runs.
5. Benchmark the three on your corpus. Ignore vendor-published numbers; measure your own.

• Langfuse docs: tracing, evals, prompt mgmt
• Arize Phoenix docs: auto-instrumentation, drift
• Comet Opik: optimization + guardrails
• OpenTelemetry GenAI semantic conventions: the schema all three consume