CapstoneTypeScript (UI), Python / TypeScript (ingest + evals), SQL (ClickHouse)

Capstone 11 — LLM Observability & Eval Dashboard

Langfuse went open-core, Arize Phoenix published the 2026 GenAI semconv mappings, and Helicone and Braintrust both doubled down on per-user cost attribution. Traceloop's OpenLLMetry became the default SDK instrumentation. The production shape is ClickHouse for traces, Postgres for metadata, Next.js for the UI, and a pool of eval jobs (DeepEval, RAGAS, LLM-judge) running over sampled traces. Build one self-hosted, ingest from at least six SDK families, and catch an injected regression in under five minutes.

TIME: 25 hoursPREREQUISITES: Phase 11 (LLM engineering), Phase 13 (tools), Phase 17 (infrastructure), Phase 18 (safety)

Learning Objectives

Build an ingest path where OpenTelemetry GenAI spans from six SDK families flow through a tail-sampling collector into ClickHouse (spans) and Postgres (metadata).
Run batch eval jobs (DeepEval, RAGAS, a custom PII-leak LLM-judge) over sampled traces and write the scores back as eval spans linked to the parent trace.
Detect drift by computing PSI on pooled prompt embeddings against a trailing baseline, and route threshold breaches through Prometheus and Alertmanager to Slack or PagerDuty.
Measure time to detect on an injected PII-leak regression, and reason about how sampling rules and eval cadence affect whether you meet the five-minute bar.

Problem

LLM observability is the practice of capturing model calls as traces, scoring those traces for quality and safety, and alerting when behavior shifts. Any AI team running production traffic in 2026 keeps this plane next to the model: cost attribution, hallucination detection, drift monitoring, jailbreak signal, SLO dashboards, PII leak alerts. The open-source references (Langfuse, Phoenix, OpenLLMetry) converged on the OpenTelemetry GenAI semantic conventions as the ingest schema, so one SDK can now instrument OpenAI, Anthropic, Google, LangChain, LlamaIndex, and vLLM and emit compatible spans.

You will build a self-hosted dashboard that ingests from at least six SDK families, runs a small set of eval jobs over sampled traces, detects drift, and alerts. The measurement bar is concrete. Inject a regression on purpose (a prompt that starts producing PII), and the dashboard has to catch it and fire an alert in under five minutes.

Warm-up0/3

1. A team ships an LLM feature and wants to know which customers drive most of the model bill. What should each request record?Warm-up

2. Which check would catch a model that echoes a customer's social security number in its answer?Warm-up

3. You need to average latency per hour across hundreds of millions of rows while reading only two columns. Which kind of store fits best?Warm-up

Concept

The dashboard is a pipeline of four parts: ingest, storage, evals, and drift plus alerting. Ingest is OTLP HTTP. The SDK produces GenAI-semconv spans carrying attributes like gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.response.id, llm.prompts, and llm.completions. Spans land in ClickHouse for columnar analytics, while metadata (users, sessions, apps) lands in Postgres.

Evals run as batch jobs over sampled traces. DeepEval scores faithfulness, toxicity, and answer relevance. RAGAS scores retrieval metrics when the trace carries retrieval context. Custom LLM-judges handle domain-specific checks such as PII leaks and off-policy responses. Every eval run writes back to the same ClickHouse as eval spans linked to the parent trace, so a score always sits next to the call it judged.

Drift detection watches embedding-space distributions over time (PSI or KL divergence on prompt embeddings) alongside eval-score trends. Alerts feed Prometheus Alertmanager, which routes to Slack or PagerDuty. The UI is Next.js 15 with Recharts.

Check: Concept0/10

1. What is an eval span?Define

2. What is the GenAI semantic convention in OpenTelemetry?Define

3. What does PSI measure in this dashboard?Define

4. What is an LLM-judge?Define

5. What is cost per user?Define

6. How do spans get from an instrumented app into the collector?Define

7. Why does the storage tier split between ClickHouse and Postgres?Explain

8. Why does an eval write its score as a span linked to the parent trace instead of a separate report?Explain

9. PSI is computed on prompt embeddings. What does a high value tell you?Explain

10. One app's prompts change sharply, but the pooled PSI across all apps stays at 0.05. What explains it?Analyze

Architecture

production apps:
  OpenAI SDK  +  Anthropic SDK  +  Google GenAI SDK
  LangChain + LlamaIndex + vLLM
       |
       v
  OpenTelemetry SDK with GenAI semconv
       |
       v  OTLP HTTP
  collector (ingest, sample, fan-out)
       |
       +-------------+-----------+
       v             v           v
   ClickHouse    Postgres    S3 archive
   (spans)       (metadata)  (raw events)
       |
       +---> eval jobs (DeepEval, RAGAS, LLM-judge)
       |     sampled or all-trace
       |     write eval spans back
       |
       +---> drift detector (PSI / KL on prompt embeddings)
       |
       +---> Prometheus metrics -> Alertmanager -> Slack / PagerDuty
       |
       v
   Next.js 15 dashboard (Recharts)

Stack

Ingest: OpenTelemetry SDKs + GenAI semantic conventions; OTLP HTTP transport
Collector: OpenTelemetry Collector with tail-sampling processor (for cost control)
Storage: ClickHouse for spans, Postgres for metadata, S3 for raw event archive
Evals: DeepEval, RAGAS 0.2, Arize Phoenix evaluator pack, custom LLM-judge
Drift: PSI / KL on pooled prompt embeddings (sentence-transformers) weekly
Alerting: Prometheus Alertmanager -> Slack / PagerDuty
UI: Next.js 15 App Router + Recharts + server actions
SDKs supported out of the box: OpenAI, Anthropic, Google GenAI, LangChain, LlamaIndex, vLLM

Check: Stack0/1

1. A seventh framework has no auto-instrumentation. How do you get it onto the dashboard without touching the storage or rollups?Design

Build It

Collector config. Set up the OpenTelemetry Collector with the OTLP HTTP receiver and a tail-sampler that keeps 100% of errored traces and 10% of successes. Export to ClickHouse and S3.
ClickHouse schema. Create a spans table with columns that mirror GenAI semconv: gen_ai_system, gen_ai_request_model, input_tokens, output_tokens, latency_ms, prompt_hash, trace_id, parent_span_id, plus a JSON bag for long payloads. Add secondary indexes on user_id and app_id.
SDK coverage test. Write a small client app for each SDK (OpenAI, Anthropic, Google, LangChain, LlamaIndex, vLLM) and auto-instrument it with OpenLLMetry. Confirm each one produces canonical GenAI spans that land in ClickHouse.
Eval jobs. Schedule a job that reads the last 15 minutes of sampled traces and runs DeepEval faithfulness, toxicity, and answer relevance. The outputs are eval spans linked to the parent trace.
Custom LLM-judge. Build a PII-leak judge. Given a response, it calls a guard LLM to score the likelihood of a leak, and high-scoring responses go to a triage queue.
Drift detection. A weekly job computes PSI between this week's pooled prompt embeddings and the trailing 4-week baseline. If PSI is above the threshold, it alerts.
Dashboard. Build the Next.js 15 pages: overview (spans/sec, cost/user, p95 latency), traces (search and waterfall), evals (faithfulness trend, toxicity), drift (PSI over time), and alerts.
Alerting chain. A Prometheus exporter reads eval score aggregates and latency percentiles. Alertmanager routes warnings to Slack and critical breaches to PagerDuty.
Regression probe. Inject a bug so the evaluated chatbot leaks fake SSNs 1% of the time, then measure MTTR from the moment the bug is deployed to the Slack alert.

Check: Build It0/27

1. What does a tail-sampling processor in the collector do?Define

2. What is a tail-sampling policy?Define

3. What role does Alertmanager play in the alert chain?Define

4. A guard LLM scores a response as very likely to contain a PII leak. What does the pipeline do with it?Define

5. Why does the ClickHouse spans table carry columns such as gen_ai_system, token counts, latency_ms, prompt_hash and trace IDs?Explain

6. Why does the keep-or-drop decision happen after a trace completes instead of when its first span arrives?Explain

7. Why do both the sampling rule and the eval cadence affect whether a regression is caught in five minutes?Explain

8. The weekly job reports PSI 0.31 between this week's prompt embeddings and the trailing 4-week baseline. What should the dashboard do?Apply

9. In main.py, TailSampler has sample_rate 0.10. A trace has no error spans but contains an eval span with pii_leak 0.95. What does decide return?Apply

10. main.py's eval_faithfulness gets the response 'the weather in Tokyo is mild' and the context 'relevant weather context Tokyo mild'. What does it return?Apply

11. What does main.py's eval_pii_leak return for the response 'reach me at [email protected]'?Apply

12. main.py's eval_pii_leak is run on the response 'your ssn is 123-45-6789'. What does the stand-in judge return, and does the alerter fire on it?Apply

13. A client sends pricing.ts a span with request model claude-opus-4-7, 2,000 input tokens and 1,000 output tokens, prices of $15 and $75 per million. What does spanCostUsd return?Apply

14. A span has request model gpt-4o-mini and response model gpt-5.4, with 100,000 input and 20,000 output tokens. What cost does spanCostUsd compute?Apply

15. rollup.ts calls percentile on the sorted latencies [100, 200, 300, 400] with p = 0.5. What does it return?Apply

16. rollUpByModel gets three spans. A: request model gpt-4o-mini, no response model. B: request model gpt-4o-mini, response model gpt-5.4. C: request model gpt-5.4, status ERROR. What is the first row of the result?Apply

17. A spans.ts RingBuffer with capacity 3 receives pushes 1, 2, 3, 4, 5 in order. What does snapshot return?Apply

18. ObservabilityStore.ingest receives an array of three spans. One is valid, one has no gen_ai.system attribute, and one ends before it starts. What do the counters show?Apply

19. Your collector keeps all errored traces and 10% of successes. A bug leaks fake SSNs in 1% of otherwise successful responses. Why will the PII judge see few of the leaks?Analyze

20. Your eval job reads the last 15 minutes of traces once per window, and the target is an alert within 5 minutes of a bad deploy. What is the problem?Analyze

21. A keep-rule on pii_leak above 0.8 never fires in a pipeline where eval jobs score traces after the collector has sampled them. Why?Analyze

22. A sampler keeps 100% of traces with toxicity above 0.5 and 10% of the rest. The average toxicity of stored traces looks much higher than reality. Why?Analyze

23. The cost-per-user chart shows $0 for a user who made hundreds of calls through a model missing from the price table. What is the cause?Analyze

24. The team reports customer email addresses leaking in responses, yet no PII alert ever fires. What does main.py's design explain?Analyze

25. You must catch leaks of a new PII type within five minutes without storing every trace. Which design fits?Design

26. You need evidence that the dashboard meets the five-minute bar. Which test fits the lesson?Design

27. Finance wants dollar cost per user across all models. Which design fits the lesson?Design

Use It

$ curl -X POST https://my-otel-collector/v1/traces -d @trace.json
[collector]  accepted 1 trace, 3 spans
[clickhouse] inserted 3 spans (app=chat, user=u_42)
[eval]       DeepEval faithfulness 0.82, toxicity 0.03
[drift]      weekly PSI 0.08 (below 0.2 threshold)
[ui]         live at https://obs.example.com

Check: Use It0/2

1. Running main.py prints 'ingested spans: 165'. Each kept trace stores a root, an llm_call and an eval span. How many traces did the sampler keep?Apply

2. main.py prints 'PSI (current vs baseline): 0.083' with a 0.2 threshold. What does it print next?Apply

Ship It

outputs/skill-llm-observability.md is the deliverable. Point the dashboard at an LLM application and it ingests the traces, runs evals, alerts on drift, and shows a cost/user breakdown in Next.js.

Weight	Criterion	How it is measured
25	Trace-schema coverage	Number of SDK families producing canonical GenAI spans (target: 6+)
20	Eval correctness	DeepEval / RAGAS scores vs hand-labeled set
20	Dashboard UX	MTTR on injected regression (under 5 minutes target)
20	Cost / scale	Sustained ingest at 1k spans/sec without backlog
15	Alerting + drift detection	Prometheus/Alertmanager chain exercised end to end
100

Check: Ship It0/2

1. How does the rubric score trace-schema coverage, the heaviest criterion at 25 points?Define

2. You swap DeepEval for Phoenix evaluators and need to know whether the new scores can be trusted. Which check fits the lesson's rubric?Design

Author's Note

My bet is that the sampler in step 1 and the probe in step 9 will collide, so I'd test them against each other early. The sampler keeps every errored trace but only 10% of successes, and a response that leaks a fake SSN is not an error. It is a successful call with bad content, which means it sits in the 10% bucket and most leaks never reach the PII judge from step 5. Exercise 5 shows the kind of keep-rule that changes this, but it keys on toxicity, which an SSN leak won't trip, so you would need a rule on a PII signal instead.

I'd also look hard at the timing before building the dashboard pages. Step 4 reads the last 15 minutes of sampled traces, and the Problem section sets the bar at an alert in under five minutes. The lesson never says how often that job runs, but if it runs once per window, the window alone can blow the bar, and the Dashboard UX row in the rubric is scored on that MTTR. I'd settle the eval cadence first and polish the charts after.

Exercises

Add custom instrumentation for the Haystack framework. Verify that canonical spans land in ClickHouse with faithful gen_ai.* attributes.
Swap DeepEval for Phoenix evaluators on the same traces. Measure the score drift between the two eval engines.
Sharpen the drift detector by computing PSI per app-id instead of globally. Show per-app drift trails.
Add a "user impact" page with cost-per-user and failure-rate-per-user sparklines.
Build a tail-sampling policy that keeps 100% of traces with toxicity > 0.5 plus a 10% stratified sample of the rest. Measure the sampling bias it introduces.

Key Terms

Term	What people say	What it actually means
GenAI semconv	"OTel LLM attributes"	The 2025 OpenTelemetry spec for LLM span attributes (system, model, tokens)
Tail sampling	"Post-trace sample"	The collector decides whether to keep or drop a trace after it completes, so it can see errors first
PSI	"Population stability index"	A drift metric that compares two distributions; > 0.2 typically signals meaningful drift
LLM-judge	"Eval as model"	An LLM that scores another LLM's output against a rubric (faithfulness, toxicity, PII)
Tail-sampling policy	"Keep-rule"	The rule that decides which traces to persist and which to drop; errored traces plus a sample rate
Eval span	"Linked eval trace"	A child span that carries an eval score and links back to the original LLM call span
Cost per user	"Unit economics"	Dollar cost attributed to a user_id over a window; a key product metric