Capstone 85 — Content Classifier Integration

An output-side classifier is a check that runs on the model's response before it reaches the user. It sees what the model actually said, not what the user asked, and answers one question: regardless of how this prompt got here, is what we are about to ship acceptable?

That question matters because inputs are not the only attack surface. A model that passed its input checks can still produce an output that leaks PII, repeats slurs from its training distribution, or echoes the system prompt back in response to a clever question.

Teams often skip output classification because input classification feels like enough, and because output classifiers add latency. I don't think either argument holds up. Skipping the output side gives an attacker a one-shot bypass: any new attack family the input pipeline misses can land on the user. Latency is a real cost. In production, classifiers can run in parallel with token streaming, with the gate buffering the final chunk and applying the classifier verdict before flush. This demo is simpler: it classifies the finished text, one classifier after another.

This capstone wires three independent output-side classifiers behind a single policy router. Toxicity is rule-based harassment detection over a short list of insult phrases. PII uses regexes for emails, phone numbers, SSN-shaped strings, credit-card-shaped strings, and IP addresses. Instruction leakage is a heuristic for system prompt echo that compares the output to a known system prompt by cosine similarity over character trigrams. The router collects the verdicts, picks a severity, and applies an action policy: block, redact, warn, or log.

A policy router turns the verdicts from several classifiers into one action. Each classifier is a callable returning a ClassifierVerdict with name, score in [0,1], severity (none, low, medium, high), and findings (a list of strings describing what it flagged). The router takes a list of verdicts and applies a rule table:

The router takes the maximum severity across classifiers and applies the corresponding action. Block wins. A redact plus a warn becomes a redact, and a log plus a warn becomes a warn. The result is an Action object with verb, output, severity, verdicts, and metadata. Downstream, the safety gate in lesson 87 writes the metadata into a trace and does one of three things: ships the redacted output, ships the original with a warning, or swaps in a policy refusal.

Each classifier has its own redactor. The PII redactor replaces emails with [redacted-email], phone numbers with [redacted-phone], SSN-shaped strings with [redacted-ssn], and IPv4 addresses with [redacted-ip]. It replaces a card-shaped run (13 to 19 digits) with [redacted-card] only if the digits pass the Luhn checksum, so random digit strings survive. The instruction-leakage redactor takes the first line of the system prompt as a marker and, if that marker appears in the output, replaces it case-insensitively with [redacted-system-prompt]. The toxicity redactor replaces matched terms with [redacted-language]. The router runs a classifier's redactor whenever that classifier's own verdict is above none, so an output flagged for both toxicity and PII passes through both. Redaction only happens when the maximum severity is medium: at high the output is swapped for the refusal, and at low the original ships with a note.

Each classifier maps its findings to a severity with a small rule table:

The toxicity classifier is rule-based on purpose. It matches a curated list of ten insult and harassment phrases with regex \b word boundaries, plus a negation check: if not, never, or no appears in the 20 characters to the left of a match, the match is skipped, so "you are not stupid" does not trip the rule. The list is deliberately short because the point here is plumbing, not lexicon-building. The PII classifier uses standard regexes for the common shapes. The instruction-leakage classifier takes a system_prompt at construction and computes cosine similarity between the character-trigram counts of the prompt and of the output. A score at or above the 0.55 threshold is the leakage signal.

code/classifiers.py defines all three classifiers. Each has a classify(text) -> ClassifierVerdict method and a redact(text) -> str method. code/main.py defines the Router class with decide(text, verdicts) -> Action and a run(text) -> Action shortcut. The demo wires the three classifiers behind one router and runs a small corpus of crafted outputs that exercise each severity.

Run python3 main.py. The demo prints the action verb, severity, and fired classifiers for each test output and writes classifier_report.json into an outputs/ folder one level above the folder holding main.py. It asserts nothing, but the six fixtures cover all four verbs: log, warn, redact, and block. The highest-severity fixture that fires is a Luhn-valid card (PII), not a toxicity case, and the instruction-leakage fixture lands on block too. python3 tests.py is where the verbs and redaction tags are actually asserted. Latency is artificially zero because all the classifiers are rule-based. With a real model and neural classifiers the same plumbing applies, but the per-classifier latency goes up.

outputs/skill-content-classifier-integration.md documents the verdict and action structures so the gate in lesson 87 can consume them.

1. Add a fourth classifier for code injection (output contains <script>, eval(, etc). Decide its severity policy and integrate it.
2. Make the router apply a per-classifier severity weight so PII counts more than toxicity. Demonstrate the change on the same fixtures.
3. Add a confidence threshold so low-score verdicts downgrade by one severity level. Sweep the threshold and report how block rate changes.

Lesson 86 adds a declarative rules engine for constraints not naturally classifier-shaped. Lesson 87 composes both with the input-side detector.