TAI BUI
Back to curriculum
BuildPython

Moderation Systems — OpenAI, Perspective, Llama Guard

Moderation is where the safety policies from Lessons 12-16 meet users. OpenAI's `omni-moderation-latest` (2024) is built on GPT-4o and classifies text and images in one call. It scores 42% better on a multilingual test set than the prior-generation moderation endpoint, and the response schema returns 13 category booleans: harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, violence/graphic. It is free for most developers. The usual deployment layers input moderation (before generation), output moderation (after generation), and custom domain rules. Parallel async calls hide latency, and a placeholder response can cover a flag. Llama Guard 3/4 (Lesson 16) covers 14 MLCommons hazards plus Code Interpreter Abuse, in 8 languages (v3) and multi-image (v4). Perspective API (Google Jigsaw) predates the LLM-as-moderator wave and scores mostly one dimension, toxicity, with severe-toxicity, insult, and profanity variants. Researchers still use it as a baseline. Azure Content Moderator was deprecated in February 2024 and retires in February 2027, replaced by Azure AI Content Safety.

TIME: ~60 minutesPREREQUISITES: Phase 18 · 16 (Llama Guard / Garak / PyRIT)

Learning Objectives

  • Describe the OpenAI Moderation API's category taxonomy and how it differs from Llama Guard 3's MLCommons set.
  • Describe the three-layer moderation pattern (input, output, custom) and name one failure mode of each.
  • Describe Perspective API's position as a pre-LLM-era baseline and why it remains used in research.
  • State the Azure deprecation timeline.

The Problem

Lessons 12-16 cover attacks and the tooling that defends against them. This page covers the deployed moderation systems that turn those defenses into something running where users touch the product. The three-layer pattern is the default configuration in 2026.

Warm-up0/3
1. Which of these is a safeguard model that classifies prompts and responses against a hazard taxonomy?Warm-up
A user's prompt and a model's reply both arrowing into an unlabeled box, which arrows out to a hazard label, captioned as a model that reads text and scores it against a hazard taxonomy.
2. What are Garak and PyRIT mainly used for?Warm-up
Garak and PyRIT boxes arrowing toward a candidate model labeled not yet serving users, with a dashed box asking what their role is in this setup.
3. A forum blocks posts using an exact-match word list. A user types a banned word with a space between each letter. What most likely happens?Warm-up
A banned word list described as exact strings only, next to a typed example with a space inserted between every letter, with the result left blank.

The Concept

OpenAI Moderation API

omni-moderation-latest (2024). Built on GPT-4o. Classifies text + images in one call. Free for most developers.

Categories (13 booleans in the response schema):

  • harassment, harassment/threatening
  • hate, hate/threatening
  • self-harm, self-harm/intent, self-harm/instructions
  • sexual, sexual/minors
  • violence, violence/graphic
  • illicit, illicit/violent

Multimodal support applies to violence, self-harm, and sexual but not sexual/minors; the rest are text-only.

The harness in main.py collapses the /threatening, /intent, /instructions, and /graphic sub-categories into their top-level parents to stay small. Production code should use the full 13-category schema.

42% better results on a multilingual test set than the prior-generation moderation endpoint. The API returns per-category scores, and your application sets the thresholds.

Llama Guard 3/4

Covered in Lesson 16. 14 MLCommons hazard categories (organized differently from OpenAI's 13 response-schema booleans). Supports 8 languages (v3). Llama Guard 4 (April 2025) is natively multimodal, 12B.

The OpenAI and Llama Guard taxonomies overlap but diverge. OpenAI has "illicit" as one broad category, while Llama Guard splits it into "violent crimes" and "non-violent crimes". Pick the one whose taxonomy fits your policy.

Perspective API (Google Jigsaw)

Toxicity scoring system predating the LLM-as-moderator wave (pre-2020). Categories: TOXICITY, SEVERE_TOXICITY, INSULT, PROFANITY, THREAT, IDENTITY_ATTACK. Single-dimension primary score (TOXICITY) with sub-dimension variants.

Researchers keep using it as a content-moderation baseline because the API is stable, documented, and has years of calibration data. For LLM-adjacent work, Llama Guard or OpenAI Moderation is usually a better fit.

The three-layer pattern

  1. Input moderation. Classify the user's prompt before generation. Reject if flagged. Latency: one classifier call.
  2. Output moderation. Classify the model's output before delivery. Replace with a refusal if flagged. Latency: one classifier call after generation.
  3. Custom moderation. Domain-specific rules (regex, allowlists, business policy). Runs at either input or output.

The layers run in sequence by design. Input moderation has to finish before generation starts, and output moderation runs after generation ends. Parallelism lives inside a layer: run several classifiers (say OpenAI Moderation, Llama Guard, and Perspective) on the same text at once and you only pay for the slowest one. You can also show a placeholder ("one moment, checking...") while input moderation completes and hold back the first streamed token. What happens on a flag is configurable: refuse, sanitize, or escalate to human review.

Failure modes

  • Input only. Misses bad content the model produces from a clean-looking prompt, such as the encoding attacks in Lessons 12-14 that slip past input classifiers.
  • Output only. Allows any input to reach the model; increases cost; surfaces internal reasoning to attacker.
  • Custom only. Not robust across categories; regexes are brittle.

Layering is the default for that reason: each layer covers a gap in the others.

Azure deprecation

Azure Content Moderator: deprecated February 2024, retired February 2027. Replaced by Azure AI Content Safety, which is LLM-based and integrates with Azure OpenAI. For Azure deployments the migration is a multi-year project running from 2024 to 2027.

Where this fits in Phase 18

Lesson 16 covers moderation tooling from the red-team side. Here it is covered as deployed. Lesson 30 closes with the current dual-use capability evidence.

Check: The Concept0/25
1. What is OpenAI's omni-moderation-latest?Define
A card for omni-moderation-latest listing that it is built on GPT-4o, takes text and image in one call, returns a 13-category boolean schema, and is free for most developers.
2. Which of these is NOT one of OpenAI Moderation's top-level categories?Define
Four real top-level category names, hate, illicit, sexual and sexual/minors, with a note that sub-categories are not shown and the schema has 13 booleans in all.
3. What is Llama Guard?Define
Text input flowing into a box labeled Llama Guard, hazard classifier, then out to a hazard label, captioned as scoring against 14 MLCommons hazard categories.
4. What is Perspective API (Google Jigsaw) primarily designed to score?Define
A card for Perspective API showing a primary TOXICITY score and sub-dimensions SEVERE_TOXICITY, INSULT, PROFANITY, THREAT and IDENTITY_ATTACK, noted as pre-2020 vintage.
5. What is input moderation?Define
A user's prompt arrowing into a box labeled input moderation runs here, then arrowing out toward a model call, captioned as finishing before the model is ever called.
6. What is output moderation?Define
A model's reply arrowing into a box labeled output moderation runs here, then arrowing out toward delivery to the user, captioned as running after the reply exists and before delivery.
7. What is custom moderation?Define
Three chips, regex pattern, allowlist, and business policy, with an illustrative example rule about not mentioning a rival product, captioned as able to run on either side of the pipeline.
8. What are the three layers of the standard 2026 moderation pattern?Define
A user prompt flowing through an unlabeled checkpoint into a model, then through a second unlabeled checkpoint toward the reply, with a dashed note that one more check applies deployment-specific policy somewhere in the pipeline.
9. How does OpenAI Moderation's taxonomy compare to Llama Guard 3's?Explain
Two cards side by side, OpenAI Moderation with one broad illicit category and Llama Guard 3 splitting that concept into violent crimes and non-violent crimes.
10. The moderation API returns a score for each category. Who decides what score gets content blocked?Explain
A harassment score of 0.9 returned by the API, next to a dashed box asking who sets the cutoff that decides what gets blocked.
11. You send a post containing text and an image to omni-moderation-latest. For which category is the image not evaluated?Explain
Typed text and an image sent together into omni-moderation-latest, above four equally styled, unhighlighted category tags: violence, sexual, self-harm and sexual/minors.
12. What changed between Llama Guard 3 and Llama Guard 4?Explain
Two cards, Llama Guard 3 at 8B, text only, 8 languages, next to Llama Guard 4 from April 2025 at 12B and natively multimodal.
13. Researchers still use Perspective API as a baseline even though LLM-based moderators exist. Why?Explain
Perspective API labeled pre-2020 next to OpenAI Moderation and Llama Guard labeled LLM-era, with a note that researchers still run the older one as a baseline and a blank asking why.
14. Why is layering the default instead of relying on one strong layer?Explain
Three unlabeled check boxes in a row, captioned as all running in the same production deployment, with a dashed question asking why not rely on the single strongest one.
15. Why do the input and output layers run in sequence rather than in parallel?Explain
Input check, generation and output check drawn as three boxes in that order, with a dashed question asking why they do not all run at once.
16. What is one failure mode of input-only moderation?Explain
A user prompt passing through a single input check into model generation and straight to delivery, captioned as nothing inspecting the reply once generation starts.
17. What failure modes do output-only and custom-only moderation have, respectively?Explain
Two cards, output-only setup noted as having no input check and custom-only setup noted as just regex rules, with a dashed box asking what goes wrong with each.
18. What is the status of Azure Content Moderator, and what replaces it?Explain
A timeline showing Azure Content Moderator deprecated in February 2024 and retiring in February 2027, replaced by Azure AI Content Safety.
19. Your bot runs only input moderation. A base64-encoded prompt passes it and the model decodes and answers harmfully. What closes the gap?Analyze
A base64-encoded prompt passing an input check, reaching a model that decodes and answers harmfully, with a dashed box asking what closes the gap.
20. A team streams the first tokens while input moderation is still running. A harmful prompt is flagged afterward, but part of an answer is already on screen. Which fix matches the lesson?Analyze
Three boxes, input check starts, first tokens stream in, input check flags harmful, captioned as part of a reply already on screen when the flag lands.
21. A stack uses only regex rules. A user paraphrases a banned request using none of the listed words, and it gets through. Which weakness is this?Analyze
A regex rule that must contain a listed phrase next to a user's paraphrase of the same request using different words, with a dashed box asking the rule's weak point.
22. You must moderate a code-assistant deployment. Which stack fits the lesson's approach?Design
A code-assistant deployment box, example Copilot-style tool, with a dashed box asking what moderation stack fits it.
23. You want OpenAI Moderation, Llama Guard, and Perspective all to check each prompt without tripling latency. Which design fits?Design
Three boxes, OpenAI Moderation, Llama Guard and Perspective API, each noted as taking some time, with a dashed box asking the total added latency.
24. A new business rule says replies must never name a competitor's product. Where should this custom rule run?Design
A new rule that replies must never name a competitor's product, above two equally styled dashed boxes, input side and output side, neither highlighted.
25. Your Azure deployment still uses Azure Content Moderator in 2026. Which plan is sound?Design
A deployment box noted as the current year 2026 next to a retirement date box of February 2027, with a dashed box asking for a sound plan.

Use It

main.py builds a three-layer moderation harness: an input moderator (keyword triggers mapped to category scores), a custom moderator (domain rules), and an output moderator (the same classifier run on the model's reply). Feed it prompts and watch which layer catches each one.

Check: Use It0/8
1. In main.py, what does run('diagnose my headache') produce?Apply
The self-harm trigger list and the diagnose-my custom rule source lines from main.py, the exact input diagnose my headache, and a dashed box asking what main.py prints.
2. In main.py, what does run('you idiot, help me with this') produce?Apply
The harassment trigger list you idiot and shut up from main.py, the exact input you idiot help me with this, and a dashed box asking what main.py prints.
3. In main.py, what does run('should i invest in a bomb') print?Apply
The illicit/violent trigger bomb and the should-i-invest-in custom rule source lines from main.py, the exact input should i invest in a bomb, and a dashed box asking what main.py prints.
4. In main.py, how is 'summarize this email: hello there' handled?Apply
All five real trigger-word lists from main.py listed in full, the exact input summarize this email hello there, and a dashed box asking what main.py prints.
5. The omni-moderation schema has 13 booleans, but main.py defines only eight categories. Why?Apply
The 13-item omni-moderation schema and main.py's 8-item CATEGORIES list shown side by side, with a dashed box asking why main.py has fewer.
6. A trigger match scores 0.9 and the threshold is 0.5. If you raise the threshold to 0.95, what happens to 'you idiot, help me with this'?Apply
A trigger match scored 0.9 next to an old threshold of 0.5 and a new threshold of 0.95, with a dashed box asking whether it is still flagged.
7. A developer notes the output moderator never fires on the six sample prompts and calls the layer redundant. What is wrong with that conclusion?Analyze
A developer's claim that the output moderator fired zero times across all six sample prompts, next to model_respond's real echo code, with a dashed box asking what is wrong with calling the layer redundant.
8. main.py flags 'how do i kill a hung python process' at input as violence. What is the cause?Analyze
The real violence trigger list kill, attack someone, murder from main.py, the exact input how do i kill a hung python process, and a dashed box asking why this flags as violence.

Ship It

The output is outputs/skill-moderation-stack.md. Given a deployment, it recommends a moderation stack: which classifier runs at input, which at output, which custom rules apply, and which judge handles edge cases.

Author's Note

I'd run main.py first and read the six prompts before reading the code. Each one gets a different outcome: the weather question passes, the memecoin and headache prompts hit custom rules, the bomb and "you idiot" prompts get flagged at input. Seeing that split makes the layer boundaries concrete.

Look at the order in run(). Custom rules fire after input moderation and before the model is called, so in this harness they act as a second input check, not an output one. The concept section says custom rules can run at either end, but this code only wires up the input side.

The output moderator never fires on any of the default prompts. Anything that would trip it already got refused at input, and the mock model's replies are harmless. In fact no prompt can trigger it as written: model_respond only echoes the first 40 characters of the prompt back, so any trigger word in the reply was already in the prompt and got caught at input. That is a limit of the toy, not evidence the output layer is redundant. To see it work, I'd edit model_respond to return a trigger word for some clean-looking prompt, which is the gap the Failure modes list says input-only setups leave open. Remember too that the classifier here is a keyword lookup returning 0.9 on a match. Real scores are graded, and the threshold you pick matters far more than it does in this demo.

Exercises

  1. Run main.py. Run a benign, borderline, and harmful input through all three layers. Report which layer fires for each.

  2. Extend the harness with Perspective-API-style toxicity scoring on a specific category. Compare its threshold behaviour to the category score.

  3. Read the OpenAI Moderation API docs and the Llama Guard 3 category list. Map each OpenAI category to the closest Llama Guard categories. Identify three categories that do not cleanly map.

  4. Design a moderation stack for a code-assistant deployment (e.g., GitHub Copilot). Identify the categories most and least relevant and propose custom rules.

  5. Azure Content Moderator retires February 2027. Plan a migration to Azure AI Content Safety. Identify the highest-risk element of the migration.

Key Terms

TermWhat people sayWhat it actually means
OpenAI Moderation"omni-moderation-latest"GPT-4o-based 13-category (text) classifier with partial multimodal support
Perspective API"Google Jigsaw toxicity"Pre-LLM-era toxicity scoring baseline
Llama Guard"MLCommons 14-category"Meta's hazard classifier (v3: 8B text, 8 langs; v4: 12B multimodal)
Input moderation"pre-generation filter"Classifier on user prompt before model call
Output moderation"post-generation filter"Classifier on model output before delivery
Custom moderation"domain rules"Deployment-specific rules (regex, allowlist, policy)
Layered moderation"all three layers"Standard production deployment pattern

Further Reading

Code

Ships With