Structured Output — JSON Schema, Pydantic, Zod, Constrained Decoding

An agent reading a purchase-order email needs to turn free text into {customer, line_items, total_usd}. There are three ways to do it.

Approach one: prompt for JSON. "Reply in JSON with fields customer, line_items, total_usd." This works 85 to 95 percent of the time on frontier models. The rest fail in six ways: a missing brace, a trailing comma, wrong types, hallucinated fields, truncation at the token limit, and leaked prose like "Here is your JSON:".

Approach two: validate after generation. Generate freely, parse, validate against the schema, retry on failure. It is reliable but you pay for every retry, and each truncation bug costs one extra turn.

Approach three: constrained decoding. The provider enforces the schema at decode time. Invalid tokens are masked out of the sampling distribution, so the output parses and validates. Failure collapses to one mode, refusal, where the model decides the input does not fit the schema.

Every 2026 frontier provider ships some form of approach three.

• OpenAI. response_format: {type: "json_schema", strict: true}, plus a refusal in the response if the model declines.
• Anthropic. Schema enforcement on tool_use inputs. There is no stop_reason: "refusal"; the signal is end_turn with no tool call.
• Gemini. responseSchema at request level. In 2026 Gemini ships token-level grammar constraints for selected types.
• Pydantic AI. output_type=InvoiceModel returns a RunResult typed to InvoiceModel.
• Zod (TypeScript). A runtime parser that validates provider output against a Zod schema. It pairs with OpenAI's beta.chat.completions.parse.

Declare the schema once and enforce it end to end.

JSON Schema 2020-12, the lingua franca

Every provider accepts JSON Schema 2020-12. These are the constructs you will reach for most:

• type: one of object, array, string, number, integer, boolean, null.
• properties: map of field name to subschema.
• required: list of field names that must appear.
• enum: closed set of allowed values.
• minimum / maximum (numbers), minLength / maxLength / pattern (strings).
• items: subschema applied to every array element.
• additionalProperties: false forbids extra fields (default varies by mode).

OpenAI strict mode adds three requirements: every property must be listed in required, additionalProperties: false must be set at every object level, and no $ref can be left unresolved. Break one and the API returns a 400 at request time.

Pydantic, the Python binding

Pydantic v2 generates JSON Schema from dataclass-shaped models via model_json_schema(). Pydantic AI wraps this so you write:

class Invoice(BaseModel):
    customer: str
    line_items: list[LineItem]
    total_usd: Decimal

and the agent framework translates the schema into OpenAI strict mode, Anthropic input_schema, or Gemini responseSchema at the edge. The output comes back as a typed Invoice instance, and validation errors raise ValidationError with typed error paths.

Zod, the TypeScript binding

Zod (z.object({customer: z.string(), ...})) is the TS equivalent. OpenAI's Node SDK exposes zodResponseFormat(Invoice), which translates to the API's JSON Schema payload.

Refusals

Strict mode cannot force the model to answer. If the input cannot fit the schema ("the email was a poem, not an invoice"), the model emits a refusal field with the reason. Handle it as a first-class outcome, not a failure. It doubles as a safety signal: a model asked to extract a credit card number from a protected-content email returns a refusal with the safety reason attached.

Constrained decoding in the open

Open-weights implementations use three techniques.

1. Grammar-based decoding (outlines, guidance, lm-format-enforcer): build a deterministic finite automaton from the schema; at every step, mask the logits of tokens that would violate the FSM.
2. Logit masking with a JSON parser: run a streaming JSON parser in lockstep with the model; at every step, compute the valid-next-token set.
3. Speculative decoding with a verifier: cheap draft model proposes tokens, verifier enforces the schema.

Commercial providers pick one of these behind the scenes. The 2026 state of the art is faster than plain generation for short structured outputs and roughly the same speed for long ones.

The three failure modes

1. Parse error. The output is not valid JSON. Strict mode rules this out, but non-strict providers can still produce it.
2. Schema violation. The output parses but breaks the schema. Strict mode rules this out too. Outside it, this one is common.
3. Refusal. The model declines. Handle it as a typed outcome.

Retry strategy

When you are outside strict mode (Anthropic tool use, non-strict OpenAI, older Gemini), the recovery pattern is:

generate -> parse -> validate -> if fail, inject error and retry, max 3x

One retry usually fixes it. Three catch the flakes from weaker models. If you need more than three, suspect the schema: the model cannot satisfy it for some inputs, and the prompt or the schema needs fixing.

Small-model support

Constrained decoding works on small models. A 3B-parameter open model with grammar enforcement outperforms a 70B-parameter model with raw prompting on structured tasks. That is the main reason structured outputs matter in production: reliability stops depending on model size.

main.py ships a minimal JSON Schema 2020-12 validator in stdlib (types, required, enum, min/max, pattern, items, additionalProperties). It wraps an Invoice schema and runs a fake LLM output through the validator, which exercises the parse error, schema violation, and refusal paths. In production, swap the fake output for any provider's real response.

What to look at:

• The validator returns a typed [ValidationError] list with a path and a message. That is the shape you want to feed into the retry prompt.
• The refusal branch does not retry. It logs and returns a typed refusal. Phase 14 · 09 uses refusals as a safety signal.
• The additionalProperties: false check fires on the adversarial test input, which shows why strict mode shuts the door on hallucinated fields.

This lesson produces outputs/skill-structured-output-designer.md. Give it a free-text extraction target (invoices, support tickets, resumes) and it returns a strict-mode-compatible JSON Schema 2020-12 and a matching Pydantic model, with typed refusal and retry handling stubbed in.

1. Run main.py. Add a sixth test case whose total_usd is a negative number. Confirm the validator rejects it with the minimum constraint path.

2. Extend the validator to support oneOf with a discriminator. The common case: line_item is either a product or a service, tagged by kind. Strict mode has subtle rules here; check OpenAI's structured outputs guide.

3. Write the same Invoice schema as a Pydantic BaseModel and compare model_json_schema() output to your hand-rolled schema. Identify the one field Pydantic sets by default that the hand-rolled version omits.

4. Measure refusal rates. Construct ten inputs that should not be extractable (a song lyric, a math proof, a blank email) and run them through a real provider with strict mode. Count refusals vs hallucinated outputs. This is your ground truth for refusal-aware retries.

5. Read OpenAI's structured outputs guide top to bottom. Identify the one construct it explicitly forbids in strict mode that plain JSON Schema allows. Then design a schema that uses the forbidden construct non-essentially and refactor it to be strict-compatible.

• OpenAI: Structured outputs: strict mode, refusals, and schema requirements
• OpenAI: Introducing structured outputs: August 2024 launch post explaining the decoding guarantee
• Pydantic AI: Output: typed output_type bindings that serialize to each provider
• JSON Schema: 2020-12 release notes: the canonical spec
• Microsoft: Structured outputs in Azure OpenAI: enterprise deployment notes and strict-mode caveats