Structured Outputs & Constrained Decoding

A classifier prompts an LLM: "Return one of {positive, negative, neutral}." The model returns "The sentiment is positive — this review is overwhelmingly favorable because the customer explicitly states that they ...". Your parser crashes. Your classifier's F1 is 0.0.

Free-form generation is not a contract. It is a suggestion. A production system needs a contract.

Three layers exist in 2026.

1. Prompting. Ask nicely. "Return only the JSON object." Works ~80% on frontier models, less on smaller ones.
2. Native structured output APIs. OpenAI response_format, Anthropic tool use, Gemini JSON mode. Reliable on supported schemas. Vendor-locked.
3. Constrained decoding. Modify the logits at every generation step so the model cannot emit invalid tokens. 100% valid by construction. Works on any local model.

This lesson builds intuition for all three, and tells you exactly when to reach for each.

Constrained decoding masks invalid tokens before the model can ever sample them. At each generation step, the LLM produces a logit vector over the full vocabulary (~100k tokens). A logit processor sits between the model and the sampler: it checks which tokens are valid given the current position in the target grammar (JSON Schema, regex, context-free grammar) and sets every invalid token's logit to negative infinity. Softmax over what's left puts all the probability mass on valid continuations, so the model can't sample its way into broken output.

Implementations in 2026:

• Outlines. Compiles JSON Schema or regex into a finite-state machine. Every token gets an O(1) valid-next-token lookup. FSM-based, so recursive schemas need flattening.
• XGrammar / llguidance. Context-free grammar engines. Handle recursive JSON Schema. Near-zero decoding overhead. OpenAI credited llguidance in their 2025 structured output implementation.
• vLLM guided decoding. vLLM, a popular open-source inference server, ships guided_json, guided_regex, guided_choice, guided_grammar built in, via Outlines, XGrammar, or lm-format-enforcer backends.
• Instructor. A wrapper over any LLM built on Pydantic (Python's schema-validation library). Retries on validation failure. Cross-provider, but it never touches logits: it leans on retries plus structured-output-aware prompts instead.

The counterintuitive result

Constrained decoding is often faster than unconstrained generation, for two reasons. It shrinks the next-token search space, and clever implementations skip token generation entirely for forced tokens (scaffolding like {"name": ", where every byte is already determined).

The pitfall that costs you

Field order matters here. Put answer before reasoning, and the model commits to an answer before it has thought anything through. The JSON still validates. The answer is still wrong, and no validator catches it, because nothing about that failure looks like a schema violation.

// BAD
{"answer": "yes", "reasoning": "because ..."}
 
// GOOD
{"reasoning": "... therefore ...", "answer": "yes"}

Schema field order is logic, not formatting.

Step 1: regex-constrained generation from scratch

See code/main.py for a standalone FSM implementation. The core idea in 30 lines:

def mask_logits(logits, valid_token_ids):
    mask = [float("-inf")] * len(logits)
    for tid in valid_token_ids:
        mask[tid] = logits[tid]
    return mask
 
 
def generate_constrained(model, tokenizer, prompt, fsm):
    ids = tokenizer.encode(prompt)
    state = fsm.initial_state
    while not fsm.is_accept(state):
        logits = model.next_token_logits(ids)
        valid = fsm.valid_tokens(state, tokenizer)
        logits = mask_logits(logits, valid)
        tok = sample(logits)
        ids.append(tok)
        state = fsm.transition(state, tok)
    return tokenizer.decode(ids)

The FSM tracks what parts of the grammar we have satisfied so far. valid_tokens(state, tokenizer) computes which vocabulary tokens can advance the FSM without leaving an accepting path.

Step 2: Outlines for JSON Schema

from pydantic import BaseModel
from typing import Literal
import outlines
 
 
class Review(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    confidence: float
    evidence_span: str
 
 
model = outlines.models.transformers("meta-llama/Llama-3.2-3B-Instruct")
generator = outlines.generate.json(model, Review)
 
result = generator("Classify: 'The wait staff was attentive and the food arrived hot.'")
print(result)
# Review(sentiment='positive', confidence=0.93, evidence_span='attentive ... hot')

Zero validation errors. Ever. The FSM makes invalid output unreachable.

Step 3: Instructor for provider-agnostic Pydantic

import instructor
from anthropic import Anthropic
from pydantic import BaseModel, Field
 
 
class Invoice(BaseModel):
    vendor: str
    total_usd: float = Field(ge=0)
    line_items: list[str]
 
 
client = instructor.from_anthropic(Anthropic())
invoice = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    response_model=Invoice,
    messages=[{"role": "user", "content": "Extract from: 'Acme Corp $420. Widget, Gizmo.'"}],
)

Different mechanism: Instructor never edits a logit. It formats the schema into the prompt, parses the output, and retries on validation failure (default 3 times). Works with any provider, but retries add latency and cost. Cross-provider portability is the selling point.

Step 4: native vendor APIs

from openai import OpenAI
 
client = OpenAI()
response = client.responses.create(
    model="gpt-5",
    input=[{"role": "user", "content": "Classify: 'The food was cold.'"}],
    text={"format": {"type": "json_schema", "name": "sentiment",
          "schema": {"type": "object", "required": ["sentiment"],
                     "properties": {"sentiment": {"type": "string",
                                                  "enum": ["positive", "negative", "neutral"]}}}}},
)
print(response.output_parsed)

Server-side constrained decoding. Reliability parity with Outlines for supported schemas. No local model management. Locks you to the vendor.

• Recursive schemas. Outlines flattens recursion to a fixed depth. Tree-structured outputs (nested comments, AST) need XGrammar or llguidance (CFG-based).
• Huge enums. 10,000-option enum compiles slowly or times out. Switch to a retriever: predict top-k candidates first, constrain to those.
• Grammar too strict. Force a date: "YYYY-MM-DD" regex and the model can't output "unknown" for a missing date, so it fabricates one instead. Allow null or a sentinel.
• Premature commitment. See field-order pitfall above. Always put reasoning first.
• Vendor JSON mode without schema. Pure JSON mode only guarantees valid JSON, not valid for your use case. Always provide a full schema.

The 2026 stack:

Save as outputs/skill-structured-output-picker.md:

---
name: structured-output-picker
description: Choose a structured output approach, schema design, and validation plan.
version: 1.0.0
phase: 5
lesson: 20
tags: [nlp, llm, structured-output]
---
 
Given a use case (provider, latency budget, schema complexity, failure tolerance), output:
 
1. Mechanism. Native vendor structured output, Instructor retries, Outlines FSM, or XGrammar CFG. One-sentence reason.
2. Schema design. Field order (reasoning first, answer last), nullable fields for "unknown", enum vs regex, required fields.
3. Failure strategy. Max retries, fallback model, graceful `null` handling, out-of-distribution refusal.
4. Validation plan. Schema compliance rate (target 100%), semantic validity (LLM-judge), field-coverage rate, latency p50/p99.
 
Refuse any design that puts `answer` or `decision` before reasoning fields. Refuse to use bare JSON mode without a schema. Flag recursive schemas behind an FSM-only library.

1. Easy. Prompt a small open-weights model (e.g., Llama-3.2-3B) without constrained decoding for Review(sentiment, confidence, evidence_span). Measure the fraction that parse as valid JSON on 100 reviews.
2. Medium. Same corpus with Outlines JSON mode. Compare compliance rate, latency, and semantic accuracy.
3. Hard. Implement a regex-constrained decoder from scratch for phone numbers (\d{3}-\d{3}-\d{4}). Verify 0 invalid outputs on 1000 samples.

• Willard, Louf (2023). Efficient Guided Generation for LLMs: the Outlines paper.
• XGrammar paper (2024): fast CFG-based constrained decoding.
• vLLM: Structured Outputs: inference server integration.
• OpenAI: Structured Outputs guide: API reference + gotchas.
• Instructor library: Pydantic + retries across providers.
• JSONSchemaBench (2025): benchmarking 6 constrained decoding frameworks.