Capstone 86 — Constitutional Rules Engine

A constitutional rules engine loads a declarative file of rules, evaluates each rule against a candidate output, and returns a structured result for every rule that fired. Classifiers cover the recognizable failures. Rules engines cover the contractual ones. A team writing a coding assistant wants a constraint like "every response that contains code must end in either a runnable block or a stated assumption." A team running a customer support bot wants "every refusal must offer a next step." Neither is a natural classifier target. Each is a predicate over the response, the conversation, and the system policy, and a non-engineer needs to be able to read it.

A declarative file fits that job. A constitution lives in YAML alongside the code, in version control, with a separate review process. Each rule has a name, a predicate, a severity, and an explanation template. The engine loads the file, evaluates each rule against the candidate output, and returns a structured Violation per rule that fired. Predicates compose with all_of, any_of, and not_, so a single rule can express "if the response contains code, it must end with a runnable block AND not reference an internal-only library."

Revision is the second half. An engine that only blocks leaves the work to someone else. An engine that proposes a fix closes the loop: the assistant drafts a response, the engine flags violations, a fixer produces a revised response, and the engine checks that the revision satisfies the rules. This capstone ships a minimal fixer (regex replacement per rule) and a structured diff (line-by-line additions, removals, edits) between draft and revised.

A rule has this shape:

- name: end-with-runnable-or-assumption
  severity: medium
  applies_when:
    contains_regex: '```python'
  must:
    any_of:
      - ends_with_regex: '```\s*$'
      - contains_regex: 'assumption:'
  explanation: "Code responses must end in either a closing fence or an explicit assumption."
  fix:
    append_if_missing: "\n\nAssumption: example inputs are valid."

name, explanation, must, and a severity of low, medium, or high are required, and the engine refuses to load a rule without them. applies_when and fix are optional.

Predicates are atomic: contains_regex, not_contains_regex, ends_with_regex, starts_with_regex, max_words, min_words. Compositions are all_of, any_of, not_. Each predicate is a small mapping, and the engine evaluates it recursively into a pair: whether it held, and the matched span if there is one. The engine evaluates applies_when first. If the rule does not apply, the result is recorded as not_applicable. Otherwise the engine evaluates must and records either pass or violation.

Severities are low, medium, high, the same three levels lesson 85 uses for classifier verdicts. The engine itself blocks nothing. It reports max_severity() for a draft, and a router such as lesson 87's gate could treat a high rule violation the same as a high classifier verdict and block.

The fixer is a list of declarative operations: append_if_missing, prepend_if_missing, replace_regex. Each operation maps a rule by name to a transform, and only rules that carry a fix key are registered with the fixer. A violation of any other rule is reported but left unrepaired. The fixer is limited to local edits on purpose. Structural rewrites belong in a separate refusal-and-help layer that this capstone does not cover.

The diff compares the original against the revised text and returns a list of Change records, each with an op (add, remove, edit-removed, or edit-added) and the relevant line of text. An edited line shows up as a removed line plus an added line. A caller can log the diff, which gives a human reviewer a way to audit the fixer's behavior over time.

code/rules.yml holds the constitution. The loader in code/main.py reads rules.yml through yaml_subset.load_yaml, which uses PyYAML when it is installed and otherwise falls back to a small bundled parser for the YAML subset the constitution uses, so it runs on a standard Python install with no dependencies. Both paths parse the shipped rules.yml to the same structure. code/tests.py covers that parser and the default constitution. code/main.py defines the Engine and Fixer classes and a diff function. Compositions are evaluated recursively, and any_of short-circuits.

The constitution as shipped:

• no-empty-refusal (medium) - a refusal must include either a suggestion or a redirect
• end-with-runnable-or-assumption (medium) - code responses must close cleanly
• no-pii-in-examples (high) - example data must not contain emails or phone shapes
• cite-when-asserting-fact (low) - lines beginning with "According to", "studies show" or "research finds" must contain a parenthetical citation
• no-internal-library-leak (high) - the words internal-only and policybot-internal must not appear in the output
• bounded-length (low) - responses must not exceed 800 words

Run python3 main.py. The demo runs five draft responses (refusal-without-redirect, code-without-assumption, pii-in-example, internal-library-leak, and clean-response) through the engine, prints the violations, runs the fixer, prints the diff, and writes rules_report.json into an outputs/ folder next to the code directory. One fixture has a rule that does not apply (the draft has no code block), and the report marks that rule not_applicable, so you can see the engine evaluated it explicitly.

outputs/skill-constitutional-rules-engine.md documents the rule grammar and the fixer operations.

1. Add a rule that requires a response to include the phrase "If this is urgent" when the prompt mentions safety. Use composition.
2. Replace the regex fixer with a templating fixer that takes named slots. Rewrite one rule under the new design to show it working.
3. Add a metrics endpoint that takes a corpus of drafts and returns the per-rule violation rate, so the team can see which rule is over-firing.

Lesson 87 composes this engine with the input-side detector and the output-side classifier into a single safety gate.