Hypothesis Generator

A generate-then-filter pipeline samples several drafts, drops any that sit too close to one already accepted, and ranks the survivors into a queue. A planner that asks one model one time gets one hypothesis. That is fine for a worked example, but a research loop wants a queue with depth, so when the first hypothesis fails the runner has the next one ready without paying for another full sampling pass.

Two ideas combine to produce that queue. The first is temperature ramping: each pass through the sampler raises the temperature a notch, so later drafts are encouraged to wander. The second is novelty filtering: after each draft, the generator measures the embedding distance from every prior survivor and rejects anything inside the cluster.

The lesson ships a mock language model that returns scripted token sequences for fixed prompts. It is enough to exercise the full path: seed prompt in, temperature ramp applied, candidates parsed, novelty filter run, ranked queue out.

A Hypothesis is a typed record for one candidate claim and the fields a runner needs to test it.

Hypothesis
  id             : int           (monotonic within a run)
  text           : str           (the claim)
  variables      : list[str]     (what changes between conditions)
  metric         : str           (what the runner will measure)
  baseline_ref   : str | None    (which paper or run the comparison cites)
  draft_pass     : int           (which sampler pass produced this)
  temperature    : float         (the sampler setting at draft time)
  novelty_score  : float         (distance from prior survivors, 0..1)
  rank_score     : float         (weighted sum used for ordering)

variables and metric are not free text. The parser pulls them from a tagged response, and the runner in lesson fifty-two reads these fields directly when it builds the experiment config.

baseline_ref is optional but recommended. The evaluator in lesson fifty-three needs a baseline to compare against, and if the hypothesis omits one, the evaluator falls back to the previous run on the same metric.

The loop is straightforward. The interesting part is that each box has a hard contract.

The temperature ramp is a schedule of n_passes evenly spaced temperatures from t_min to t_max, with step (t_max - t_min) / (n_passes - 1). GeneratorConfig.schedule() produces the values, and each pass calls the sampler at the current one. The mock model honors temperature by switching between a small set of scripted responses keyed on (prompt, temp_bucket). The buckets are half-open ranges: t < 0.35 is bucket 0, t < 0.65 is bucket 1, t < 0.95 is bucket 2, and anything higher is bucket 3. A small change in temperature can therefore land in a different bucket and produce a different draft. In production the sampler would be a real model with temperature=t passed through.

The default schedule is six passes from 0.2 to 1.2. Six is enough to fill the queue without paying for samples that the novelty filter will reject anyway. With real models, a sampler below about 0.2 typically parrots the seed back, and above about 1.2 the responses tend to drift off topic and fail the parser. The mock has no such behavior; those edges are properties of real samplers, not of this code.

The novelty filter drops any draft that lands too close to a hypothesis the generator already accepted. After each draft is parsed, the generator embeds the text and compares it against every accepted hypothesis. The embedding is a small hashed bag of word tokens, normalised to unit length, and cosine distance between two unit vectors is 1 - dot(a, b). A draft passes if its minimum distance to any prior survivor is at least novelty_threshold, which defaults to 0.25. It is rejected as a duplicate when that distance is below the threshold.

The hashed embedding is plain: deterministic, zero dependencies, and enough to catch the obvious case of two drafts that share most of their nouns. A production deployment would swap in a small sentence model, and the interface stays the same.

The rank score is a weighted sum of three sub scores.

rank_score = w_novelty * novelty_score
           + w_specificity * specificity_score
           + w_testability * testability_score

novelty_score is the minimum embedding distance from prior survivors. specificity_score is the count of concrete variables in the hypothesis divided by a target count. testability_score is one if the hypothesis specifies both a metric and a baseline, half if it has only a metric, and zero otherwise.

Default weights are 0.4, 0.3, 0.3. They live in the generator config, so a downstream lesson can shift them without forking the code.

The mock language model is a scripted stand-in for a real sampler, with the signature below.

class MockLLM:
    def sample(self, prompt: str, temperature: float, seed: int) -> str:
        ...

It is deterministic given a (prompt, temperature, seed) triple. The mock keeps a scripted response table keyed on (prompt_signature, temperature_bucket). If the table has no entry for a key, the sampler returns a fallback that fails the parser, and one of the tests exercises that path.

The seed selects a draft from the scripted bank for that key (bank[seed % len(bank)]), so a bank with several entries gives different drafts for different seeds. In the demo every bank holds exactly one entry, so the seed has no effect on which draft comes back. The generator passes base_seed + pass_index as the seed, and tests pin base_seed to keep results reproducible.

run(seed_prompt) returns a tuple (survivors, logs). There is no separate queue object: survivors is a list of Hypothesis records sorted by (-rank_score, id), so higher rank comes first and ties go to the earlier id, and logs is a list of GenerationLog entries, one per pass, recording the temperature, seed, accepted id or reject reason. The runner in lesson fifty-two treats that survivors list as its queue and pops the head and runs the experiment, then the evaluator in lesson fifty-three writes a verdict back. If the verdict says the hypothesis was wrong, the runner pops the next one.

The queue is finite. When it is empty, the orchestrator can either widen the seed prompt and run the generator again or stop and report the budget exhausted.

code/main.py defines Hypothesis, MockLLM, HypothesisGenerator, and a deterministic demo. The generator exposes a single run(seed_prompt) method that returns a tuple (survivors, logs), with survivors sorted by rank; the pass count is read from GeneratorConfig.n_passes rather than passed as an argument. The embedding is a hashed bag of tokens. Novelty and rank are methods on the generator (_novelty and _score), not standalone functions. Nothing depends on numpy; the embedding math is pure stdlib so the lesson stays portable.

code/tests/test_generator.py covers the linear path, the duplicate rejection path, the parser failure path, the temperature ramp boundaries, and the rank ordering.

Lesson fifty produces the queue. Lesson fifty-one takes the head of the queue and runs a literature search to confirm or refute it. Lesson fifty-two takes the same head and runs an actual experiment. Lesson fifty-three reads both outputs and writes a verdict. Together the four lessons make a research loop with no human in it, and a human can step in at any boundary.