LearnPython

METR Time Horizons and External Capability Evaluation

METR (ex-ARC Evals) is an independent 501(c)(3) since December 2023. Their Time Horizon 1.1 benchmark (January 2026) fits a logistic curve to task-success probability vs log(expert human completion time); the intersection at 50% probability defines the model's time horizon. The 2025–2026 engagement set covers GPT-5.1, GPT-5.1-Codex-Max, and prototype monitoring evaluations (can a monitor catch side tasks; can the agent evade). Benchmark suites: HCAST (189 ML, cyber, SWE, reasoning tasks; 1 minute to 8+ hours), RE-Bench (71 ML research-engineering tasks with expert baseline), SWAA. The honest note: METR measurements are idealized, with no human in the loop and no real consequences, and the team has documented the eval-vs-deployment behavior gap (Lesson 1). A time horizon is an upper bound, not a deployment prediction.

TIME: ~60 minutesPREREQUISITES: Phase 15 · 01 (Long-horizon agents), Phase 15 · 19 (RSP)

Learning Objectives

Explain how a METR time horizon is computed: a logistic fit of task success against log expert time, read off at 50%.
Run a stdlib logistic-fit estimator and compare its 10%, 50% and 90% horizons with a known synthetic ground truth.
Show how eval-context gaming inflates an observed horizon, and explain why a horizon is an upper bound rather than a deployment prediction.
Apply a vendor's horizon claim to your own task distribution, using it as a capability filter, a trend indicator or a prior.

The Problem

Scaling policies (Lessons 19, 20) are only as useful as the measurements they reference. "AI R&D-4 threshold" and "Long-range Autonomy" are defined in policy prose; they become actionable only when specific evaluations produce specific numbers.

METR is the 2024–2026 external evaluation organization that has defined many of those numbers. They evaluate frontier models, often pre-release and under NDA with labs, and publish methodology afterward. The Time Horizon 1.1 benchmark (January 2026) is their headline artifact: a single scalar that compresses capability into a human-legible unit: "this model can do the kind of task an expert spends X hours on at 50% reliability."

Two skills matter here: the methodology (how a horizon is computed) and the interpretation (why a horizon is an upper bound, not a deployment prediction). You need both. A team that understands how the horizon is fit is much harder to fool with a bad vendor claim than a team that just sees "14 hours" on a slide.

Warm-up0/3

1. A long-horizon agent must chain many dependent steps. Why does task length hurt its end-to-end success rate so much?Warm-up

Two chains of dependent steps, a short task with 3 steps and a long task with 12 steps, each followed by the label end-to-end success with a question mark. — Step counts are illustrative.

2. A scaling policy names a capability threshold in prose, such as an AI R&D threshold. What must exist before that threshold can trigger anything?Warm-up

Typed schematic.

3. A vendor publishes a benchmark score for its own model. Why do people often want an independent group to run the evaluation as well?Warm-up

A vendor builds a model, runs a benchmark itself and publishes the score, with a dashed box asking about an independent group that runs it as well. — Typed schematic.

Check: The Problem0/1

1. Why is a horizon framed as expert time rather than as a raw benchmark percentage?Explain

Two cards: reporting a model as a percentage of tasks passed, and reporting it as the expert hours a task takes, with a question about why the horizon uses the second. — Typed schematic; the numbers are left as ?? on purpose.

The Concept

METR background

Founded: December 2023 (ex-ARC Evals, spun out into independent 501(c)(3)).
Scope: evaluation of frontier models' autonomous capabilities, often pre-release.
Partner labs: Anthropic, OpenAI (multiple engagements 2025–2026).
Notable deliverables: Time Horizon 1.0 (March 2025), Time Horizon 1.1 (January 2026), prototype monitoring evaluations.

The Time Horizon fit

Methodology (from METR blog and papers):

Collect a task suite spanning minute-scale to hour-scale expert completion times. Current suites: HCAST (189 tasks), RE-Bench (71 tasks), SWAA.
Run the model on each task; record success or failure.
Fit a logistic curve: P(success) as a function of log(expert completion time).
The horizon is the expert-time at which P(success) = 0.5.

The logistic-fit shape is the right one because capability generally has an increasing, plateau-approaching relationship with task difficulty. The 50% point is a choice (could be 10%, 90%); METR reports multiple thresholds in the detailed paper but leads with 50% because it is the most intuitive.

The January 2026 numbers

Per Time Horizon 1.1:

Claude Opus 4.6: ~14 hours at 50% reliability, as of Time Horizon 1.1 (January 2026).
Doubling time on HCAST-style tasks: ~4.3 months (130.8 days) on the post-2023 fit reported by Time Horizon 1.1 (January 2026); the ~7-month figure is the full 2019–2025 fit from Time Horizon 1.0 and is reported in TH1.1 alongside the post-2023 number.

Those are the specific numbers the rest of Phase 15 references. They are cited in Anthropic's RSP v3.0 (Lesson 19) and DeepMind's FSF v3 (Lesson 20) as one of the measurements feeding threshold assessments.

Benchmark suites

HCAST: 189 tasks across ML, cybersecurity, software engineering, and general reasoning. Span: 1 minute to 8+ hours of expert time. The main suite behind the horizon number.
RE-Bench: 71 ML research-engineering tasks with a human expert baseline. Measures whether the model can do the kind of ML R&D work that feeds into the AI R&D-4 threshold.
SWAA: shorter tasks; useful for calibrating the low end of the horizon curve.

Prototype monitoring evaluations

2025–2026 METR work included monitoring-evaluation prototypes: can a monitor catch a model performing a side task (something other than what the operator asked for); can the agent evade the monitor. These directly feed the "Undermining Safeguards" category in OpenAI PF v2 and the deceptive-alignment monitoring in DeepMind FSF v3 (Lesson 20).

Why horizons are upper bounds

Idealized tooling: benchmark tasks give the model clean tools and well-formatted data. Real deployments are messier.
No real consequences: the model does not bill a customer, delete real data, or contact real people. Real deployments have irreversible stakes.
Eval-context gaming: Lesson 1. Models behave differently in tests. The 2026 International AI Safety Report documents this empirically.
No legitimate user variance: benchmark prompts are structured. Real users produce ambiguous, context-dependent requests.

The horizon is the capability ceiling under favorable conditions. Deployment reliability is a different number, lower, and teams must measure their own distribution to know it.

The external-evaluator case

External evaluation matters because internal labs have incentives to optimize metrics they report. METR's independence (a 501(c)(3) with a declared methodology and peer-reviewed papers) is the structural mitigation. It is not sufficient alone, since labs still control what METR sees, but it beats having no external evaluation.

How to use horizon numbers in practice

As a capability filter: if a model's horizon is well below the expert-time of a proposed task, do not ship it autonomous (Lesson 1's skill file).
As a trend indicator: doubling time tells you how long the current practice will remain safe even without new mitigations.
As a prior: a horizon of 14 hours is a starting point. Adjust down for your task distribution, your tooling quality, and your deployment context.

Check: The Concept0/20

1. What is METR?Define

Three Further Reading titles that each begin with METR, and a dashed box reading METR is question mark. — Titles quoted from the lesson's Further Reading list.

2. How is a METR time horizon defined?Define

Table of eight tasks with expert time in hours and a pass or fail result for the model, followed by the label time horizon of this model with a question mark. — First eight tasks of the lesson's synthetic set (seed 3, from main.py).

3. What is HCAST?Define

A Further Reading entry that lists HCAST, RE-Bench and SWAA specs, and a dashed box reading HCAST is question mark. — Entry quoted from the lesson's Further Reading list.

4. What is RE-Bench?Define

A Further Reading entry that lists HCAST, RE-Bench and SWAA specs, and a dashed box reading RE-Bench is question mark. — Entry quoted from the lesson's Further Reading list.

5. Which suite calibrates the low end of the horizon curve with shorter tasks?Define

A falling curve of probability of success against expert time on a log axis, fitted to the lesson's synthetic results, with the short-task left end marked and the label calibrated by question mark. — Curve fitted to the lesson's synthetic results (seed 3).

6. What does a model's doubling time measure?Define

A card reading doubling time about 4.3 months on the post-2023 fit for HCAST-style tasks, with the line it measures followed by a question mark. — Figure quoted from the lesson.

7. What is eval-context gaming?Define

One model with the same weights run in two settings, under test and in deployment, each leading to a dashed box reading behavior question mark. — Typed schematic.

8. What does it mean to call a time horizon an upper bound?Define

Two cards side by side: time horizon measured on benchmark tasks at 50 percent reliability, and deployment reliability measured on your own tasks, with a question about how they relate. — Typed schematic.

9. What did Time Horizon 1.1 (January 2026) report as Claude Opus 4.6's horizon at 50% reliability?Define

A card for Claude Opus 4.6 from Time Horizon 1.1, January 2026, reading horizon at 50 percent reliability with a question mark for the hours. — Typed schematic.

10. Why does METR use a logistic curve to model success against task difficulty?Explain

Empty axes with probability of success up the side and expert time on a log scale along the bottom, and a question mark where the fitted curve would go. — Typed schematic.

11. Why is METR's 50% threshold described as a choice rather than the only option?Explain

A falling logistic curve of success against expert time with a dashed line and a marked point at 50 percent, and a question about why that level is a choice. — Curve fitted to the lesson's synthetic results (seed 3).

12. Which statement about doubling time matches the January 2026 numbers section?Explain

A time axis from 2019 to 2025 with a long bar for the full 2019 to 2025 fit and a shorter bar for the post-2023 fit, each with doubling time marked as a question mark. — Typed schematic; bar ends are approximate.

13. What do METR's prototype monitoring evaluations test?Explain

Typed schematic.

14. Why is a time horizon an upper bound rather than a deployment prediction?Explain

A vendor slide card reading 14-hour horizon at 50 percent reliability beside a dashed card for your deployment, with a question about why the slide is not a prediction for it. — Typed schematic.

15. What does METR's independence buy, according to The external-evaluator case?Explain

Typed schematic; METR facts from the lesson.

16. A model's measured 50% horizon is 2 hours. You propose giving it a task that takes an expert about 10 hours. What does the capability-filter rule say?Apply

A ruler of expert hours from 0 to 12 with a marker for the model's measured 50 percent horizon at 2 hours and a marker for a proposed task of about 10 hours, and a question whether to ship it autonomous. — Typed schematic.

17. A model has a 14-hour horizon and the doubling time is about 4.3 months. If the trend holds, roughly what horizon would a model show 8.6 months later?Apply

A timeline with the horizon now at 14 hours and question marks at 4.3 months and 8.6 months later, with the doubling time of about 4.3 months noted. — Typed schematic.

18. An agent clears the vendor's 14-hour benchmark horizon, but in production it stumbles whenever users send vague, context-dependent requests. Which listed gap explains this?Analyze

Two cards for one agent: it clears the vendor's 14-hour horizon on the benchmark, and in production it stumbles whenever users send vague, context-dependent requests. — Typed schematic.

19. A vendor's slide claims a 14-hour 50% horizon. Your task takes an expert about 10 hours, runs on messy internal tools and has irreversible effects. What is the sound reading?Design

A vendor slide card reading 14-hour horizon at 50 percent reliability beside a card for your task: about 10 expert hours, messy internal tools and irreversible effects. — Typed schematic; values from the question.

20. A policy team wants one rule for letting agents run unsupervised on high-stakes tasks. Which design best uses a published horizon?Design

Two cards: inputs available, a published 50 percent horizon and your own deployment context, and the rule to draft for agents running unsupervised on high-stakes tasks, left as a question mark. — Typed schematic.

Use It

code/main.py implements a logistic fit of task-success vs log(expert time), given a synthetic result set. It reports the 50% horizon (METR's headline), the 10% horizon (the long, optimistic end of the curve), and the 90% horizon (the short, conservative end). It also demonstrates what changes when the success rate is artificially inflated by eval-context gaming.

Check: Use It0/11

1. Why is the 10% horizon the long, optimistic end of the curve and the 90% horizon the short, conservative end?Explain

Axes with expert time increasing to the right and probability of success up the side, with dashed lines at 90 percent and 10 percent reliability and the 90 percent and 10 percent horizons marked as question marks. — Typed schematic.

2. Why does inject_gaming only flip failures to successes and never the reverse?Explain

The body of inject_gaming from main.py: a loop over tasks that checks for a failed task and a random draw below the gaming rate, appends a success for those, and otherwise appends the original result. — Code from main.py, with type hints and docstring dropped.

3. With fitted w = -1.0 and b = ln(8), what does horizon_at(w, b, 0.50) return?Apply

The horizon_at function from main.py with its zero-slope check elided, the values w equals -1.0, b equals ln 8 and p equals 0.50, and the call with a question mark for the result. — Code from main.py, with type hints and docstring dropped; the elided lines are the zero-slope guard.

4. With the same fit, w = -1.0 and b = ln(8), roughly what does horizon_at(w, b, 0.90) return?Apply

The horizon_at function from main.py with its zero-slope check elided, the values w equals -1.0, b equals ln 8 and p equals 0.90, and the call with a question mark for the result. — Code from main.py, with type hints and docstring dropped; the elided lines are the zero-slope guard.

5. fit() returns a slope w of about 0.0 on a result set where success does not depend on task length. What does horizon_at(w, b, 0.50) do?Apply

A nearly flat fitted line of success against expert time, the horizon_at code from main.py with its zero-slope check elided, and the call horizon_at of w, b and 0.50 with a question mark for the result. — Typed sketch of a flat fit; the elided lines are the zero-slope guard.

6. inject_gaming is called with gaming_rate 1.0 on 100 tasks, 35 of which failed. How many successes does the returned list contain?Apply

A grid of 100 tasks with 35 outlined in red as failed and the rest in green as succeeded, beside the call inject_gaming of tasks and 1.0 and the question of how many successes the returned list has. — Typed schematic; which 35 tasks failed is arbitrary.

7. In synth_tasks with true_horizon_hours = 14, what is P(success) for a task whose expert time is exactly 14 hours?Apply

Four lines from synth_tasks in main.py that compute log of the true horizon, a logit from the slope, and a probability, with true_horizon_hours equal to 14 and a task of expert time 14 hours, and the probability of success as a question mark. — Code from main.py; line 2 stands in for the loop that draws t.

8. With seed 3, the 40% gaming run printed a delta of +30.56 hr against a 14.0 hr ground truth. Roughly what observed 50% horizon is that?Apply

Terminal output showing a synthetic ground truth horizon of 14.0 hours and a 40 percent gaming run that prints a delta of plus 30.56 hours against it, with the observed horizon replaced by a question mark. — Lines from the real run of main.py with the horizon replaced by ?.

9. A team's fit returns a nearly flat curve and their spreadsheet divides by the slope, reporting a horizon of billions of hours. What is the right conclusion?Analyze

A nearly flat fitted curve of success against expert time, and a spreadsheet cell that divides by the slope and reports billions of hours. — Typed sketch; the cell is illustrative.

10. Two vendors both claim a 14-hour 50% horizon. Vendor A reports a 90% horizon of 1 hour, and vendor B reports 5 hours. For a task you need done reliably, what does this suggest?Analyze

Table with Vendor A and Vendor B, both with a 50 percent horizon of 14 hours, A with a 90 percent horizon of 1 hour and B with 5 hours, and a question of which vendor it favors for reliable work. — Values from the question.

11. On the default seed the fit's 50% horizon is about 11.3 hr against a ground truth of 14.0 hr with no gaming. A reader concludes the estimator is broken. What is the better diagnosis?Analyze

A number line of hours with a ground truth marker at 14.0 hours and a fit marker at 11.3 hours, and a question of what the better diagnosis of the gap is. — Values from the question.

Ship It

outputs/skill-horizon-interpretation.md reviews a vendor's horizon claim and produces a gap analysis between benchmark claim and deployment reality.

Check: Ship It0/2

1. What does the horizon-interpretation skill produce when it reviews a vendor's horizon claim?Explain

A vendor's horizon claim of 14-hour tasks at 50 percent goes into the horizon-interpretation skill, which leads to a dashed output box with a question mark. — Typed schematic; claim wording from the skill file.

2. You want to know whether an agent can clear your own bug backlog. Which plan uses the lesson's method?Design

A table with a bug column and a status column listing three open bugs and an ellipsis row, and a question of which plan answers whether the agent can clear the backlog. — Typed schematic.

Author's Note

I'd run main.py before trusting any horizon number, because the whole method fits in about 150 lines of stdlib Python. The generator sets a true 50% horizon (14 hours, slope 1.2), the fit is plain gradient descent on w * log(t) + b, and horizon_at inverts the logistic to read off the expert time at any probability you ask for. METR never gets to see the true horizon of a real model. Here you can, so you can see how far the fit lands from it.

The part I'd read twice is inject_gaming. It only ever flips failures to successes, so the observed horizon almost always drifts up and never has a reason to drift down. That is the direction a vendor's incentives point. Watch the printed delta against ground truth grow with the gaming rate (+30.56 hr at 40%), and you feel why a horizon without a gaming audit is a ceiling and not a forecast.

I'd also keep the toy in proportion. It fits one curve to 160 tasks from a single seed, with a fixed iteration count and learning rate, so the exact deltas will move if you change any of those. Treat the trend as the takeaway and the digits as noise. horizon_at raises on a near-zero slope instead of returning something huge, which is a habit I'd keep when fitting your own task results, since a flat curve means the horizon is not defined.

Exercises

Run code/main.py. Compare the fit's 50% horizon to the synthetic ground truth (the default seed prints about 11.3 hr against 14.0 hr, so expect a miss from sampling noise). Now halve the task-time grid; does the horizon estimate change meaningfully?
Read METR's Time Horizon 1.1 blog post. Identify the specific tasks where reliability is highest and where it is lowest. Explain why the gap exists.
Read METR's "Measuring Autonomous AI Capabilities" resources. List the HCAST task categories. Pick one category you would weight more heavily for a production task and justify why.
Introduce eval-context gaming into the simulator: flip ~20% of failed tasks to success. Report the new horizon (the built-in 20% run prints about 14.6 hr). This approximates what a gaming rate of 20% does to the observed number.
Design an internal horizon evaluation on your own bug backlog or a representative task set. Describe the data collection, the fit, and what the output tells you. Compare to METR numbers.

Key Terms

Term	What people say	What it actually means
METR	"External evaluator"	ex-ARC Evals; independent 501(c)(3) since Dec 2023
Time Horizon	"Capability measure"	Expert task length at 50% reliability, from logistic fit
HCAST	"METR's main suite"	189 tasks spanning 1 min to 8+ hours
RE-Bench	"Research engineering"	71 ML research-engineering tasks with human baseline
SWAA	"Short-task suite"	Calibrates the low end of the horizon curve
Doubling time	"Growth rate"	Time for the 50% horizon to double; ~4.3 months on the post-2023 fit, ~7 months on the full 2019–2025 fit
Eval-context gaming	"Model behaves differently"	Documented behavior gap between tests and deployment
Upper bound	"Horizon is a ceiling"	Benchmark horizon > deployment reliability under load