LearnPython

The Autonomous Coding Agent Landscape (2026)

SWE-bench Verified went from 4% to 80.9% in under three years. Same Claude Sonnet 4.5 scored 43.2% on SWE-agent v1 and 59.8% on Cline autonomous, so the scaffolding around the model now matters as much as the model itself. OpenHands (formerly OpenDevin) is the most active MIT-licensed platform and its CodeAct loop executes Python actions directly in a sandbox instead of JSON tool calls. The headline numbers hide a methodological issue: 161 of 500 SWE-bench Verified tasks require only a 1–2 line change, and SWE-bench Pro (10+ line tasks) sits at 23–59% for the same frontier models.

TIME: ~45 minutesPREREQUISITES: Phase 14 · 07 (Tool use), Phase 15 · 01 (Long-horizon agents)

Learning Objectives

Explain why scaffolding (retrieval, planner, sandbox, verifier loop) moves SWE-bench scores as much as the base model does, using the 43.2% vs 59.8% Sonnet 4.5 result.
Compare JSON tool calls with CodeAct on turns, compositionality, auditability, and blast radius.
Describe the SWE-bench Verified easy tail and why SWE-bench Pro (10+ line changes) is a better proxy for production work.
Run the toy scaffold comparison and read turns and blast radius for each execution model.
Choose a coding agent by testing candidate scaffolds on a Pro-like slice of your own bug backlog.

The Problem

"Which coding agent is best" is the wrong question. The right question is: on a task distribution that matches my work, with the scaffolding I will run in production, what end-to-end reliability do I get?

Between 2022 and 2026 the field learned that scaffolding is load-bearing. That means the retrieval layer, the planner, the sandbox, the edit-verify loop, and the feedback format. Claude Sonnet 4.5 on SWE-agent v1 scored 43.2% on SWE-bench Verified; the same model inside Cline's autonomous scaffold scored 59.8%. That is 16.6 absolute points of difference on the same weights. The base model is a component; the loop is the product.

The companion problem is that benchmark saturation hides regressions. SWE-bench Verified is close to saturated, and the easy-task tail (161 of 500 tasks requiring ≤2 lines) pulls top scores up. Real-world quality is better measured on distributions like SWE-bench Pro (10+ line changes), where the same leaders still sit at 23–59%.

Warm-up0/3

1. In a standard JSON tool-calling loop, who actually executes the action the model asks for?Warm-up

Typed schematic; the JSON text is the format the lesson's toy scaffold prints.

2. An agent succeeds on each step independently with probability 0.95. Roughly what is its end-to-end success rate over a 20-step trajectory?Warm-up

Twenty small boxes in two rows of ten, numbered 1 to 20, each labelled 0.95 for its per-step success probability. Below them a dashed box asks the end-to-end success of the whole run with a question mark. — Each box is one step with success probability 0.95.

3. A team proposes a bug fix for a codebase that has a unit test suite. What is the most direct automatic check that the fix works?Warm-up

Two panels. The left lists a toy repo with app.py, util.py, cli.py and a tests folder. The right shows a one-line diff in app.py changing a minus to a plus. A dashed box asks which automatic check shows the fix works. — Toy repo and diff from the lesson's example code.

Check: The Problem0/3

1. How many of SWE-bench Verified's 500 tasks need only a 1-2 line change?Define

A grid of 500 identical squares, one per SWE-bench Verified task, with a caption asking how many need only a 1-2 line change. — No task is singled out in the grid.

2. Sonnet 4.5 scores 43.2% in SWE-agent v1 and 59.8% in Cline's autonomous scaffold. What does the 16.6-point gap show?Explain

Two horizontal bars for Claude Sonnet 4.5 on SWE-bench Verified: SWE-agent v1 at 43.2 percent and Cline's autonomous scaffold at 59.8 percent, with a bracket marking a 16.6 point gap. — Scores as given in the question.

3. SWE-bench Verified has 500 tasks, and 161 of them need only 1-2 lines. How many tasks need a bigger change?Apply

A grid of 500 squares with the first 161 highlighted as 1-2 line tasks and the rest dim, with a line asking for the count of tasks needing a bigger change. — Counts from the question.

The Concept

SWE-bench, one paragraph

SWE-bench (Jimenez et al.) takes real GitHub issues with ground-truth patches and asks an agent to produce a patch that makes the test suite pass. SWE-bench Verified (OpenAI, 2024) is a human-curated 500-task subset with the ambiguous and broken tasks removed. SWE-bench Pro is the harder successor, with tasks requiring 10+ lines of change, where current frontier agents sit at 23–59%.

What the 2022 → 2026 curve actually shows

2022: research models at ~4% on raw SWE-bench.
2024: GPT-4 + Devin-style scaffolding at ~14%; SWE-agent at ~12%.
2025: Claude 3.5/3.7 Sonnet inside Aider and SWE-agent push into the 40–55% range.
2026: Claude Sonnet 4.5 and frontier competitors at 70–80%+ on SWE-bench Verified. Epoch AI's leaderboard tracks this live.

The slope came from three compounding sources: better base models, better scaffolding (CodeAct, reflection, verifier loops), and better benchmarks (Verified removing noise).

CodeAct vs JSON tool calls

OpenHands (All-Hands-AI, arXiv:2407.16741, formerly OpenDevin) took a specific architectural bet: instead of the model emitting JSON tool calls that a host decodes and executes, the model emits Python code and a Jupyter-style kernel runs it in a sandbox. The agent can loop over files, chain tools, and catch its own exceptions inside one action.

The trade-off:

JSON tool calls: every action is one turn; easy to audit; limited compositionality; safe by default because each call goes through an explicit validator.
CodeAct: one action can be a whole program; compositional; requires a hardened sandbox (OpenHands uses Docker isolation); failure modes include anything the sandbox runtime allows.

Both architectures are in production. CodeAct is dominant in open platforms (OpenHands, smolagents). JSON tool calls remain dominant in managed services (Anthropic Managed Agents, OpenAI Assistants) where the provider controls the executor.

Scaffolds in the 2026 landscape

Scaffold	License	Execution model	Notable property
OpenHands (OpenDevin)	MIT	CodeAct in Docker	Most active open platform; event-stream replayable
SWE-agent	MIT	Agent-Computer Interface (ACI)	First end-to-end SWE-bench scaffold
Aider	Apache-2	edit-via-diff in local repo	Minimal scaffold, strong regression stability
Cline	Apache-2	VS Code agent with tool policy	Highest-scoring open scaffold on Sonnet 4.5
Devin (Cognition)	Proprietary	Managed VM + planner	First "AI software engineer" product category
Claude Code	Proprietary	Permission modes + routines	Lesson 10 covers the agent loop in detail

Why scaffolding dominates

A coding run is a long-horizon trajectory (Lesson 1). Reliability compounds across steps. Three places where scaffolding buys points:

Retrieval: finding the right files to read is the silent bottleneck. SWE-agent's ACI, OpenHands' file-index, and Aider's repo-map all attack this.
Verifier loop: running tests, reading stack traces, and re-attempting is worth 10+ points on SWE-bench.
Failure containment: a sandbox that rolls back on error prevents compounding damage. The same model with and without a verifier loop looks like two different products.

Benchmark saturation and the real distribution

The OpenHands authors and Epoch AI both flag that SWE-bench Verified has an easy tail: 161 of 500 tasks need only 1–2 lines of change. High scores are driven partly by this tail. SWE-bench Pro restricts to 10+ line changes and returns scores in the 23–59% range even for frontier systems. Your production distribution is almost certainly closer to Pro than to Verified.

Implication for choosing an agent: run a Pro-like subset of your own bug backlog. The score that matters is the score on tasks representative of what you ship.

Check: The Concept0/25

1. What is SWE-bench?Define

A toy example of one benchmark instance: a GitHub issue saying add(2, 3) returns -1, a repo snapshot with three files, and an agent that writes a patch. Below, a dashed box asks how the agent is scored, marked with a question mark. — Illustrative instance built from the lesson's toy repo, not an actual SWE-bench task.

2. What is SWE-bench Verified?Define

A fact card titled SWE-bench Verified: introduced by OpenAI in 2024, 500 tasks, each a GitHub issue with ground-truth patch and tests, plus a line asking the meaning of Verified with a question mark. Below, a grid of 500 identical squares. — Fifty by ten squares, one per task; no per-task detail is drawn.

3. What is SWE-bench Pro?Define

Three boxes in a row naming SWE-bench by Jimenez et al., SWE-bench Verified by OpenAI in 2024, and SWE-bench Pro highlighted with a question mark. A line asks for the task set of SWE-bench Pro. — Only the names and sources given in the lesson are drawn.

4. What is CodeAct?Define

A terminal panel showing the action text the lesson's CodeAct stub produces on turn 1: three fs.write calls for app.py, util.py and cli.py. To the right a label names the style of agent as CodeAct. — Captured from the toy scaffold; the stub's snippet is a string, not run.

5. What is a JSON tool call?Define

A terminal panel with one line of JSON text, tool edit and path app.py, as printed by the lesson's JSON scaffold on turn 1. To the right a line asks for the name of this action format. — Captured from the toy scaffold.

6. What does 'scaffold' mean for a coding agent?Define

Typed schematic; the contents of the dashed region are left open.

7. What does ACI stand for, and what is it?Define

A two-column table listing three scaffolds with their execution models: SWE-agent as ACI (highlighted), Aider as edit-via-diff in a local repo, and Claude Code as permission modes plus routines. — Three rows of the lesson's scaffold table; the SWE-agent cell reads just ACI.

8. What is a verifier loop?Define

A terminal excerpt of a toy run: turn 1 is a JSON edit of app.py, followed by a line for what happens between turns and a turn 2 line, both shown as question marks. — Turn 1 comes from the lesson's toy code; everything after it is hidden.

9. Where does SWE-bench Pro (10+ line changes) place the frontier models that score 70-80%+ on Verified?Define

A percent axis from 0 to 100. A solid blue band marks frontier models on Verified at 70 to 80 percent or more. A dashed empty band spans the axis for Pro with 10+ line changes, marked with a question mark. — Only the Verified range from the question is drawn.

10. Roughly where did GPT-4 + Devin-style scaffolding and SWE-agent land in 2024, and Claude 3.5/3.7 Sonnet in 2025?Define

A percent axis from 0 to 100 with two dashed empty columns marked with question marks, one for 2024 (GPT-4 with Devin-style scaffolding, and SWE-agent) and one for 2025 (Claude 3.5 and 3.7 Sonnet). — No scores are drawn.

11. Which open scaffold does the scaffold table call the highest-scoring on Sonnet 4.5?Define

A ranking ladder with four empty dashed slots numbered 1 to 4, each marked with a question mark, titled open scaffolds ranked by SWE-bench Verified score on Claude Sonnet 4.5. — No scaffold names or scores are placed in the slots.

12. Why is a 1-2 line easy tail a problem for reading SWE-bench Verified scores?Explain

Two panels. The left shows a one-line toy diff, return a minus b changed to return a plus b, labelled a 1-2 line task. The right shows a Verified score as tasks solved over all Verified tasks. A dashed box asks about reading a score with this easy tail inside it. — Typed schematic; the diff is the lesson's toy fix.

13. Why is retrieval called the silent bottleneck in a coding run?Explain

A row of five connected boxes for a coding run: Find files (highlighted), Read, Edit, Run tests and Submit. — Typed schematic.

14. Why does the same model look like two different products with and without a verifier loop?Explain

Two side-by-side panels, one labelled with verifier loop and one without. Each has the same-weights Model box and a first attempt of editing app.py, with what happens next shown as a question mark. — Typed schematic; the model box is identical in both panels.

15. Which three places does scaffolding buy the most reliability on a long coding trajectory?Explain

A strip of twenty step squares from issue to patch, titled long coding trajectory. Below are three dashed boxes labelled place 1, place 2 and place 3, each with a question mark. — Typed schematic; the three places are left blank.

16. Why does CodeAct need a hardened sandbox when JSON tool calls usually do not?Explain

Two lanes. JSON tool call: Model, an edit payload, then Executor. CodeAct: Model, a fs.write call for cli.py, Executor, then a dashed box labelled Sandbox with a question mark. — Typed schematic.

17. How do JSON tool calls and CodeAct differ on auditability?Explain

Two terminal panels showing the first logged action of each toy scaffold. The JSON scaffold log shows one line, an edit of app.py. The CodeAct log shows turn 1 with three fs.write calls for app.py, util.py and cli.py. — Captured from the lesson's toy scaffolds, turn 1 only; the stub's snippet is a string.

18. What is the key trade-off of CodeAct compared with JSON tool calls?Explain

A comparison table with columns for JSON tool calls and CodeAct. The first row shows an action as the toy scaffolds write it: a JSON edit payload versus fs.write calls. The rows for what each gains and what each costs are question marks. — Action text from the lesson's toy scaffolds; the gain and cost cells are left open.

19. Which execution model dominates in managed services versus open platforms, and why?Explain

Two panels. Managed services lists Anthropic Managed Agents and OpenAI Assistants; open platforms lists OpenHands and smolagents. Each has a dashed box asking for its execution model with a question mark. — Example systems are those named in the lesson.

20. An agent scores 78% on SWE-bench Verified but fixes only about a third of your team's real bugs, which usually span several files. Which explanation fits best?Analyze

Two horizontal bars for one agent: 78 percent on SWE-bench Verified and about one third on the team's own multi-file bugs. — Figures from the question.

21. A team blames a weak model because its agent keeps submitting patches that fail the tests. Logs show the agent never runs the test suite after editing. Which fix targets the real cause?Analyze

A typed agent log: edit util.py, edit cli.py, submit patch, then two warning lines saying the agent ran the test suite 0 times and the benchmark tests on the submitted patch fail. — Typed, not captured; file names reuse the lesson's toy repo.

22. A CodeAct agent in a pilot runs a generated script that deletes files outside the project directory. Which missing ingredient is the main cause?Analyze

Typed schematic of the stem's incident.

23. Logs show an agent often never opens the file that contains the bug and edits an unrelated file instead. Which scaffold component is the weak link?Analyze

A typed agent log on the toy repo: the failing test is in cli.py, but the agent opened and edited util.py, never opened cli.py, and tests end at 2 of 3 passed. — Typed, not captured; test count computed from the lesson's stub runner.

24. A vendor shows an agent at 78% on SWE-bench Verified. Your team's typical bug touches 2-3 files and 20+ lines. What should you expect and do?Design

Two boxes: a vendor claim of 78 percent on SWE-bench Verified, and the team's typical bug touching 2 to 3 files and 20 or more lines. A dashed box below asks for the expected score and what to do. — Figures from the question.

25. You run a managed coding service where your company controls the executor, and every action must be independently validated and logged. Which execution model fits, and why?Design

A pipeline: Model, an action box with execution model question mark, an Executor owned by the company, and a repo. A line states the requirement that every action is validated and logged. — Typed schematic of the stem's setup.

Use It

code/main.py compares two toy agent scaffolds on a fixed mini-task distribution:

A JSON tool-call scaffold that takes one action per turn.
A CodeAct scaffold that can emit a small Python snippet per action.

Both use a stub "model" (deterministic rules) so the comparison isolates the scaffold from model quality. The toy repo has three one-line bugs. Both scaffolds fix all three, so the difference is in the cost: the JSON scaffold uses 4 turns (three edits, then a done action) and touches 1 file per action, while the CodeAct scaffold uses 2 turns (one snippet that edits all three files, then done) and has a blast radius of 3 files.

Check: Use It0/4

1. In the toy repo, app.py is already fixed, so only util.py and cli.py fail. How many turns does the JSON scaffold take?Apply

A table of the three toy tests: the app.py test passes, while the util.py and cli.py tests fail. Below, a line asks how many turns the JSON scaffold takes with a question mark. — Test states computed by the lesson's stub run_tests.

2. In the toy repo only cli.py is broken. What do turns and blast radius come out as for the CodeAct scaffold?Apply

A table of the three toy tests: app.py and util.py pass, cli.py fails. Below, a line asks for the CodeAct scaffold's turns and blast radius, both as question marks. — Test states computed by the lesson's stub run_tests.

3. What does the toy run print for the CodeAct scaffold on the three-bug repo?Apply

A table of the three toy tests, all failing on the unfixed repo. Below, a terminal shows the report line format for the CodeAct stub with question marks in place of passed, turns and blast radius. — Test states computed by the lesson's stub run_tests; the line format is the lesson's report().

4. In the Use It toy comparison, what does CodeAct give up in exchange for finishing in 2 turns instead of 4?Apply

Two horizontal bars of turns on the toy repo: JSON scaffold 4 turns, CodeAct scaffold 2 turns. Below, a line asks what CodeAct gives up for the saved turns. — Turn counts from the lesson's toy run.

Ship It

outputs/skill-scaffold-audit.md helps you audit a proposed coding-agent scaffold before adoption: retrieval quality, verifier presence, sandbox isolation, and benchmark-to-distribution fit.

Author's Note

I'd run main.py first and read the two result lines before anything else. Both scaffolds pass 3/3, so the only things that differ are turns and blast radius. The stub model is a FIXES table plus string checks, which means no model quality leaks into the comparison. Whatever gap you see comes from how each scaffold packages actions.

The number I'd stare at is the CodeAct blast radius. It is worst_touched, the largest count of files a single action actually edited, and not len(repo). I chose to measure it that way because a static upper bound would grow every time someone adds an untested file, and it would tell you nothing about what the agent did.

The toy is also thin in a way worth knowing about. The CodeAct "snippet" is a string of fs.write(...) calls that never executes, and nothing is sandboxed. Both scaffolds do re-run the tests every turn, so they each have a verifier loop, but neither has a retrieval problem because the repo is three files. For a real choice I'd test retrieval and the verifier on a slice of my own backlog, the way the Concept section suggests.

Exercises

Run code/main.py. How many turns does each scaffold take on the same task set? What is the per-action blast radius of each?
Read the OpenHands paper (arXiv:2407.16741). The paper argues CodeAct beats JSON tool calls on complex tasks. Identify one failure mode the paper acknowledges and write one sentence on when that mode would dominate in production.
Pick one task from your bug backlog that would require 10+ lines of change across two files. Estimate the end-to-end success probability for a frontier model under (a) JSON tool calls and (b) CodeAct. Justify the gap.
SWE-bench Verified has 161 single-file, 1–2 line tasks. Construct a score that excludes them. How does the leaderboard shuffle?
Read "Introducing SWE-bench Verified" (OpenAI). Explain the specific methodology used to remove ambiguous tasks, and name one category the curation would miss.

Key Terms

Term	What people say	What it actually means
SWE-bench	"Coding benchmark"	Real GitHub issues with ground-truth patches and test suites
SWE-bench Verified	"Cleaned subset"	500 human-curated tasks, easier-tail present
SWE-bench Pro	"Harder subset"	10+ line changes; frontier sits at 23–59%
CodeAct	"Code-as-action"	Agent emits Python; Jupyter-style kernel executes in sandbox
JSON tool call	"Function calling"	Each action is a structured JSON payload validated before execution
Scaffold	"Agent framework"	Retrieval + planner + executor + verifier loop around the base model
ACI (Agent-Computer Interface)	"SWE-agent's format"	Command set designed for LLM ergonomics, not human shells
Verifier loop	"Test-and-retry"	Run tests, read output, revise patch; biggest non-model reliability gain