LearnPython

Benchmarks: SWE-bench, GAIA, AgentBench

Three benchmarks anchor agent evaluation in 2026. SWE-bench tests code patching, GAIA tests generalist tool use, and AgentBench tests multi-environment reasoning. Know what each one is made of, how it got contaminated, and what it can't tell you.

TIME: ~60 minutesPREREQUISITES: Phase 14 · 06 (Tool Use)

Learning Objectives

Name SWE-bench's test harness (FAIL_TO_PASS) and explain why it gates on unit tests.
Explain why SWE-bench Verified (OpenAI, 500 tasks) exists and what it removes.
Describe GAIA's design: simple for humans, hard for AI; three difficulty levels.
Name AgentBench's eight environments and its primary blocker for open-source LLMs.
Summarize the SWE-bench+ contamination finding and its implications.

The Problem

A leaderboard tells you which model won one benchmark. It doesn't tell you the three things you need to know before trusting the number:

Whether the benchmark is contaminated, meaning solutions leaked into training data or the answer leaked into the task itself.
Whether it measures what you care about: code, browsing, or generalist tool use.
Whether the evaluator is reliable (AST matching, state checks, human review).

Learn the three anchoring benchmarks and how each one fails before you quote a number.

Warm-up0/4

1. An agent says it fixed a bug in your repository. Which evidence would you trust most?Warm-up

2. A model scores very high on a public benchmark whose tasks were posted online years before the model's training cutoff. What is the most reasonable worry?Warm-up

3. A report gives only one average success rate for an agent. What does that single number hide?Warm-up

4. A question needs a web search, then reading a PDF, then some arithmetic. What does an agent need to answer it?Warm-up

Check: The Problem0/1

1. What does contamination mean for a benchmark?Define

The Concept

SWE-bench (Jimenez et al., ICLR 2024 oral)

SWE-bench is a benchmark of 2,294 real GitHub issues from 12 popular Python repos, where the agent's job is to write a patch that resolves the issue.

Input: the codebase at the pre-fix commit plus the natural-language issue description.
Output: a patch.
Evaluator: apply the patch and run the repo's test suite. The patch has to flip the FAIL_TO_PASS tests (failing before, passing now) without breaking any PASS_TO_PASS tests.

SWE-agent (Yang et al., 2024) hit 12.5% at release by emphasizing agent-computer interfaces (file editor commands, search syntax the model understands).

SWE-bench Verified

SWE-bench Verified is a human-curated 500-task subset of SWE-bench, released by OpenAI in Aug 2024. It drops ambiguous issues, unreliable tests, and tasks where the intended fix was unclear. This is the number to quote when the question is whether your agent ships real patches.

Contamination

Over 94% of SWE-bench issues predate most model cutoffs, so the models may have seen them. The sharper finding comes from SWE-bench+: 32.67% of successful patches leaked the solution in the issue text (the model read the fix in the description), and another 31.08% were suspicious because the tests were too weak to catch a wrong patch. Verified is cleaner but not contamination-free.

In practice, a model that scores 50% on SWE-bench may score 35% on SWE-bench+. If you claim SWE-bench performance, report the Verified and SWE-bench+ numbers together, not the raw SWE-bench score alone.

GAIA (Mialon et al., Nov 2023)

GAIA is a generalist benchmark of 466 questions in total. The private leaderboard at huggingface.co/gaia-benchmark scores the 300 that are held out. The design philosophy is "conceptually simple for humans (92%) but hard for AI (GPT-4 with plugins: 15%)."

Tests reasoning, multi-modality, web browsing, and tool use together.
Three difficulty levels. Level 3 needs long tool chains across modalities.

Run GAIA when you want to measure generalist capability, and don't confuse it with the code-specific benchmarks above.

AgentBench (Liu et al., ICLR 2024)

AgentBench is a multi-turn suite of 8 environments across code (Bash, DB, KG), games (Alfworld, LTP), web (WebShop, Mind2Web), and open-ended generation, at roughly 4k-13k turns per split. This lesson names seven environments individually, so check the paper for the full list of eight.

Its primary finding: long-term reasoning, decision-making, and instruction following are what keep open-source LLMs from catching commercial ones.

What these do not measure

Real-world operational cost (tokens, wall-clock).
Safety behavior under adversarial conditions.
Performance on your domain. For that, write your own evals (Lesson 30).
Tail failures. Benchmarks average, and production operators care about the worst 1%.

Where benchmarking goes wrong

Single-number fixation. "SWE-bench 50%" tells you less than the P50/P75/P95 distribution of cost and step count.
Contaminated claims. Reporting SWE-bench without mentioning Verified or SWE-bench+ is misleading.
Benchmark as development target. Optimize for the benchmark and you drift away from what's useful in production.

Check: The Concept0/20

1. What is SWE-bench?Define

2. What is SWE-bench Verified?Define

3. What is a FAIL_TO_PASS test in SWE-bench?Define

4. What is a PASS_TO_PASS test in SWE-bench?Define

5. What did SWE-bench+ find about successful SWE-bench patches?Define

6. What is GAIA's design philosophy?Define

7. What is AgentBench?Define

8. Why does SWE-bench check PASS_TO_PASS tests as well as FAIL_TO_PASS tests?Explain

9. Why was SWE-bench Verified created?Explain

10. How does GAIA structure its difficulty?Explain

11. SWE-agent reached 12.5% at release by emphasizing agent-computer interfaces. What does that mean?Explain

12. SWE-bench+ reports two kinds of problem in successful patches. How do they differ?Explain

13. Which of these do SWE-bench, GAIA, and AgentBench NOT tell you about your product?Explain

14. According to AgentBench, what mainly keeps open-source LLMs from catching commercial ones?Explain

15. A patch makes the target failing test pass but breaks two other tests in the repo. How does SWE-bench score it?Apply

16. Your agent scores 50% on SWE-bench. Which way of reporting it is most defensible?Apply

17. A wrong patch passes every test in a task, yet a reviewer sees it does not fix the issue. Which SWE-bench+ finding does this match?Analyze

18. Your agent hits 90% on a public benchmark, yet users complain about rare disastrous runs. What explains the gap?Analyze

19. Agent A is announced as 'SWE-bench 50%'. Agent B scores 48% with a far tighter spread of cost and step count. Which reading is better supported?Analyze

20. You are building an assistant that must browse the web, read files, and combine the results. Which benchmark fits to measure it?Design

Build It

code/main.py is a toy SWE-bench-style harness:

Synthetic bug-fix tasks (3 tasks).
A scripted "agent" that proposes patches.
A test runner that checks FAIL_TO_PASS (the bug is fixed) and PASS_TO_PASS (nothing else broke).
A GAIA-style difficulty classifier based on question decomposition depth.

Run it:

python3 code/main.py

The output gives resolution rate per task and per difficulty, which makes the evaluator rules concrete.

Check: Build It0/6

1. In the toy harness, t003 patches the state to counter 2 and flag False. What does run_task report for it?Apply

2. For t002 the state starts at counter 1 and multiplier 0, and the patch sets multiplier to 2. What does run_task return?Apply

3. What does the demo print for the resolution rate across t001, t002, and t003?Apply

4. gaia_level scores 'What is the capital of France?'. What level does it return, and why?Apply

5. gaia_level is given the Visit, find, PDF, then count, and return question. Which score and level result?Apply

6. A harness marks t003 as resolved even though it breaks the flag test. Which change to run_task would cause that?Analyze

Use It

SWE-bench Verified for code agents. Report Verified scores alongside SWE-bench+ results, since Verified is cleaner but not contamination-free.
GAIA for generalist agents. Use the private leaderboard split.
AgentBench for comparing across environments.
Custom evals (Lesson 30) for the shape of your actual product.

Check: Use It0/1

1. Two vendors quote different public leaderboard scores, and you must pick an agent for your support product. What is the best plan?Design

Ship It

outputs/skill-benchmark-harness.md builds a SWE-bench-style harness for any codebase-task pair, with FAIL_TO_PASS / PASS_TO_PASS gating.

Author's Note

I treat a leaderboard number as a claim I still have to check, and I don't repeat it until I have. My bet is that a 50% that may be 35% on a cleaner variant tells you more about the benchmark than about the model. A public score can narrow the field for me, but it can't pick the agent. That job belongs to evals I wrote against my own tasks, because only those can show the worst 1% of runs, which an average will never show me.

Exercises

Port the toy harness to a real repo of yours. Write 3 FAIL_TO_PASS tests for known bugs.
Add a step-count metric. On your 3 tasks, how many agent steps does each resolution take?
Read the SWE-bench+ paper. Implement a solution-leakage check that pattern-matches the issue text against the diff.
Download a GAIA question from the public split and trace what a GPT-4-class agent would do. Which tools does it need?
Read AgentBench's per-environment breakdown. Which environment mirrors your product surface, and what does "SOTA" look like there?

Key Terms

Term	What people say	What it actually means
SWE-bench	"Code agent benchmark"	2,294 GitHub issues; patch must flip FAIL_TO_PASS tests
SWE-bench Verified	"Clean SWE-bench"	500 human-curated tasks, OpenAI
FAIL_TO_PASS	"Fix gate"	Tests previously failing that must pass after the patch
PASS_TO_PASS	"No-regression gate"	Tests that were passing and must still pass
GAIA	"Generalist benchmark"	466 total (300 held out for the private leaderboard), human-easy / AI-hard multi-tool questions
AgentBench	"Multi-env benchmark"	8 environments; long-horizon multi-turn
Contamination	"Training-set leak"	Benchmark tasks present in model training
SWE-bench+	"Contamination audit"	32.67% solution leakage found in successful SWE-bench patches