LearnPython

Benchmarks: WebArena and OSWorld

WebArena tests web-agent capability across four self-hosted apps. OSWorld tests desktop-agent capability across Ubuntu, Windows, and macOS. At release (2023–2024) the best agents trailed humans badly on both: 14.41% against 78.24% on WebArena, 12.24% against 72.36% on OSWorld. Those numbers, and the failure modes behind them, are what this lesson teaches you to measure.

TIME: ~60 minutesPREREQUISITES: Phase 14 · 19 (SWE-bench, GAIA)

Learning Objectives

Describe WebArena's four self-hosted apps and why execution-based evaluation matters.
Explain why OSWorld uses real OS screenshots instead of accessibility APIs.
Name the two primary OSWorld failure modes: GUI grounding and operational knowledge.
Summarize what OSWorld-G and OSWorld-Human add on top of the base benchmark.

The Problem

Generalist agents can call tools. The harder question is whether they can drive a browser through 20 clicks to finish a shopping checkout, or configure a Linux box with nothing but a keyboard and mouse. WebArena and OSWorld exist to answer exactly that, and they answer it by checking what actually happened in the app rather than what the agent said it did.

Warm-up0/2

1. A benchmark runs agents against a live public website whose layout and data change every week. What is the main problem for comparing scores over time?Warm-up

2. In SWE-bench, how is a proposed patch judged correct?Warm-up

Check: The Problem0/1

1. An agent's final message says the order was placed, but the evaluator marks the task failed. Which should you trust, and why?Apply

The Concept

WebArena (Zhou et al., ICLR 2024)

WebArena is a benchmark of 812 long-horizon tasks that an agent must complete by operating real web apps. The apps are self-hosted, so every run hits the same pinned software.

Four apps: a shopping site, a forum, a GitLab-like dev tool, and a business CMS.
Plus utilities: map, calculator, scratchpad.
Evaluation is execution-based via gym APIs. Was the order placed, was the issue closed, was the CMS page updated?
At release, the best GPT-4 agent hit 14.41% success against a human 78.24%.

Self-hosting is what keeps the benchmark honest. The target apps are pinned and reproducible, so a failed run means the agent failed, not that a live site changed under it.

Extensions

VisualWebArena adds visually grounded tasks where success depends on interpreting images. Screenshots become first-class observations.
TheAgentCompany (Dec 2024) adds a terminal and coding, which makes it closer to a real remote-work environment.

OSWorld (Xie et al., NeurIPS 2024)

OSWorld is a benchmark of 369 real computer tasks where an agent controls actual applications with free-form keyboard and mouse input. It spans Ubuntu, Windows, and macOS, and the agent sees the machine only through 1920×1080 screenshots.

At release the best model scored 12.24% against a human 72.36%.

Primary failure modes

GUI grounding. Mapping pixels to elements. Models struggle to localize UI elements reliably in a 1920×1080 screenshot.
Operational knowledge. Knowing which menu holds the setting, which keyboard shortcut does the job, which preference pane to open. Humans build up this long tail of knowledge over years.

Follow-ups

OSWorld-G is a 564-sample grounding suite plus the Jedi training set. It separates grounding from planning, so you can measure each on its own.
OSWorld-Human is a set of manually curated gold action trajectories. It shows that top agents take 1.4-2.7x more steps than necessary, the trajectory-efficiency gap.

Why this matters

Claude computer use, OpenAI CUA, and Gemini 2.5 Computer Use (Lesson 21) all train on workloads shaped by WebArena and OSWorld. The benchmarks set the target, and the production models are the shipped answer.

Where benchmarking goes wrong

Screenshot-only evals. OSWorld is screenshot-driven. If you evaluate an agent that reads the DOM or accessibility APIs, you skip the grounding challenge entirely.
Ignoring trajectory length. Scoring on success rate alone hides the 1.4-2.7x step inefficiency that OSWorld-Human surfaces.
Stale self-hosted apps. WebArena's apps pin specific versions. Update one without re-curating the tasks and you break comparability.

Check: The Concept0/20

1. What is WebArena?Define

2. What is VisualWebArena, and how does it differ from TheAgentCompany?Define

3. What is OSWorld?Define

4. What is GUI grounding?Define

5. What is operational knowledge in the OSWorld failure analysis?Define

6. What is OSWorld-G?Define

7. What is OSWorld-Human?Define

8. What is trajectory efficiency?Define

9. What are the two primary failure modes OSWorld surfaces?Define

10. Why does WebArena self-host its four target apps?Explain

11. Why does execution-based evaluation matter for WebArena tasks?Explain

12. Why does OSWorld use real OS screenshots rather than accessibility APIs?Explain

13. At release, WebArena's best GPT-4 agent succeeded on 14.41% of tasks while humans succeeded on 78.24%. What does that gap show?Explain

14. Why does OSWorld-G pull grounding apart from planning?Explain

15. How do WebArena and OSWorld relate to computer-use models like Claude computer use and OpenAI CUA?Explain

16. OSWorld's best model at release scored 12.24% against a human 72.36%. Roughly how many times more tasks did humans complete?Apply

17. Two agents pass the same share of OSWorld tasks, but one takes about twice as many steps as the gold trajectory. What does success rate alone hide?Analyze

18. You upgrade one of WebArena's pinned apps and rerun your old tasks. Why can't you compare the new scores with the old ones?Analyze

19. A desktop agent fails often when it only sees screenshots, but most failures disappear when you also hand it the accessibility tree. Which failure mode does this point to?Analyze

20. Two agents both pass 40% of desktop tasks. One mostly clicks beside the right element, and the other mostly picks wrong steps. What does the pass/fail column fail to tell you, and what helps?Analyze

Build It

code/main.py implements a toy web-agent harness:

A minimal "shopping app" state machine: list_items, add_to_cart, checkout.
Gold trajectories for 3 tasks.
A scripted agent that attempts each task.
An execution-based evaluator (state check) and a trajectory-efficiency metric (steps vs gold).

Run it:

python3 code/main.py

The output is per-task success rate and trajectory efficiency, mirroring OSWorld-Human's methodology.

Check: Build It0/4

1. In the toy shopping harness, a scripted agent calls list_items and add_to_cart but never calls checkout. The evaluator checks the final state for a placed order. What result does it record?Apply

2. A scripted agent finishes a toy task in 9 steps, and the gold trajectory takes 4. What is its trajectory efficiency?Apply

3. Per-task step counts for a toy agent against gold are 6 vs 3, 8 vs 4 and 5 vs 5. What are the three trajectory efficiencies?Apply

4. You extend the toy harness with a forum app. Besides the app's state machine, what must you add for it to be a proper benchmark?Design

Use It

WebArena's self-hosted apps run on an internal cluster for continuous evaluation.
OSWorld in a VM fleet for desktop agents.
Computer-use agents (Lesson 21): Claude, OpenAI CUA, and Gemini are the kind of agents these benchmarks are used to measure.
Your own product flows: capture gold trajectories for your top 20 tasks and run agents against them weekly.

Check: Use It0/1

1. You must track a browser agent on your product's top 20 flows every week with scores that stay comparable. Which design fits best?Design

Ship It

outputs/skill-web-desktop-harness.md is a skill that builds a web or desktop agent harness with execution-based eval and a trajectory-efficiency metric.

Author's Note

Success rate is the number everyone quotes, and it's the least informative one on its own. Check what the agent actually did to the app, then count how many steps it took against a gold trajectory. Two agents can land on the same score and cost you very different amounts to run.

When a score is bad, find out why before you change anything. Grounding failures and planning failures look identical in a pass/fail column, but they have different fixes, which is the whole reason OSWorld-G pulls them apart. And pin your target apps. If the environment moves, your numbers stop meaning anything over time.

Exercises

Extend the toy harness with a second app (a forum). Write 3 tasks plus gold trajectories.
Add trajectory-efficiency reporting per task. On your toy, is the agent 1x, 2x, or 3x over gold?
Implement a "distractor" tool that the gold trajectory never uses. Does the scripted agent get tempted?
Read OSWorld-G. How would you separate grounding failures from planning failures in your own evals?
Read WebArena's apps README. What breaks when you upgrade one of the pinned app versions?

Key Terms

Term	What people say	What it actually means
WebArena	"Web agent benchmark"	812 tasks across 4 self-hosted apps; gym-style evaluation
VisualWebArena	"Visual WebArena"	Visually grounded WebArena; screenshots are observations
OSWorld	"Desktop agent benchmark"	369 tasks on real Ubuntu/Windows/macOS
GUI grounding	"Pixel-to-element mapping"	Model localizing UI elements in 1920×1080
Operational knowledge	"OS know-how"	Which menu, which shortcut, which preference pane
OSWorld-G	"Grounding suite"	564 grounding-only samples + training set
OSWorld-Human	"Gold trajectories"	Manual expert action sequences to measure efficiency
Trajectory efficiency	"Steps over gold"	Agent step count divided by human minimum