Evaluation and Coordination Benchmarks

When a paper claims "our multi-agent system is better," ask three things: better than what, on what, and measured how? In 2023 and 2024 the answers were a mess. Everyone picked their own metrics, baselines, and task sets, so two papers on the same idea could not be compared. The 2025-2026 benchmarks added structure.

Shared benchmarks are not enough on their own, because a benchmark that leaks into training stops measuring anything. SWE-bench Verified became partially contaminated in training corpora by mid-2025 and frontier scores inflated. Pro was designed as an uncontaminated reality check.

This lesson covers five 2026 benchmarks, what each one measures, and how to read a benchmark claim skeptically.

MultiAgentBench (MARBLE), ACL 2025

arXiv:2503.01935. It evaluates four coordination topologies (star, chain, tree, graph) on research, coding, and planning tasks. Milestone-based KPIs track partial progress, so a run that gets halfway still earns credit instead of scoring zero.

Reported results:

• Graph topology is best for research scenarios, since it supports any-to-any critique.
• Chain is best for stepwise-refinement coding.
• Star is best for fast-factual consolidation.
• A coordination tax appears past ~4 agents on graph.
• Cognitive planning adds ~3% milestone achievement across topologies.

Use it when you want to compare coordination topologies on equal terms. The MARBLE repo (https://github.com/ulab-uiuc/MARBLE) provides the evaluator.

COMMA: multimodal asymmetric information

COMMA covers tasks where agents have different observation modalities and must coordinate without sharing everything they see. The reported result is uncomfortable: frontier models including GPT-4o struggle to beat a random baseline on agent-agent collaboration. The signal is that multimodal coordination is under-trained and under-evaluated. LLMs cope with single-modality cooperation, and multimodal coordination falls apart.

Use it when your system has multimodal or asymmetric-information coordination. COMMA's null result is a warning to measure before you claim anything.

MedAgentBoard: a domain stress test

arXiv:2505.12371. Four medical task categories: diagnosis, treatment planning, report generation, and patient communication. It compares multi-agent against single-LLM and conventional rule-based systems.

The finding is that multi-agent does not dominate single-LLM on most categories. The advantage is narrow. Task decomposition helps when the subtasks separate cleanly (diagnosis plus treatment), and it hurts when coordination overhead outweighs the gain from specialization (report generation).

Use it when your domain has a clear single-LLM baseline. If MedAgentBoard's result generalizes, a lot of proposed multi-agent systems are over-engineered.

AgentArch: enterprise architectures

arXiv:2509.10769. Enterprise settings layer tool use, memory, and orchestration together. The benchmark isolates what each layer contributes. How much does adding tools help? Memory? Multi-agent orchestration?

Use it when you are designing an enterprise agent stack and need to justify each layer. It keeps you from paying for features whose value you cannot measure.

SWE-bench Pro: the reality check

arXiv:2509.16941. 1865 problems across 41 repositories spanning business apps, B2B services, and developer tools. It is designed to be uncontaminated, with later training cutoffs in mind. Frontier models score ~23% on Pro against 70%+ on Verified, and that gap is the contamination signal.

April 2026 scores:

• Claude Opus 4.7 on Pro: 64.3%. This is reported with explicit agent-teams coordination, and no Anthropic primary source is published yet, so treat it as preliminary.
• Verdent (agent scaffold) on Verified: 76.1% pass@1 (technical report).
• Frontier raw scores on Pro without agent scaffolding: ~23-35% (SWE-bench Pro paper).

The takeaway: "we beat SWE-bench Verified" is no longer evidence of capability, and Pro is the current gating test. Agent-team scaffolding produces measurable gains on Pro (a ~30-40 point delta), which is one of the strongest empirical arguments for multi-agent coordination in 2026.

AAAI 2026 WMAC

The AAAI 2026 Bridge Program workshop on Multi-Agent Coordination (https://multiagents.org/2026/) is the 2026 community focal point for multi-agent AI research. Its accepted papers and proceedings are the canonical venue for evaluating new methods. For production decisions, prefer WMAC-accepted claims over arXiv preprints.

Reading benchmark claims skeptically: the 2026 checklist

When someone claims a multi-agent result, check these in order:

1. Which benchmark, which split? SWE-bench Verified versus Pro matters a lot. A number reported on the wrong split tells you little.
2. Contamination. Was the benchmark released after the model's training cutoff? If not, be careful.
3. Baseline comparison. Against a single LLM, against random, against prior multi-agent work. An untuned version of the same system does not count.
4. Statistical significance. Number of trials, p-value, confidence interval. Frontier models are high-variance, and single runs mislead.
5. Task diversity. One task or many? Production needs generalization.
6. Cost disclosure. Tokens per task and wall-clock time. A 90% solution at 20x the cost is a business decision, not a capability claim.

What the benchmarks measure poorly

• Long-horizon coordination. Days of wall-clock interaction. All current benchmarks run short.
• Adversarial resilience. What happens when one agent is malicious or compromised?
• Drift under deployment. Benchmarks are static and production distributions shift.
• Cost-normalized performance. Most benchmarks report raw accuracy, not accuracy per dollar.

Building an internal benchmark for the axis you care about is often the right move.

code/main.py is a non-interactive walk-through:

• Simulates 3 multi-agent systems on a toy task set, with 40 "seen" tasks and 160 held-out tasks per system.
• Computes MARBLE-style milestone rates for each system.
• Runs a contamination check by comparing accuracy on seen tasks against held-out tasks. A gap above 0.1 gets flagged.
• Subtracts a random baseline from held-out accuracy.
• Prints a checklist of questions to ask about any benchmark claim.

Run:

python3 code/main.py

Expected output: a scorecard with seen and held-out accuracy, the delta between them, held-out milestone rate, cost per task, cost per milestone, and the delta against random, followed by the claims checklist and a few takeaways.

outputs/skill-benchmark-reader.md takes any multi-agent benchmark claim and runs it through the checklist above. The output is a grade plus caveats.

What to do in production:

• Build an internal benchmark that reflects your actual production distribution. Public benchmarks inform your choices but do not replace it.
• Include a random baseline in each comparison. If you cannot beat random by a wide margin on a coordination task, the task may be ill-posed.
• Report cost next to accuracy, both tokens and wall-clock. Ops teams need both numbers.
• Rebuild the benchmark quarterly. Production distributions shift, and a stale benchmark misleads.
• Watch for overfitting to published benchmarks. If your team optimizes specifically for SWE-bench Pro numbers, expect to regress on production.

1. Run code/main.py. Identify which of the three simulated systems has the best cost-per-milestone. Does it match the highest raw-accuracy system?
2. Read MultiAgentBench (arXiv:2503.01935). For your own task domain, decide which of the four topologies MARBLE would recommend. Justify from the paper's results.
3. Read the SWE-bench Pro paper. What specifically makes it contamination-resistant? Could the same technique be applied to other benchmarks you care about?
4. Read COMMA's finding on multimodal coordination. Design a simple multimodal coordination task you could add to your internal benchmark. What would count as a useful signal?
5. Apply the benchmark-claims checklist to one recent multi-agent paper's headline result. What grade would you give the claim?

• MultiAgentBench / MARBLE: topology benchmark with milestone KPIs
• MARBLE repository: reference implementation
• MedAgentBoard: domain stress test where multi-agent often does not dominate
• AgentArch: enterprise agent architectures
• SWE-bench leaderboards: Verified and Pro scores for frontier models
• AAAI 2026 WMAC: the 2026 community focal point