Theory of Mind and Emergent Coordination

Multi-agent coordination often looks magical: agents divide labor, anticipate each other, avoid redundancy. Usually that "emergence" is an artifact of prompt engineering. Someone told the agents to "coordinate," and when you remove the prompt, the coordination goes with it.

Riedl's 2025 finding is stricter. Under controlled conditions, coordination emerged only when agents were prompted to reason about other agents' minds (ToM). Without the ToM prompt, the coordination patterns even strong models showed did not survive statistical controls. This matters in production, because teams ship "multi-agent coordination" features that are prompt-dependent and brittle.

Here I treat ToM as a specific capability (reasoning about beliefs about beliefs), build a minimal ToM-aware agent, and measure what real coordination looks like next to what prompt dressing looks like.

What ToM means

Developmental psychology gives the ladder. A 3-year-old thinks anyone's inner world matches theirs. A 5-year-old understands that others hold different beliefs. A 7-year-old can reason about beliefs about beliefs ("she thinks that I think the ball is under the cup"). Those are zeroth-, first-, and second-order ToM.

For LLM agents, the orders map to:

• Zeroth-order: no model of others. The agent acts on its own observations only.
• First-order: the agent has a model of each other agent's beliefs. "Alice believes X."
• Second-order: the agent models recursive beliefs. "Alice believes that Bob believes X."

Li et al. 2023 found that first- and second-order ToM emerge in LLM agents in cooperative games, but degrade over long horizons and with unreliable communication.

The Sally-Anne test, in brief

A 1985 false-belief test: Sally puts a marble in basket A and leaves. Anne moves it to basket B. Where will Sally look when she returns? A child with first-order ToM says basket A, because Sally's belief differs from reality. A child without it says basket B.

GPT-4-era LLMs pass Sally-Anne-style tests when the question is posed plainly. They fail when the narrative is long, the scene changes several times, or the question is phrased indirectly. That is the practical 2026 state of ToM in production LLMs.

Riedl's coordination measurement

Riedl (arXiv:2510.05174) built a population-scale test: N agents, a cooperative objective, variable prompt conditions. It measures three things:

1. Identity-linked differentiation. Do agents develop stable role distinctions over time?
2. Goal-directed complementarity. Do agents' actions complement each other (different subtasks) rather than duplicate?
3. Higher-order synergy. A statistical measure of whether the group achieves what no subset could.

The result: all three metrics produce signal above baseline only under the ToM prompt condition. Without it, metrics hover near chance for moderate-capacity models. Large models show some coordination without explicit ToM prompting, but the effect is smaller than with it.

The coordination illusion

Without statistical controls, "emergent coordination" in demos usually traces back to one of these:

• Prompt engineering that bakes in coordination (system prompts that say "work together").
• Observer bias: we see the patterns we expect.
• Post-hoc selection of successful runs.

Treat a production system that markets "emergent coordination" without a measurable signal as marketing. Measure before claiming.

A minimal ToM-aware agent

Structure:

agent state:
  own_beliefs:    {facts the agent believes}
  other_models:   {other_agent_id -> {beliefs_the_agent_attributes_to_them}}
  actions_last_N: [history of others' actions]

observation update:
  - update own_beliefs from direct observation
  - update other_models[agent_id] from their action + prior beliefs

action selection:
  - enumerate candidate actions
  - for each, predict what each other agent will do next given their modeled beliefs
  - pick action that maximizes joint outcome under those predictions

The other_models attribute is the ToM state. First-order ToM keeps one level. Second-order adds other_models[i][other_models_of_j], which is what I think agent i thinks agent j believes.

Why long-horizon hurts

Li et al. document that context limits make agents forget which belief belongs to whom, and hallucination adds false beliefs to other-agent models. Both produce "I thought he thought X" errors that compound over time.

Mitigations from the paper and from 2024-2026 follow-ups:

• Explicit ToM state in the prompt. Use a structured format, {agent_id: belief_list}, so retrieval preserves the identity-belief binding.
• Shorter reasoning chains. Fewer ToM updates per turn means less compounding hallucination.
• External ToM store. Keep the model outside the LLM context and inject only the relevant parts each turn.

Where ToM fails in production

• Adversarial settings. An agent with good ToM is easier to manipulate, because you can model what it models of you and exploit that.
• Heterogeneous teams. When the models differ, a ToM model that works for one opponent does not generalize.
• Ground-truth-dependent tasks. ToM is about beliefs. If correctness depends on facts, ToM can be a distraction.

The coordination you can actually measure

Three practical signals separate real coordination from the prompt-dressed kind:

1. Complementarity over time. Over a multi-turn task, do agents' actions cover disjoint sub-tasks?
2. Anticipation. Does agent A's action at turn T+1 depend on a prediction about B's action at T+2 that turned out correct?
3. Correction. When A misreads B's belief at turn T, does A correct by turn T+2?

You can measure all three in a logged multi-agent system. They are the substantive version of the "coordination" narrative.

code/main.py implements:

• ToMAgent (the Agent dataclass in the file), which tracks a list of observations of the other agents.
• A cooperative task: three agents must collect three tokens from three boxes, and each box holds one token. Agents cannot communicate. They infer intent from each other's actions.
• Two configurations: zeroth_order (no ToM, picks uniformly among remaining boxes) and first_order (ToM, avoids boxes it saw others target recently).
• Measurement over 200 randomized trials with a 10-turn budget: full completions, duplications per trial (extra agents targeting a box someone else also targeted), and average turns to completion.

Run:

python3 code/main.py

On my run, the zeroth-order agents averaged 0.96 duplications per trial and 1.85 turns, and the first-order ToM agents averaged 0.00 duplications and 1.00 turns. Both configurations completed 200 of 200 trials inside the budget, so completion rate does not separate them here. Duplications and turns do, and that gap is the measurable coordination effect.

outputs/skill-tom-auditor.md is a skill that audits a multi-agent system's claim of "emergent coordination." It checks for prompt dressing, statistical significance against a control, and measured complementarity.

Coordination claims checklist:

• Control condition. Run a version of your system without the coordination prompt and measure both.
• Statistical test. Is the difference between system and control significant at p < 0.05 on your metric?
• Complementarity measure. Track action-disjointness over time, not just final success.
• Failure-case log. When agents miscoordinate, what does the ToM state look like?
• Model-capacity disclosure. If the effect vanishes on smaller models, say so.

1. Run code/main.py. Confirm first-order ToM cuts duplications per trial from 0.96 to 0.00 and average turns from 1.85 to 1.00. Does the gap persist when you scale to 5 agents and 5 boxes?
2. Implement second-order ToM (agent A models what B thinks about C). Does it improve over first-order? On what tasks?
3. Inject a hallucination into the ToM state: randomly flip one belief per turn. How much does this degrade first-order performance?
4. Read Li et al. (arXiv:2310.10701). Reproduce the "long-horizon degradation" finding: as turns grow from 10 to 30, how does your first-order ToM performance change?
5. Read Riedl 2025 (arXiv:2510.05174). Implement the higher-order synergy statistic on your simulation logs. Is the effect present without the ToM prompt condition?

• Li et al.: Theory of Mind for Multi-Agent Collaboration via Large Language Models: emergent ToM in cooperative games; long-horizon failure modes
• Riedl: Emergent Coordination in Multi-Agent Language Models: population-scale measurement; ToM prompting is the load-bearing condition
• Premack & Woodruff: Does the chimpanzee have a theory of mind?: the 1978 origin of the ToM concept
• Baron-Cohen, Leslie, Frith: Does the autistic child have a theory of mind?: the Sally-Anne paper (1985)