MARL — MADDPG, QMIX, MAPPO

LLM-agent systems increasingly train policies for inter-agent coordination: when to defer, when to act, which peer to call. The literature on how to train those policies is Multi-Agent Reinforcement Learning (MARL), which predates the LLM wave and has a small set of dominant algorithms.

MARL papers are painful to read without the pattern vocabulary. Centralized training with decentralized execution (CTDE), value decomposition, and centralized critics are not buzzwords. Each one answers a specific problem:

• Independent RL (each agent learns alone) is non-stationary from each agent's perspective, because the other agents keep changing underneath it.
• Centralized RL (one agent controls the whole team) does not scale and breaks the execution constraint that each agent acts on local information.
• CTDE splits the difference: train with global information, deploy with local policies.

Three environments the papers use

• Particle World (multi-agent particle env). Simple 2D physics with cooperative and competitive tasks. MADDPG's original testbed.
• StarCraft Multi-Agent Challenge (SMAC). Cooperative micro-management with partial observation. QMIX's testbed. Discrete actions, continuous states.
• Google Research Football, Hanabi, MPE. MAPPO's benchmarks.

The environments differ in action and observation types, and each algorithm is picked to match.

MADDPG (2017): the CTDE pattern

Each agent i has an actor mu_i(o_i) that maps its own observation to an action. Each agent also has a critic Q_i(x, a_1, ..., a_n) that sees all observations and all actions during training. The actor is updated by policy gradient against the critic's evaluation.

actor update:    grad_theta_i J = E[grad_theta mu_i(o_i) * grad_a_i Q_i(x, a_1..n) at a_i=mu_i(o_i)]
critic update:   TD on Q_i(x, a_1..n) given next-state joint estimate

CTDE pays off because at training time you know everyone's actions, and using them reduces the variance of each critic. At deploy time each agent sees only o_i and calls mu_i(o_i).

The failure mode is scale. Critic input grows with the number of agents (it includes all actions), so MADDPG does not scale past roughly 10 agents without approximations.

QMIX (2018): value decomposition

QMIX is cooperative only. The joint value is a monotone function of the per-agent Q-values:

Q_tot(tau, a) = f(Q_1(tau_1, a_1), ..., Q_n(tau_n, a_n)),   df/dQ_i >= 0

Monotonicity means argmax_a Q_tot can be found by each agent choosing argmax_{a_i} Q_i independently, which is the decentralized execution property you need. At training time a mixing network produces Q_tot from the per-agent Qs.

QMIX wins on SMAC because cooperative StarCraft micro-management has homogeneous agents, local observations, and a global reward, which is a natural fit for value decomposition.

The monotonicity constraint is restrictive. Some tasks have reward structures that are not monotone decomposable (one agent sacrificing for the team), and extensions like QTRAN and QPLEX relax the constraint.

MAPPO (2022): the overlooked default

Multi-Agent PPO is PPO with a centralized value function. Each agent has its own policy, and the value functions (shared or per-agent) see the full state. Yu et al. benchmarked MAPPO against MADDPG, QMIX, and their extensions on five benchmarks and found:

• MAPPO matches or beats off-policy MARL methods on particle-world, SMAC, Google Research Football, Hanabi, and MPE.
• It needs minimal hyperparameter tuning.
• Training is stable and reproducible across seeds.

The community underrated on-policy MARL until this paper. In 2026 MAPPO is the default baseline for cooperative MARL, and any new method must beat it.

Why LLM-agent engineers should care

There are three direct uses:

1. Router training. A meta-agent chooses which sub-agent handles a task. That is a MARL problem with N decentralized sub-agents and one centralized router, and MAPPO fits.
2. Role emergence. In generative-agent simulations, training agents to adopt complementary roles over time is a MARL problem in disguise. QMIX-style value decomposition pushes toward complementarity by construction.
3. Multi-agent tool use. When agents share tools and compete for budget, training them via CTDE produces deployable local policies that respect resource constraints.

The practical caveat: in 2026 most production LLM-agent systems prompt their policies instead of training them. MARL enters when you have lots of interaction data, a clear reward signal, and the willingness to invest in training infrastructure.

CTDE as a design pattern beyond RL

Even without training, CTDE is a useful architectural pattern. During design, assume full team visibility. At runtime, enforce decentralized execution so each agent sees only o_i.

The pattern forces you to keep per-agent state explicit and to think about partial observability up front. Many production multi-agent systems silently assume shared state everywhere, and CTDE discipline catches that early.

The non-stationarity problem

When multiple agents learn at once, each agent's environment (which includes the other agents' policies) is non-stationary, and the classical single-agent RL proofs break. The three algorithms here each address it differently:

• MADDPG: the global critic sees all actions, so its value estimate is stationary.
• QMIX: value decomposition moves learning into a joint-Q space where optimality is well-defined.
• MAPPO: the centralized value function dampens variance from other agents' policy changes.

In LLM-agent systems non-stationarity shows up as "my agent worked last month, and now that the upstream agent changed, mine misbehaves." Training with CTDE is the principled fix. Prompt-level fixes are faster but less durable.

What this lesson does not cover

Training actual networks is a Phase 09 topic. The code here uses scripted-policy versions that demonstrate the CTDE, value-decomposition, and centralized-value patterns without gradient updates. Get the patterns down first, then pick up a full MARL library (PyMARL, MARLlib, RLlib multi-agent).

code/main.py runs four scripted policies on a tiny 2-agent cooperative grid-world. No networks are trained.

• Environment: 2 agents and 2 pellets on a 4x4 grid. Each pellet is collected when an agent steps onto it, and the episode ends when both are gone. The score is steps to goal.
• run_independent: each agent heads for its own nearest pellet and ignores the other agent, so both often chase the same one. Baseline.
• run_maddpg_style: a central assignment (the stand-in for the centralized critic) gives each agent a distinct pellet by minimizing total Manhattan distance, and each actor then steps toward its own target.
• run_qmix_style: the same assignment framed as value decomposition, with a special case when one pellet remains.
• run_mappo_style: calls the MADDPG-style runner directly, standing in for a centralized value function.

Each runner is benchmarked on the same 500 seeded episodes and reports average steps-to-goal.

Run:

python3 code/main.py

Expected output: the independent baseline averages 3.21 steps, and the three CTDE-style runners all average 2.73. The identical numbers are expected, since the scripted variants share one assignment rule. What the gap shows is the value of coordination, not a ranking of the algorithms.

outputs/skill-marl-picker.md is a skill that picks a MARL algorithm for a given multi-agent task. It weighs cooperative versus competitive, homogeneous versus heterogeneous, action-space type, scale, and reward signal.

MARL in production is rare. When you do use it:

• Start with MAPPO. The 2022 paper established it as the baseline, and reproducing it first saves weeks of chasing fancier methods.
• Log each agent's observation and action stream. Debugging MARL without per-agent traces is close to impossible.
• Separate training code from execution code. CTDE is a discipline, so make the execution path see only o_i.
• Watch reward shaping. MARL is very sensitive to reward design. One coordination bug in the shaping and agents learn to exploit it, so run adversarial tests.
• For LLM agents, try prompt-level policies first. Invest in MARL training only when interaction data, a reward signal, and infrastructure are present together.

1. Run code/main.py. Measure the steps-to-goal gap between independent and MAPPO-style agents. Does the gap grow or shrink on a 6x6 grid?
2. Implement a competitive variant: two agents, one pellet, only the first to reach gets reward. Which pattern handles competition cleanly? MADDPG historically.
3. Read MADDPG (arXiv:1706.02275) Section 3. Implement the exact critic update rule symbolically in pseudocode in your own words.
4. Read MAPPO (arXiv:2103.01955). Why do the authors argue centralized value + PPO beats off-policy MARL on their benchmarks? List the three strongest claims.
5. Apply CTDE as a design pattern to a hypothetical LLM-agent system (e.g., research agent + summarizer + coder). What is the joint information available at design time that is not available at runtime?

• Lowe et al., Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments: MADDPG; NeurIPS 2017
• Rashid et al., QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning: QMIX; ICML 2018
• Yu et al., The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games: MAPPO; NeurIPS 2022
• BAIR blog post on MAPPO: readable framing of the MAPPO result
• SMAC repository: StarCraft Multi-Agent Challenge