Swarm Optimization for LLMs (PSO, ACO)

You have a prompt that scores 62% on your task eval and you want it higher. Manual tweaking scales badly. Reinforcement learning needs a reward signal and enough rollouts to train on. Backprop through a prompt isn't really possible, because the prompt is a discrete string and not a differentiable parameter.

Classical bio-inspired optimizers were built for exactly this regime: PSO for continuous search spaces, ACO for path selection. Both are gradient-free, population-based, and cheap per evaluation. Put an LLM inside the search step and you get a practical optimizer.

The same patterns work for routing in multi-agent systems. An ACO-style pheromone trail records which agent did best on which task type. The router exploits the trail, and decay lets it rediscover routes when things change.

PSO refresher (Kennedy & Eberhart 1995)

Particle Swarm Optimization keeps a population of particles in a continuous search space. Each particle has a position x_i and a velocity v_i. Each iteration:

v_i <- w * v_i + c1 * r1 * (p_best_i - x_i) + c2 * r2 * (g_best - x_i)
x_i <- x_i + v_i
evaluate fitness(x_i)
update p_best_i if improved
update g_best if global best

p_best is the particle's own best and g_best is the swarm's best. w, c1, and c2 are the inertia, cognitive, and social weights, and r1, r2 are random factors.

PSO on LLM outputs: LMPSO

arXiv:2504.09247 adapts PSO to LLM-generated structured outputs (math expressions, programs). Each particle is a candidate output. Velocity is a prompt describing how to modify the current output toward the personal or global best, and the LLM generates the new output from it. Inertia becomes a prompt like "make small incremental changes."

This works well when:

• The output is structured (parseable, evaluable).
• Fitness is automatic (test runs, arithmetic evaluation).
• The population is small (~10-30 particles), so total LLM calls stay manageable.

It breaks down when fitness needs human review, because the per-iteration cost gets prohibitive.

Model Swarms

arXiv:2410.11163 moves PSO off the output layer and into the model layer. Each particle is an expert LLM (its parameters), and the swarm moves toward the collective best with a gradient-free update. The paper reports a 13.3% average gain over 12 baselines on 9 datasets, with just 200 instances per iteration.

The key insight is that LLM experts sit close together in a shared parameter manifold (adapter weights, LoRA deltas). PSO on that low-dimensional subspace is cheap and effective.

ACO refresher (Dorigo 1992)

Ant Colony Optimization sends ants across a graph where each path carries a pheromone trail. Move probabilities weight by pheromone strength. Ants that finish the task deposit pheromone in proportion to solution quality, and pheromone decays over time.

AMRO-S: ACO for agent routing

arXiv:2603.12933 applies ACO to multi-agent routing. Each task type is a destination and each agent is a possible route. Pheromone strengthens the routes that produce good outputs. The paper's main contributions:

• Interpretable routing evidence. Pheromone strength is a human-readable signal.
• Quality-gated asynchronous update. Pheromone updates only after quality checks pass, which decouples inference from learning.
• 4.7x speedup on the multi-agent routing benchmark.

The quality gate matters. Without it, fast-but-wrong agents accrue pheromone and the system locks in on bad routes.

When to use PSO / ACO for LLMs

Use PSO when:

• The search space is continuous or maps to continuous parameters (prompt embeddings, LoRA weights, numeric generation parameters).
• Fitness is cheap and automatic.
• The population can be small (10-30).

Use ACO when:

• You have a routing or path-selection problem.
• Decisions reinforce over time because the same task types keep coming back.
• You need interpretable evidence for routing decisions.

Skip both when:

• Fitness requires human review (too expensive per iteration).
• The search space is discrete and combinatorial in a way PSO doesn't cover. Use genetic algorithms instead.
• Real-time decisions have strict latency limits. PSO and ACO converge slowly compared to single-pass heuristics.

Why bio-inspired still wins

Gradient-based methods need differentiable signals, and LLM outputs and routing decisions aren't trivially differentiable. Pseudo-gradient methods (reinforcement-learned routers, DPO-style prompt tuners) work but need expensive training.

PSO and ACO need only an evaluator function. If you can score a candidate output or a routing decision, you can optimize over the space. That lowers the bar for applicability a lot.

Practical limits

• Population budget. N particles × T iterations × per-eval cost. At ~$0.02 per LLM call, a 20-particle PSO running 50 iterations costs ~$20. Plan for it.
• Exploration vs exploitation. Pheromone decay rate and PSO inertia trade these off. Decay that is too fast forgets good solutions, and decay that is too slow sticks on early local optima.
• Catastrophic drift. Both algorithms can converge and later diverge if the fitness landscape shifts (a new data distribution). Monitor best-fitness stability.

main.py is stdlib-only Python with two parts:

• run_lmpso is PSO over two numeric prompt parameters (temperature and a top_k weight), with 20 particles and 30 iterations by default. There are no LLM calls. The "generation" step is replaced by a scripted fitness function, so the velocity is a plain number and not a prompt. It records g_best after every iteration.
• run_amro_s is ACO-style routing with a PheromoneRouter: 3 agents, 4 task types, a pheromone table, and 200 routed tasks. Each deposit decays that task type's row and then reinforces the chosen agent, but only if the quality score clears the threshold (0.6). The same task stream also goes through a random router for comparison, and the script compares average quality.

Run:

python3 main.py

On the default seed:

• LMPSO: g_best climbs from 0.96 at iteration 0 to just over 1.0 within about 5 iterations, then plateaus.
• AMRO-S: the pheromone table concentrates on one agent per task type, and ACO routing beats random by about 31% on average quality (0.651 against 0.498). The script does not measure latency or retries.

outputs/skill-swarm-optimizer.md helps you choose between PSO, ACO, genetic algorithms, and gradient-based optimizers for an LLM or agent optimization problem.

• Start small. 10-20 particles and 20-50 iterations. Scale up only if the convergence curve shows a clear gain.
• Log pheromones or g_best every iteration. Debugging a swarm optimizer without a trail is painful.
• Quality-gate updates, especially for ACO routing. Fast-and-wrong agents must not accrue pheromone.
• Reset decay on distribution shift. When your eval distribution changes, aged pheromones are stale. Reset them, or double the decay rate for a while.
• Cap the per-iteration cost. Emit a cost-per-iteration metric. A PSO run that costs $500 per iteration and gains 0.5% isn't shippable.

1. Run main.py. Observe LMPSO convergence. Vary population size 5, 10, 20, 50. At what size does time-to-converge saturate?
2. Implement a "catastrophic drift" experiment: after iteration 30, change the fitness function. How fast does PSO adapt? Does resetting p_best help?
3. Raise the AMRO-S quality gate from 0.6 to 0.7, then remove it entirely. How does convergence change in each case?
4. Read LMPSO (arXiv:2504.09247). Map the paper's "velocity as a prompt" back to your numeric velocity. What is lost in the simulation and what is preserved?
5. Read AMRO-S (arXiv:2603.12933). Implement the decoupled "inference fast-path" with asynchronous pheromone update. How does this change system latency under sustained load?

• Kennedy & Eberhart, Particle Swarm Optimization: the 1995 PSO paper
• Dorigo, Ant Colony Optimization: 1992 ACO foundations
• LMPSO, Language Model Particle Swarm Optimization: PSO for structured LLM outputs
• Model Swarms, gradient-free LLM expert optimization: PSO on model-weight subspace
• AMRO-S, ant-colony multi-agent routing: pheromone-driven routing with quality gate