Async and Hogwild! Inference

Modern LLMs solve hard problems with long chains of reasoning. 5000 tokens of step-by-step logic is common, and deep math problems can run to tens of thousands. At 35 tokens/sec decode on a 70B model, 50k tokens takes 24 minutes. That is not interactive.

Speculative decoding (Phase 10 · 15) gets you a 3-5x speedup by parallelizing within one sequence. Past that, the sequential dependency of autoregressive decoding is the ceiling, because each new token depends on every prior token.

So the next question is whether you can parallelize across sequences. Run several copies of the same model on the same problem, let them cooperate, and have them split the work.

Prior work does this in a few ways: voting ensembles (run N models, pick the majority answer), tree-of-thought (branch reasoning paths and recombine), and multi-agent frameworks (give each agent a sub-task and use a coordinator). Each helps in specific domains, and each brings its own coordination machinery: voting rules, branch-and-prune logic, agent-to-agent messaging protocols.

Hogwild! Inference drops that machinery. N workers share a single KV cache, and each one reads the others' generated tokens as if they were its own context. With no training or fine-tuning, the workers work out how to divide the task. Reasoning models (QwQ, DeepSeek-R1, Claude-family reasoning mode) can read the shared cache and write things like "worker 2 already handled the base case, so I'll work on the inductive step."

The speedup depends on the workload, and as of April 2026 the method is experimental. It is still worth knowing, because it is a different way to spend parallel hardware on one problem.

The setup

Start N worker processes, all running the same LLM. Instead of per-worker KV caches, keep one shared cache. When worker i generates token t_j, the token is written into the shared cache at the next position. When worker k takes its next step, it reads the current state of the cache (which holds everything the N workers have generated so far).

At step time, workers race to write tokens. There is no per-worker position index, so the cache is a single growing sequence and order is set by write arrival time.

Why coordination emerges

The workers share a prompt, typically something like "You are one of N instances working together on this problem. Each instance reads the shared memory and can see what other instances have written. Avoid redundant work." That prompt plus the shared cache is enough. Reasoning models read the cache, notice which parts of the problem have been attempted, and often pivot to parts nobody has touched.

The Hogwild! paper (Rodionov et al., 2025) reports observations like:

• Workers formulate plans and communicate them to other workers via the cache.
• Workers notice errors in other workers' reasoning and call them out.
• Workers adapt when a plan fails and propose alternatives.
• When prompted to check for redundancy, workers detect it and pivot.

None of this needs fine-tuning. The behavior comes from reasoning the model already has.

The naming

The paper's name riffs on Hogwild! SGD (Recht et al., 2011), an asynchronous-update optimizer. The analogy is direct. SGD's asynchronous workers all write to a shared parameter vector; Hogwild! Inference's workers all write to a shared KV cache. Both rely on empirical convergence rather than synchronization guarantees.

RoPE makes this tractable

Rotary Position Embeddings (RoPE, Su et al. 2021) encode position information via rotation in the Q and K vectors. Because positions are rotations and not baked-in offsets, a token's position can shift without recomputing the KV cache entry. When worker i writes into the shared cache at position p, other workers reading that position can use the cached entry directly, with no re-rotation.

With learned or absolute positions, Hogwild! would need cache invalidation on each concurrent write. RoPE keeps the cache stable.

Wall-time math

Let T_serial be the time for one worker to solve the problem alone. Let p be the task-level parallelizable fraction. Let c be the per-step coordination overhead (reading the extended cache, deciding what to write).

Single-worker time: T_serial. N-worker Hogwild! time, if coordination is free: T_serial * ((1 - p) + p / N). Classic Amdahl. With coordination overhead: T_serial * ((1 - p) + p / N) + c * steps_per_worker.

For a worker to be productive, c must be small relative to the per-step decode time. On reasoning models producing 5k+ tokens, workers can afford hundreds of tokens of coordination overhead and still come out ahead. On short chat tasks, coordination dominates and Hogwild! is worse than serial.

Concrete example

Reasoning problem: 10k tokens of chain-of-thought. Suppose the problem has p = 0.7 parallelizable content (different proof strategies, different case analyses) and c = 200 tokens of coordination overhead per worker. With N = 4 workers:

• Serial time: 10000 decode steps.
• Hogwild! time: 10000 * (0.3 + 0.7 / 4) + 200 * 4 = 10000 * 0.475 + 800 = 5550 decode steps.
• Speedup: 10000 / 5550 = 1.8x.

That is modest. On longer reasoning problems (50k tokens), the coordination overhead amortizes and the speedup pushes 2.5-3x. Think of it as thread-level parallelism for inference.

When to reach for Hogwild!

• Long reasoning problems (thousands of tokens) where the task can be parallelized across independent sub-goals.
• Reasoning models trained to think step by step. Non-reasoning models do not self-coordinate well.
• Single-node deployments with enough VRAM to hold the shared cache plus N worker processes. The cache is shared, but each worker has its own activation memory.

When not to

• Short interactive chat. Coordination overhead dominates.
• Tasks that don't parallelize, like a single linear proof or a single compilation. N=1 is the best you can do.
• Non-reasoning models, where little coordination emerges.
• Multi-node deployments. The shared cache needs fast cross-worker synchronization. Intra-node is fine, but cross-node latency hurts badly.

The experimental status

As of April 2026, Hogwild! is a research method with an open-source PyTorch implementation. Production adoption has not happened. Three blockers:

1. Shared KV cache management across concurrent processes is non-trivial engineering.
2. Emergent coordination is task-dependent; benchmarks are still being built.
3. The speedups are modest compared to what speculative decoding already delivers, and the two can be combined but the combined engineering is another layer.

Worth knowing and worth experimenting with, but I would not bet a product on it yet.

code/main.py implements a toy Hogwild! simulator:

• Two worker processes, each a deterministic "LLM" that produces one of several token categories (work-token, observe-token, coordinate-token) with known probabilities.
• A shared cache (just a list of tokens) that both workers read and write.
• A simple coordination logic: when a worker sees that the other has already produced enough work tokens in a category, it picks a different category.

The simulator runs for a fixed step budget and reports:

• Total work-tokens produced.
• Total wall time (number of worker steps).
• Effective speedup over a single worker.
• A trace of which worker wrote which token.

Step 1: the shared cache

A list that both workers append to. A real implementation would need locking (Python threading.Lock). The simulator skips it and has the workers take turns within each step.

Step 2: the worker loop

Each worker, on each step:

• Reads the current shared cache.
• Decides what category of token to write based on what is already there.
• Writes one token.

Step 3: the coordination heuristic

If category X already has K tokens in the cache and worker's intended category is X, worker switches to category Y. This is a toy stand-in for the reasoning-model behavior of "notice this is already covered, do something else instead."

Step 4: measured speedup

Run the simulator with N=1 worker and with N=2 workers, same step budget, and compare unique progress. With strong coordination, N=2 lands at roughly 1.5-1.8x the N=1 baseline because the workers divide the task.

Step 5: stress the coordination

Reduce the coordination heuristic's sensitivity and run again. At weight 0.0, both workers stay in category A, redundant tokens are counted once, and the speedup falls to about 1x, so the second worker adds almost nothing. That matches the paper's point that the trick depends on the workers being able to self-coordinate.

As of April 2026, Hogwild! in production is research-grade. The reference implementation from Yandex/HSE/IST is PyTorch-based and targets single-node multi-process setups on DeepSeek-R1 and QwQ models.

Pragmatic adoption path:

1. Profile your reasoning-task workload. Measure the fraction of tokens that are exploratory (multiple strategies, case analyses, search) vs linear.
2. If exploration dominates, run a two-worker Hogwild! experiment. Measure wall-time improvement.
3. If the improvement is under 1.3x, you are in the coordination-dominated regime. Revert to single-worker.
4. If the improvement is over 1.5x, push to N=4 and measure again. Diminishing returns typically hit around N=4-8.

You can combine it with speculative decoding, since each Hogwild! worker can use spec decode on its own. The two speedups roughly multiply, so a 3x spec decode gain and a 1.8x Hogwild! gain give an effective 5.4x over naive single-worker decoding.

This lesson produces outputs/skill-parallel-inference-router.md. Given a reasoning workload profile (token budget, task parallelism profile, model family, deployment target), it routes between voting, tree-of-thought, multi-agent, Hogwild!, and speculative decoding strategies.

1. Run code/main.py with the default settings. Confirm the N=2 Hogwild! configuration produces more work-tokens than the N=1 baseline in the same wall time.

2. Reduce the coordination heuristic's strength (set coordination_weight=0.1). Re-run. Show that the speedup collapses. Explain why: the workers duplicate effort when they cannot coordinate.

3. Compute the expected Hogwild! speedup for a 50k-token reasoning task with p=0.8, c=500 and N=4 workers. Do the same for a 1k-token chat task with p=0.3, c=200 and N=4. Why is one a win and the other a loss?

4. Read the Hogwild! paper's Section 4 (preliminary evaluation). Identify the two failure modes the authors report. Describe how a better coordination prompt might mitigate each.

5. Combine Hogwild! with speculative decoding in the toy: each worker uses a 2-token spec-decode internally. Report the multiplicative speedup. What bookkeeping problem arises when two workers both want to extend the same shared-cache prefix?

• Rodionov et al.: Hogwild! Inference: Parallel LLM Generation via Concurrent Attention (arXiv:2504.06261): the Hogwild! paper, preliminary evaluation on QwQ and DeepSeek-R1
• Recht, Re, Wright, Niu: Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (arXiv:1106.5730, NeurIPS 2011): the original Hogwild!, the naming origin
• Su et al.: RoFormer: Enhanced Transformer with Rotary Position Embedding (arXiv:2104.09864): RoPE, the property that makes shared-cache inference tractable
• Yao et al.: Tree of Thoughts: Deliberate Problem Solving with Large Language Models (arXiv:2305.10601): the tree-of-thought reasoning strategy Hogwild! sits orthogonal to
• Leviathan et al.: Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192): speculative decoding, the within-sequence parallelism Hogwild! composes with
• Hogwild! reference PyTorch implementation: the single source of truth for the paper's experiments