DualPipe Parallelism

Training a 671B MoE model on 2k H800 GPUs runs into three bottlenecks at once:

1. Memory pressure. Each GPU holds a slice of the model, and activation memory at sequence 8k across 61 layers on 128 heads is enormous.
2. Pipeline bubbles. Traditional pipeline parallelism (GPipe, 1F1B) leaves GPUs idle while they wait for their stage's input or gradient. At 8 stages, roughly 12% of GPU time can be bubble even with 1F1B scheduling.
3. Cross-node all-to-all. MoE with expert parallelism scatters experts across nodes. A forward pass triggers one all-to-all to dispatch tokens to their experts and another to combine the results. At 2k GPUs this easily reaches a 1:1 compute-to-comm ratio.

Each one has its own fix: gradient checkpointing for memory, Zero Bubble (Sea AI Lab, 2023) for pipeline bubbles, expert-parallel comm kernels for all-to-all. DualPipe's contribution is making them work together. It overlaps compute and comm inside a single forward-backward chunk, injects micro-batches from both ends of the pipeline at once, and uses the resulting schedule to hide the all-to-all inside compute windows.

The reported result is a near-elimination of pipeline bubbles, with over 95% GPU utilization in DeepSeek-V3's 14.8T-token training run.

Pipeline parallelism refresher

Split an N-layer model across P devices. Device i holds layers i * N/P .. (i+1) * N/P - 1. A micro-batch flows forward through devices 0 to P-1, then backward from P-1 to 0. Each device can only start its forward stage when the prior device sends its output and can only start backward when the downstream device sends the upstream gradient.

GPipe (Huang et al., 2019) schedules one micro-batch at a time, which wastes most GPU time. 1F1B (Narayanan et al., 2021) interleaves forward and backward passes for multiple micro-batches. Zero Bubble (Qi et al., 2023) splits the backward pass into two parts, backward-for-input (B) and backward-for-weights (W), and schedules them to fill the bubble. After Zero Bubble the pipeline is almost tight.

DualPipe adds two ideas on top of that.

Idea 1: chunk decomposition

Each forward chunk is split into four components:

• Attention. Q/K/V projections, attention, output projection.
• All-to-all dispatch. Cross-node communication that sends tokens to their experts.
• MLP. The MoE expert computation.
• All-to-all combine. Cross-node communication that brings expert outputs back.

A backward chunk adds gradient versions of each of these. DualPipe schedules them so that all-to-all dispatch runs in parallel with the attention compute of the next chunk, and all-to-all combine runs in parallel with the MLP compute of the following chunk.

Idea 2: bidirectional scheduling

Most pipeline schedules inject micro-batches at stage 0 and let them flow toward stage P-1. DualPipe injects from both ends. Stage 0 sees forward micro-batches that originate there, and so does stage P-1. The two streams meet in the middle.

For this to work, device i has to hold both the early-pipeline layer i and the late-pipeline layer P - 1 - i. That is the "dual" in DualPipe: each device keeps two copies of the layers it serves, one per direction. At DeepSeek-V3's scale this is a 2x parameter replication cost, which is affordable because Expert Parallelism already spreads the MoE experts so thin that replicating the non-expert layers is a small addition.

The forward stream in one direction and the backward stream in the other overlap where the bubbles would sit in a single-direction schedule, and that overlap is what squeezes the bubbles out.

A hand-traced schedule

Consider P = 4 ranks, 8 micro-batches, divided 4 forward / 4 reverse. Time moves left to right; rows are device ranks.

           Time →
rank 0:  F1 F2 F3 F4  F5R F6R F7R F8R  B1 B2 B3 B4  ...
rank 1:     F1 F2 F3  F4/F5R F6R F7R   B1 B2 ...
rank 2:        F1 F2  F3/F5R F4/F6R    B1 ...
rank 3:           F1  F2/F5R F3/F6R    ...

Reading the "F4/F5R" notation: rank 1 is running forward of micro-batch 4 (going left-to-right in the pipeline) AND forward of micro-batch 5 (going right-to-left) in the same time slot. That is what "bidirectional" means operationally.

At rank 2 the cross streams overlap sooner, and at rank 0 and P-1 they overlap latest. In the stable middle phase, each rank runs the forward pass of one direction overlapped with the backward pass of the other, so compute stays busy. All-to-all dispatches for the forward pass hide inside backward compute, and all-to-all combines hide inside forward compute.

Bubble accounting

Standard 1F1B pipeline bubble (time wasted per rank):

bubble_1F1B = (P - 1) * forward_chunk_time

Zero Bubble brings that down but not to zero. DualPipe has zero bubble in the stable phase if the micro-batch count is divisible by 2 times the pipeline depth. Warmup and cooldown still carry some bubble, but it does not grow with the number of micro-batches, which is the property the paper highlights.

The marketing term is "bubble-free". The technical claim is narrower: bubbles do not grow with micro-batch count. Sea AI Lab's follow-up analysis (DualPipeV / Cut-in-half) shows full zero-bubble only when Expert Parallelism is not the bottleneck. With EP-driven all-to-all, expect some scheduling compromise.

DualPipeV, the refinement

Sea AI Lab (2025) observed that the 2x parameter replication is wasteful when EP comm overlap is not the point. Their DualPipeV schedule folds the bidirectional injection into a "V-shape" schedule that runs on a single parameter copy. The bubble is slightly larger than DualPipe's, but the memory savings are substantial. DeepSeek added DualPipeV to their open-source DualPipe implementation as an EP-off mode.

The tradeoff:

What it means for a 14.8T-token run

DeepSeek-V3's pre-training consumed 14.8T tokens on 2,048 H800 GPUs in roughly 2.8M GPU-hours. With naive 1F1B, they would have lost 12-15% of that to pipeline bubbles, which is 340-420K GPU-hours, enough to train a full 70B model. DualPipe recovered most of it. Quantifying the contribution directly is difficult without the internal logs, but the paper claims over 95% GPU utilization averaged across training.

For smaller runs (under 1k GPUs), DualPipe is overkill. Pipeline bubbles are smaller relative to total cost, and dense-model training rarely hits the all-to-all bottleneck. For frontier MoE training at multi-thousand GPU scale, it is effectively required.

Where it sits in the stack

• Complementary to FSDP (Phase 10 · 05). FSDP shards the model parameters across ranks; DualPipe schedules the compute across ranks. They combine.
• Compatible with ZeRO-3 gradient sharding, though the bookkeeping for the two-copy replication has to cooperate with ZeRO's sharded gradients.
• Needs custom all-to-all kernels tuned for your cluster topology. DeepSeek's open-source kernels are the reference implementation.

code/main.py is a pipeline schedule simulator. Its summarize(P, M) function returns the bubble fraction for each of 1F1B, Zero Bubble, DualPipe, and DualPipeV, and main() prints it at P=8, M=16, then sweeps micro-batch count and pipeline depth. It is a teaching tool. The bubble formulas are closed-form approximations with unit-normalized chunk costs, so read the numbers as relative comparisons, not as measured speedups.

Run it as is, then edit the P and M values in main() and compare how the schedules rank. The script takes no command-line arguments. DualPipe stays below the others at every setting it sweeps, and DualPipeV sits just above it.

Integration considerations for a real training run:

• Pick a pipeline-parallel depth that divides cleanly into your micro-batch count.
• Make sure your expert-parallel mesh supports bidirectional all-to-all. DeepSeek's kernels are the reference.
• Budget about a week of debugging on the schedule itself the first time. The bookkeeping is fiddly.
• Monitor GPU utilization per rank, not just aggregate. DualPipe's benefit comes from tightening the stragglers.

This lesson produces outputs/skill-dualpipe-planner.md. Given a training cluster specification (GPU count, topology, interconnect, model shape), it recommends a pipeline parallelism strategy, the scheduling algorithm to use, and the expected bubble fraction at the target scale.

1. Run code/main.py and read the 1F1B and DualPipe rows of Step 1 (P=8, M=16). Compute the GPU utilization difference and express it as recovered GPU-hours per million tokens of training.

2. Sketch the schedule table for (P=4, micro_batches=8, schedule=dualpipe) by hand. Mark each time slot with the micro-batch ID and direction. Identify the first time slot where bubbles are absent.

3. Read Figure 5 of the DeepSeek-V3 technical report (arXiv:2412.19437). Identify the overlap window for all-to-all dispatch inside a DualPipe forward chunk. Explain how the compute schedule hides it.

4. Compute the 2x parameter overhead of DualPipe for a 70B dense model with P=8 pipeline stages and a 671B MoE model with P=16 pipeline stages. Show why the MoE case's overhead is proportionally smaller (most parameters are experts, sharded across a large EP group).

5. Compare DualPipe to Chimera (a competing bidirectional scheduler from 2021). Identify the two specific properties DualPipe added that Chimera did not have, using the paper's Section 3.4 as the reference.

• DeepSeek-AI, DeepSeek-V3 Technical Report (arXiv:2412.19437), Section 3.3.2 and Figure 5: the primary DualPipe reference.
• DeepSeek, DualPipe GitHub repository: the open-source reference implementation, including DualPipeV (Cut-in-half) mode.
• Qi et al., Zero Bubble Pipeline Parallelism (arXiv:2401.10241, Sea AI Lab 2023): the Zero Bubble predecessor.
• Sea AI Lab, DualPipe could be better without the Dual: the DualPipeV analysis that informed DeepSeek's EP-off mode.
• Narayanan et al., PipeDream / 1F1B (arXiv:1806.03377, 2018-2021): the 1F1B schedule DualPipe compares against.
• Huang et al., GPipe (arXiv:1811.06965, 2018): the original pipeline parallelism paper and bubble problem.