Flow Matching & Rectified Flows

DDPM's reverse process is a 1000-step stochastic walk from N(0, I) back to the data distribution. DDIM collapsed it to 20-50 deterministic steps. You want fewer, ideally one. The blocker is that the ODE behind the reverse process is stiff, because the path it follows is curved.

Suppose the path from noise to data were a straight line. Then a single Euler step from t=1 to t=0 would land on the data. Flow matching builds exactly that: define a straight-line interpolation from x_1 ∼ N(0, I) to x_0 ∼ data, train a vector field v_θ(x, t) to match its time derivative, and integrate the field at inference.

Rectified flow (Liu 2022) goes a step further. A reflow procedure straightens the learned paths iteratively, so each round gives you an ODE closer to linear than the last. After two reflow iterations, a 2-step sampler matches 50-step DDPM quality.

Straight-line flow

Define:

x_t = t · x_1 + (1 - t) · x_0,   t ∈ [0, 1]

where x_0 ~ data and x_1 ~ N(0, I). The time derivative along this straight line is constant:

dx_t / dt = x_1 - x_0

Define a neural vector field v_θ(x_t, t) and train it to match this derivative:

L = E_{x_0, x_1, t} || v_θ(x_t, t) - (x_1 - x_0) ||²

This is the conditional flow matching loss (Lipman 2023). Training is simulation-free: you never unroll the ODE. Just sample (x_0, x_1, t) and regress.

Sampling

At inference, integrate the learned vector field backwards in time:

x_{t-Δt} = x_t - Δt · v_θ(x_t, t)

Start at x_1 ~ N(0, I), Euler-step down to t=0.

Rectified flow (Liu 2022)

Straight-line flow works, but the learned paths are not actually straight. They curve because many x_0s can map to the same x_1. Rectified flow's reflow step:

1. Train flow model v_1 with random pairings.
2. Sample N pairs (x_1, x_0) by integrating v_1 from x_1 to its landing x_0.
3. Train v_2 on those paired examples. Because the pairs are now "ODE-matched", the straight-line interpolant between them is genuinely flatter.
4. Repeat.

In practice 2 reflow iterations get you to near-linear, enabling 2-4 step inference. Distilled few-step models like SDXL-Turbo, SD3-Turbo, and LCM attack the same step-count problem from the distillation side.

Why this won for images in 2024

Three reasons made the difference:

1. Simulation-free training. You never unroll the ODE during training, so the loss is short to implement.
2. Better loss geometry. Straight paths keep the signal-to-noise ratio steadier, while DDPM's ε-loss has poor SNR at the edges of the schedule.
3. Faster inference. 4-8 steps at SDXL-Turbo quality, and 1 step with consistency distillation.

Flow matching with a Gaussian-conditional path is diffusion with a specific noise schedule. Pick the x_t = α(t) x_0 + σ(t) x_1 schedule and flow matching recovers Stratonovich-reformulated diffusion with v = dx_t/dt = α'·x_0 + σ'·x_1. The straight line above is the case α(t) = 1 - t, σ(t) = t, which gives v = -x_0 + x_1 = x_1 - x_0. For Gaussian paths the two are algebraically equivalent.

What flow matching added is a clearer target (a plain velocity), a cleaner loss, and room to experiment with non-Gaussian interpolants.

code/main.py implements 1-D flow matching on a two-mode Gaussian mixture. The vector field v_θ(x, t) is a tiny MLP trained with the straight-line target. At inference, integrate 1, 2, 4, 8, and 20 Euler steps and compare sample quality. The snippets below are illustrative; they follow main.py but compress it.

Step 1: training loss

def train_step(x0, net, rng, lr):
    x1 = rng.gauss(0, 1)
    t = rng.random()
    x_t = t * x1 + (1 - t) * x0
    target = x1 - x0
    pred = net_forward(x_t, t)
    loss = (pred - target) ** 2
    # backprop + update

Step 2: multi-step inference

def sample(net, num_steps):
    x = rng.gauss(0, 1)
    for i in range(num_steps):
        t = 1.0 - i / num_steps
        dt = 1.0 / num_steps
        x -= dt * net_forward(x, t)
    return x

Step 3: compare step counts

main.py prints a histogram for each step count, plus the sample mean and how many samples landed on each side of zero. Compare the 4-step and 20-step histograms by eye. If they look alike, you have cut the sampling cost by 5x for this toy, and latency scales with step count in a real model too.

• Time parameterization. Flow matching uses t ∈ [0, 1] with t=0 at data, t=1 at noise. DDPM uses t ∈ [0, T] with t=0 at data, t=T at noise. Same direction, different scale. Papers get this wrong constantly.
• Schedule choice. Rectified flow's straight line is the standard flow-matching schedule, but you can use cosine or logit-normal t-sampling (SD3 does this) for better scale coverage.
• Reflow cost. Generating the paired dataset for reflow is a full inference pass per sample. Only do reflow when you really need 1-2 step inference.
• Classifier-free guidance still applies. Swap ε for v in the linear combination: v_cfg = (1+w) v_cond - w v_uncond.

Whenever a paper says "faster than diffusion" in 2025-2026, it is almost always flow matching + distillation.

Save outputs/skill-fm-tuner.md. Skill takes a diffusion-style model spec and converts it to a flow-matching training config: schedule choice, time sampling distribution (uniform / logit-normal), optimizer, reflow plan, target step count, eval protocol.

1. Easy. Run code/main.py and compare the 1-step and 20-step histograms against the true two-mode distribution.
2. Medium. Switch from uniform t sampling to logit-normal (concentrates sampling at mid-t). Does the model quality improve?
3. Hard. Implement one reflow iteration: generate paired (x_0, x_1) by integrating the first model, train a second model on the pairs, and compare 1-step sample quality.

Flux.1-schnell is the clearest production example: a flow-matched DiT distilled to 1-4 inference steps while keeping Flux-dev-grade quality. Niels' "Run Flux on an 8GB machine" notebook is the reference deployment recipe: T5 + CLIP encode, quantized MMDiT denoise (in 4 steps for schnell vs 50 for dev), VAE decode. The cost accounting:

The production rule: a flow-matched base plus distillation is the 2026 default for fast text-to-image. Every major vendor ships this combo: SD3-Turbo (SD3 + flow + distillation), Flux-schnell (Flux-dev + rectified-flow straightening), CogView-4-Flash. Pure diffusion bases exist only for legacy checkpoints.

• Liu, Gong, Liu (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow: rectified flow.
• Lipman et al. (2023). Flow Matching for Generative Modeling: flow matching.
• Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis: SD3, rectified flow at scale.
• Albergo, Vanden-Eijnden (2023). Stochastic Interpolants: general framework that covers FM + diffusion.
• Song et al. (2023). Consistency Models: 1-step distillation of diffusion / flow.
• Sauer et al. (2023). Adversarial Diffusion Distillation (SDXL-Turbo): turbo variant.
• Black Forest Labs (2024). Flux.1 models: flow matching in production.