Transfusion: Autoregressive Text + Diffusion Image in One Transformer

The debate over discrete versus continuous image representations is older than LLMs. Continuous representations (raw pixels, VAE latents) preserve detail. Discrete tokens (VQ indices) fit the transformer's native vocabulary but lose detail at the quantization step.

Chameleon and Emu3 went discrete. You get one loss and one architecture, but image fidelity is capped by the tokenizer.

Diffusion models went continuous. Image quality is excellent, but the model is separate from the LLM, the noise schedule takes real engineering, and there is no clean way to fold in text generation.

Transfusion asks whether you can have both: keep images continuous, still train one model, and combine two losses in one gradient step.

The two-loss architecture

A single decoder-only transformer processes a sequence that contains:

• Text tokens (discrete, from a BPE vocab).
• Image patches (continuous, 16x16 pixel blocks projected into the hidden dim with a linear embedding, the same input layer a ViT uses).
• <image> and </image> tags marking where the continuous patches live.

The forward pass runs once. Each token then gets one of two heads:

• Text tokens get standard cross-entropy on the vocab-logits head.
• Image patches get a diffusion loss on the continuous patches: predict the noise that was added to each patch.

Gradients flow through the shared transformer body, so both losses improve the same weights at the same time.

Attention mask: causal text + bidirectional image

Text has to stay causal. If a text token can see future text, teacher forcing breaks. An image is different. Its patches are one snapshot, so within the same image block they should attend to each other in both directions.

The mask:

M[i, j] = 1 if:
  (i is text and j is text and j <= i)   # causal for text
  OR (i is image and j is image and same_image_block(i, j))   # bidirectional within image
  OR (i is text and j is image and j < i_image_end)   # text attends to previous images
  OR (i is image and j is text and j < i_image_start)   # image attends to preceding text

You build it as a block-triangular mask, and the same mask is used at training and at inference.

Diffusion loss inside the transformer

The diffusion loss is the standard one. Add noise to an image patch and ask the model to undo it, either by predicting the noise or the clean patch (the two are equivalent). Transfusion's version uses flow matching, where the model predicts the velocity field from noisy to clean.

During training:

1. For each image patch x0, sample a random timestep t.
2. Sample noise ε, compute xt = (1-t) * x0 + t * ε (linear interpolation for flow matching).
3. The transformer predicts v_theta(xt, t); loss = MSE(v_theta(xt, t), ε - x0).
4. Backprop alongside the text NTP losses from the same sequence.

At inference, text tokens come from standard autoregressive sampling. Image patches come from a diffusion sampling loop (10-30 steps typical), conditioned on the text tokens before them.

MMDiT: Stable Diffusion 3's variant

Stable Diffusion 3 (Esser et al., March 2024) shipped MMDiT (Multimodal Diffusion Transformer) around the same time as Transfusion, and the two architectures are siblings.

MMDiT differs in three ways:

• Modality-specific weights per block. Each transformer block has separate Q, K, V, and MLP weights for text tokens and image patches. Attention is joint across modalities, and everything else is modality-specific.
• Rectified flow training. This is a specific flow-matching variant with known sampling and simpler math than DDPM.
• Scale. MMDiT is the backbone for SD3 (2B and 8B param variants), while the Transfusion paper scales to 7B.

Both land on the same core idea: one transformer runs NTP on text and diffusion on continuous image representations.

Why this beats Chameleon-style

The quality gap between continuous diffusion and discrete NTP for image generation is measurable. The Transfusion paper reports:

• At 7B params, it beats a same-size Chameleon-style model on FID by 3-5 points.
• No tokenizer training. The image encoder is a linear projection to the hidden dim, same as a ViT's input layer.
• Inference can denoise image patches in parallel, which autoregressive image tokens can't do.

The downside is that Transfusion is a dual-loss model, so training dynamics are trickier. Loss weights need tuning, and a schedule mismatch between NTP and diffusion can let one head dominate.

What sits downstream

Janus-Pro (Lesson 12.15) refines Transfusion by decoupling the vision encoder for understanding from the one for generation (SigLIP for one, VQ for the other) while sharing the transformer body. Show-o (Lesson 12.14) swaps diffusion for discrete diffusion (masked prediction). The unified-generation family branches quickly after Transfusion.

Production VLMs that emit images in 2026 (Gemini 3 Pro, GPT-5, Claude Opus 4.7's image generation path) almost certainly use some descendant of this family, but the details are proprietary.

code/main.py is a toy Transfusion on a tiny synthetic problem:

• Each caption is a four-integer sequence that encodes a class from 0 to 6.
• Each image is a 4x4 grid of floats whose shade depends on the class.
• The "transformer" is a stand-in. A weight dict with two scale counters fakes the model's output, so the numbers it prints come from the loss plumbing, not from a trained network.
• Each step computes a text loss and a flow-matching MSE loss on a noised image, then sums them with separate weights.
• The block-triangular attention mask is built explicitly for a sequence with one image block and printed.

The transformer is fake. The two-loss plumbing and the mask construction are the real artifacts. The code has no generation loop, so the inference procedure above is something to read, not run.

This lesson produces outputs/skill-two-loss-trainer-designer.md. Give it a new multimodal training task (text + image, text + audio, text + video) and it designs the two-loss schedule (loss weights, mask shape, shared vs modality-specific blocks) and flags implementation risks.

1. A Transfusion-style model trains 70% text tokens and 30% image patches. The image diffusion loss is ~10x the text NTP loss in magnitude. What loss weights balance them?

2. Implement the block-triangular mask for a sequence: [T, T, <image>, P, P, P, P, </image>, T]. Mark each entry 0 or 1.

3. MMDiT has modality-specific QKV weights. What parameter count overhead does this add vs Transfusion's fully-shared transformer? At 7B params, is it worth it?

4. Generation: given a text prompt, the model runs NTP for 50 tokens, then hits <image>, then runs diffusion on 256 patches over 20 denoise steps. How many forward passes total?

5. Read SD3 paper Section 3. Describe rectified flow and why it converges in fewer inference steps than DDPM.

• Zhou et al., Transfusion (arXiv:2408.11039)
• Esser et al., Stable Diffusion 3 / MMDiT (arXiv:2403.03206)
• Peebles & Xie, DiT (arXiv:2212.09748)
• Zhao et al., MonoFormer (arXiv:2409.16280)
• Xie et al., Show-o (arXiv:2408.12528)