TAI BUI
Back to curriculum
LearnPython

Emu3: Next-Token Prediction for Image and Video Generation

BAAI's Emu3 (Wang et al., September 2024) is the 2024 result that should have ended the diffusion-versus-autoregressive debate. One Llama-style decoder-only transformer, trained only on next-token prediction over a unified vocabulary of text, VQ image tokens, and 3D VQ video tokens, beats SDXL on image generation and LLaVA-1.6 on perception. There is no CLIP loss and no diffusion schedule. Classifier-free guidance shows up at inference for quality, but training is plain next-token prediction with teacher forcing. The paper was published in Nature. This lesson reads the Emu3 thesis, that a better tokenizer plus scale carries the whole approach, and contrasts it with diffusion.

TIME: ~120 minutesPREREQUISITES: Phase 12 · 11 (Chameleon)

Learning Objectives

  • Explain why Emu3's single-loss next-token objective works despite the long-held assumption that diffusion is required for image quality.
  • Describe the 3D video tokenizer: what a spatiotemporal VQ codebook looks like, why patches span time.
  • Compare Emu3 vs Stable Diffusion XL on (training compute, inference cost, quality ceiling).
  • Name the three roles the same Emu3 model plays: Emu3-Gen (image gen), Emu3-Chat (perception), Emu3-Stage2 (video gen).

The Problem

The conventional wisdom through 2024 was that image generation needs diffusion. Discrete image tokens lose too much information to reconstruct fine detail, the argument went, and autoregressive sampling accumulates error across thousands of tokens. Stable Diffusion, DALL-E 3, Imagen, and Midjourney all use some form of diffusion. Chameleon (Lesson 12.11) partly disproved this at small scale but did not match SDXL on quality.

Emu3 attacked the argument head-on. Its claim: a better visual tokenizer, enough scale, and a next-token loss give diffusion-beating image generation, in the same model that also does perception.

The bet was controversial when it published. Two years on, the open-source unified-generation family (Emu3, Show-o, Janus-Pro, Transfusion) is the default path for research, and production frontier models appear to use some variant of it.

Warm-up0/3
1. In an early-fusion model like Chameleon, how does an image enter the transformer?Warm-up
A photo thumbnail on the left with an arrow through a dashed box marked how in to a box labelled one transformer, early fusion. Under the transformer three word chips, Describe, this and photo, with an arrow up into it.
Typed schematic, not captured. The photo and prompt are illustrative.
2. One shared vocabulary holds both text tokens and image tokens. What decides whether the model emits a text token or an image token next?Warm-up
A vocabulary bar split into a text token half and an image token half. Below it four word chips, Draw, a, cat and a colon, with an arrow to a dashed box marked next token.
Typed schematic, not captured. The prompt is illustrative.
3. A language model writes a sentence one token at a time. What does it condition on when choosing each new token?Warm-up
Five word chips, The, cat, sat, on and the, numbered 1 to 5, followed by an arrow to a dashed box marked token 6.
Typed schematic, not captured. The sentence is illustrative.
Check: The Problem0/1
1. Why did many researchers expect autoregressive image generation to need diffusion instead?Explain
Two panels. Diffusion lists Stable Diffusion, DALL-E 3, Imagen and Midjourney. Token-by-token lists Chameleon, which at small scale did not match SDXL on quality. An arrow leads to a dashed box asking why researchers expected diffusion.
Grouping taken from the lesson's Problem section. Typed schematic, not captured.

The Concept

The Emu3 tokenizer

The visual tokenizer is the key ingredient. Emu3 trains a custom IBQ-class tokenizer (Inverse Bottleneck Quantizer, SBER-MoVQGAN family) at 8x8 resolution reduction per token. A 512x512 image becomes 64x64 = 4096 tokens at a codebook size of 32768.

That is more than Chameleon's 1024 tokens per 512x512 at K=8192, but each token is cheaper (smaller codebook lookups, simpler codec). The metric to watch is reconstruction PSNR: 30.5 dB, competitive with Stable Diffusion's continuous latent space at 32 dB.

For video, a 3D VQ tokenizer encodes a spatiotemporal patch (4x4x4 pixels) into one integer. A 4s clip at 8 FPS has 32 frames. At 256x256 with 4x spatial and 4x temporal reduction, the token count is (256/4) * (256/4) * (32/4) = 64 * 64 * 8 = 32,768 tokens.

Tokenizer quality sets the ceiling, so part of Emu3's contribution is simply that the team trained a very good tokenizer.

Single-loss training

Emu3 uses one objective: next-token prediction on a shared vocabulary of text tokens, 2D image tokens, and 3D video tokens. Per-modality weights balance each modality's contribution during training, and the loss function itself is the same throughout.

The training mix looks like this:

  • Image gen: <text caption> <image> image_tokens </image>
  • Image perception: <image> image_tokens </image> <question> text_tokens
  • Video gen: <text caption> <video> video_tokens </video>
  • Video perception: analogous.
  • Text only: standard NTP.

The model learns from the data distribution when to emit image tokens and when to emit text. Generation comes from predicting image tokens after the <image> tag.

Classifier-free guidance and temperature

Autoregressive image generation improves a lot with classifier-free guidance (CFG) at inference. Emu3 generates twice, once with the full caption and once with an empty caption, then mixes the logits with a guidance weight (typically 3.0-7.0). It is the same trick diffusion uses, applied to token logits.

Temperature matters too. Too high gives artifacts, too low gives mode collapse. Emu3 recommends 1.0 for perception and 0.8 for image generation.

Three roles, one model

Emu3 ships as three APIs on one underlying weight set:

  • Emu3-Gen. Image generation. Input text, output image tokens.
  • Emu3-Chat. VQA and captioning. Input image (tokens), output text.
  • Emu3-Stage2. Video generation and video VQA. Input text or video, output text or video.

There are no task-specific heads, only different prompt templates over the same checkpoint.

Benchmarks

From the Emu3 paper (September 2024):

  • Image generation: beats SDXL on MJHQ-30K FID (5.4 vs 5.6). On GenEval overall it scores 0.54 vs 0.55, which is a statistical tie. Deep-Eval's composite is on par.
  • Image perception: beats LLaVA-1.6 on VQAv2 (75.1 vs 72.4) and roughly matches it on MMMU.
  • Video generation: 4-second-clip quality at competitive FVD with Sora-era publicly benchmarked models.

Emu3 does not win everywhere. It gives up a point here for a point there. Even so, the claim that next-token prediction is enough is defensible across these modalities.

Compute cost

Emu3 was trained on roughly 300 billion multimodal tokens with a 7B-parameter model. GPU-hours come out roughly comparable to Llama-2-7B pretraining (2k-4k GPU-years on A100-class silicon). Diffusion models like Stable Diffusion 3 train on similar budgets but need separate text encoders and a more complex pipeline.

At inference, Emu3 is slower than SDXL per image. 4096 image tokens at 30 tok/s is about 2 minutes per 512x512 image, against 2-5 seconds for SDXL. Speculative decoding and KV-cache optimization narrow the gap without closing it. Autoregressive image generation is compute-heavy, and that is the standing trade-off.

Why it matters

The deeper contribution is conceptual. If next-token prediction scales to match diffusion on image generation, the unified-model path (one loss, one backbone, many modalities) is viable. Future models do not need a separate text encoder, a separate diffusion scheduler, or a separate VAE. One transformer, one tokenizer per modality, and scale.

Show-o, Janus-Pro, and InternVL-U all build on or challenge this thesis. Through 2025, Chinese labs (BAAI, DeepSeek) published in this direction more aggressively than US labs.

Check: The Concept0/25
1. How many tokens does Emu3's tokenizer produce for a 512x512 image, and from what codebook size?Define
A placeholder photo labelled 512x512 image with an arrow into a box labelled Emu3 visual tokenizer, then an arrow to a dashed box marked tokens.
Typed schematic, not captured. The picture is a placeholder photo.
2. What three roles does the same Emu3 weight set play?Define
A box labelled one Emu3 weight set, a single checkpoint, with three arrows down to three dashed boxes marked role 1, role 2 and role 3.
Typed schematic, not captured.
3. What is next-token prediction as a training objective?Define
Six word chips, a, cat, sat, on, a and mat, numbered 0 to 5, above a wide dashed box asking for the training objective on this sequence.
Typed schematic, not captured. The sentence is illustrative.
4. What is an IBQ tokenizer in Emu3?Define
A placeholder photo labelled pixels, an arrow into a box labelled IBQ tokenizer, inverse bottleneck quantizer, then a dashed box marked output and a box labelled transformer.
Name expansion from the lesson. Typed schematic, not captured.
5. What is a 3D VQ tokenizer?Define
A stack of six overlapping video frames labelled clip, 4 s at 8 FPS, 256x256 frames, an arrow into a box labelled 3D VQ tokenizer, then a dashed box marked one token equals question mark.
Typed schematic, not captured. Duration, frame rate and frame size are the lesson's example; the stack is not a frame count.
6. What is classifier-free guidance as Emu3 uses it?Define
A bar chart of five conditional logits, 2.0, 4.0, 1.0, 3.5 and 0.5 for toy codebook entries 0 to 4, with an arrow through a dashed box marked guidance to a box labelled sample.
Logit values are main.py's conditional list; the guidance step is left blank.
7. What is a unified vocabulary in Emu3?Define
A row of three word chips, four green image token squares and three purple video token squares, labelled text tokens, image tokens and video tokens, with an arrow to a dashed box asking what vocabulary these tokens are drawn from.
Typed schematic, not captured. Chips stand for tokens; the caption words are illustrative.
8. What is MJHQ-30K in the Emu3 results?Define
A results sheet with one row. The benchmark column reads MJHQ-30K and the description column is a dashed box with a question mark.
Typed schematic, not captured. Only the benchmark name is given; the rest is blank.
9. Emu3 tokenizes video with 4x4x4 spatiotemporal patches. At the same 4x spatial reduction, why does that give fewer tokens than tokenizing every frame separately?Explain
A strip of 32 thin video frames from frame 0 to frame 31 above a dashed box asking for the tokens for this clip.
Typed schematic, not captured. Frames drawn as thin slivers.
10. What loss does Emu3 use across text, image, and video tokens?Explain
Three rows of token boxes labelled text, image and video, with an arrow down to a dashed box asking for the loss applied to all of these.
Typed schematic, not captured. Boxes stand for tokens.
11. How does Emu3 use classifier-free guidance and temperature at inference?Explain
A box labelled caption plus Emu3, next-token logits, then two dashed boxes marked guidance and temperature, then a box labelled token.
Typed schematic, not captured.
12. Emu3 reports 30.5 dB reconstruction PSNR for its visual tokenizer, against 32 dB for Stable Diffusion's continuous latents. Why does that number matter for generation quality?Explain
Two horizontal bars on a dB axis: Emu3 tokenizer at 30.5 dB and SD continuous latents at 32 dB, with an arrow down to a dashed box asking why the number matters for generation.
Bar lengths drawn from the two values in the stem.
13. Emu3-Gen, Emu3-Chat and Emu3-Stage2 share one checkpoint. What makes that possible?Explain
A box labelled one shared checkpoint with arrows to three boxes named Emu3-Gen, Emu3-Chat and Emu3-Stage2, and an arrow below to a dashed box asking what makes sharing possible.
Typed schematic, not captured.
14. Why does classifier-free guidance raise the inference cost of every sampled token?Explain
Two horizontal bars for the compute of one sampled token, a solid bar for plain sampling and a dashed bar marked question mark for sampling with a guidance weight, above a dashed box asking why guidance raises the cost.
Typed schematic, not captured. Bar lengths are placeholders.
15. How does Emu3's inference time for a 512x512 image compare to SDXL?Explain
A 512x512 placeholder image feeding two boxes, Emu3 with next-token generation and SDXL with diffusion, each followed by a dashed box asking for the time.
Typed schematic, not captured.
16. On image perception, how does Emu3 compare with LLaVA-1.6 in the lesson's benchmarks?Explain
A results sheet with rows VQAv2 and MMMU and columns Emu3 and LLaVA-1.6, every score cell shown as a dashed box with a question mark.
Typed schematic, not captured. Only the benchmark and model names are given.
17. What training compute does Emu3 need, and what is a familiar reference point?Explain
A box labelled Emu3 pretraining, multimodal next-token, with an arrow to a wide dashed box asking for the training compute and a familiar reference point.
Typed schematic, not captured.
18. Which token sequence trains Emu3 on visual question answering about an image?Apply
A card holding the question How many people are in the photo beside a placeholder photo labelled image, with an arrow down to a dashed box asking for the token sequence the model trains on.
Typed schematic, not captured. The photo and question are illustrative.
19. An Emu3-style sampler returns nearly identical images across seeds, with repeated textures. Which setting is the likely cause?Analyze
Four identical checkerboard-texture thumbnails labelled seed 1 to seed 4, with an arrow down to a dashed box asking which sampler setting is behind this.
Typed schematic, not captured. Identical tile patterns stand in for the repeated textures.
20. A post says Emu3 loses to SDXL because its GenEval score is 0.54 against 0.55. What is wrong with that reading?Analyze
A typed post reading Emu3 loses to SDXL: GenEval overall 0.54 vs 0.55, above a one-row table of the same scores and a dashed box asking what is wrong with that reading.
The post is typed for this picture; the scores are the stem's.
21. A team makes its Emu3-style transformer 10x larger, but images still lose fine texture, and images decoded straight from the tokenizer look soft too. Where is the bottleneck?Analyze
Two five-box rows. The top row runs caption, transformer 10x larger, image tokens, decoder, output with soft texture. The bottom row runs original, encoder, image tokens, decoder, round trip also soft. A dashed box below asks for the bottleneck.
Typed schematic, not captured.
22. To save time, a team skips the unconditional pass and samples from the conditional logits alone. Which gamma does that match, and what is lost?Analyze
A terminal with main.py's cfg_mix source and the two sampling lines, then a card for the team's change, sample from cond alone, with an arrow to a dashed box asking for the equivalent gamma.
Source lines copied from the lesson's main.py. The result is left blank.
23. A product team needs 1024x1024 images with a 2-second latency budget. Using the 8x8 reduction and 30 tok/s figures, which approach fits?Design
Two spec cards, 1024x1024 image with a latency budget of 2 seconds, and 8x8 reduction at 30 tokens per second, with an arrow down to a dashed box asking which approach fits.
Figures are the stem's. Typed schematic, not captured.
24. One product needs image generation, image question answering and short video generation from a single deployment. Which design follows the lesson's approach?Design
Three requirement cards, image generation, image Q and A, and short video generation, with arrows converging on a dashed box asking for a single deployment design.
Typed schematic, not captured.
25. You want to add audio to a unified model in the spirit of Emu3. What does the lesson's thesis suggest?Design
A box labelled Emu3-style model handling text, image and video, a separate unconnected box labelled audio, a new modality, and a dashed box asking how to add audio.
Typed schematic, not captured. No wiring is drawn on purpose.

Use It

code/main.py builds two toy pieces:

  • A token-count calculator for 2D image and 3D video tokenizers. Give it a resolution, a spatial reduction, a clip length, an FPS, and a temporal reduction, and it prints token counts plus seconds at 30 tok/s.
  • An autoregressive image-token sampler with classifier-free guidance and temperature.

The CFG step follows the Emu3 recipe: mix conditional and unconditional logits with a guidance weight. The sampler draws random logits over a K=16 codebook, so it shows how guidance reshapes a distribution and does not generate real images.

Check: Use It0/7
1. In the token table, latency at 30 tok/s climbs steeply as image resolution rises at a fixed 8x reduction. Why?Explain
Four squares drawn to scale for images of 256x256, 512x512, 1024x1024 and 2048x2048, each with a dashed latency box marked question mark underneath.
Typed schematic, not captured. Squares drawn to scale; latency is at 30 tok/s.
2. In main.py, TokCost('image 256x256', 256, 8).tokens() is divided by 30 for the latency column. What tokens and seconds are printed?Apply
A terminal listing the TokCost dataclass fields and tokens method from main.py, then the line c equals TokCost of image 256x256, 256, 8, above two dashed boxes for the token count and the seconds.
Source lines copied from the lesson's main.py. The result is left blank. The call line is typed from the stem.
3. In main.py, a TokCost has resolution=256, reduction=4, video_seconds=10, fps=8, time_reduction=4. What does tokens() return?Apply
A terminal listing the TokCost fields and tokens method from main.py, then the call with resolution 256, reduction 4, video_seconds 10, fps 8 and time_reduction 4, above a dashed box for the value tokens returns.
Source lines copied from the lesson's main.py. The result is left blank. The call line is typed from the stem.
4. Add a 2048x2048 image at reduction 8 to the table. How many tokens and how long at 30 tok/s?Apply
A terminal showing two config lines and the latency line from main.py's token table, with a comment for the new 2048x2048 row at reduction 8, above two dashed boxes for the token count and the time.
Two config lines and the latency line come from token_table() in main.py; the new row is typed from the stem. Results are left blank.
5. With cond = [2.0, 4.0, 1.0, 3.5, 0.5] and uncond = [1.0, 2.0, 1.5, 1.8, 1.2], what is cfg_mix at gamma=3.0 for index 3?Apply
A terminal with main.py's cfg_mix source above a table of conditional logits 2.0, 4.0, 1.0, 3.5, 0.5 and unconditional logits 1.0, 2.0, 1.5, 1.8, 1.2 over indices 0 to 4, plus a gamma 3.0 row whose index 3 cell is a dashed question mark.
Source lines copied from the lesson's main.py. The result is left blank. Only index 3 of the gamma 3.0 row is asked about.
6. With the same logits as demo_cfg, what does cfg_mix return when gamma=1.0?Apply
A terminal with main.py's cfg_mix source above a table of conditional and unconditional logits over indices 0 to 4, plus a gamma 1.0 row of five dashed question marks.
Source lines copied from the lesson's main.py. The result is left blank.
7. softmax(xs, temperature=0.8) is called on the same logits as softmax(xs, temperature=1.0). What happens to the largest probability?Apply
A terminal with main.py's softmax source above two dashed boxes for the largest probability at temperature 1.0 and at temperature 0.8.
Source lines copied from the lesson's main.py. The result is left blank. Logits are the same list in both calls.

Ship It

This lesson produces outputs/skill-token-gen-cost-analyzer.md. Give it a generation product spec (image or video, target resolution, quality tier, latency budget) and it computes token counts and inference cost, then picks between the Emu3 family and diffusion.

Author's Note

I'd run main.py before reading the paper's numbers. The first table prints token counts and seconds at 30 tok/s for a few configs, and the 2048x2048 row is the one that makes the cost of autoregressive image generation concrete: the count grows with the square of the resolution, and the latency column grows with it.

My bet is that the CFG function is the part worth rereading until it feels boring. cfg_mix is uncond + gamma * (cond - uncond), so gamma=1.0 gives back the conditional logits unchanged, and anything above 1.0 pushes the distribution further away from the unconditional one. The demo_cfg output shows the distribution sharpening as gamma goes up. That also tells you what the guidance weight costs: every sampled token needs two forward passes, one conditional and one unconditional.

I'd also keep the toy in proportion. Its conditional and unconditional logits are random Gaussians, so the sampler tells you nothing about image quality, only about how gamma and temperature reshape a distribution. The Emu3 numbers in the closing printout are hardcoded strings, not measurements from anything the script runs.

Exercises

  1. Emu3 produces 4096 tokens per 512x512 image at 8x8 reduction. Compute the equivalent for 1024x1024 and 2048x2048. What happens to inference latency?

  2. Read Emu3 Section 3.3 on the video tokenizer. Describe the 3D VQ patch shape and why it is 4x4x4 not 8x8x1.

  3. Classifier-free guidance weight 5.0 vs 3.0: what visual effect? Trace the math in code/main.py.

  4. Compute training FLOPs for Emu3-7B at 300B tokens and compare to Stable Diffusion 3. Which was more expensive to train?

  5. Emu3 beats SDXL on FID but only ties it on GenEval. Explain why the unified-loss approach shows different strengths vs specialists on different benchmarks.

Key Terms

TermWhat people sayWhat it actually means
Next-token prediction"NTP"Standard autoregressive loss: predict token[i+1] given token[0..i]; works for every modality when tokenized
IBQ tokenizer"Inverse bottleneck quantizer"A class of VQ-VAE with larger codebooks (32768+) and better reconstruction than Chameleon's
3D VQ"Spatiotemporal quantizer"Codebook indexed by (time, row, col); one token covers a 4x4x4 pixel cube
Classifier-free guidance"CFG"Mix conditional and unconditional logits with weight gamma; boosts image quality at inference
Unified vocabulary"Shared tokens"Text + image + video all draw from the same integer space; model predicts whichever modality comes next
MJHQ-30K"Image gen benchmark"Midjourney-quality benchmark with 30k prompts; Emu3 reports FID here

Further Reading

Code

Ships With