BuildPython

Visual Autoregressive Modeling (VAR): Next-Scale Prediction

Diffusion models sample iteratively in time (denoising steps). VAR samples iteratively in scale: it predicts a 1x1 token, then 2x2, then 4x4, up to the final resolution, each scale conditioning on the previous. The 2024 paper reports that VAR follows GPT-style scaling laws for image generation and beats DiT at the same compute budget. This lesson builds the core mechanism.

TIME: ~90 minutesPREREQUISITES: Phase 7 Lesson 03 (Multi-Head Attention), Phase 8 Lesson 06 (DDPM)

Learning Objectives

Explain the generation-order problem in raster-order autoregressive image models and how next-scale prediction avoids it.
Describe how a multi-scale residual VQ tokenizer encodes an image as K token grids and how the decoder sums them back.
Trace VAR generation: K passes, causal across scales, parallel within a scale, and compare the pass count to diffusion.
Run the toy `main.py` and read its reconstruction MSE and per-scale conditioning counts.

The Problem

Autoregressive generation dominated language modeling because it scales predictably: more compute, more parameters, lower perplexity, better outputs. Image generation had two main AR attempts before 2024: PixelRNN/PixelCNN (pixel-by-pixel) and DALL-E 1 / Parti / MuseGAN (token-by-token on VQ-VAE codes).

Both suffered from a generation-order problem. Pixels and tokens are arranged in a 2D grid, but the AR model has to visit them in a 1D raster order. An early corner pixel has no idea what the image eventually becomes. Generation quality scaled worse than GPT-on-text and did not reach diffusion-model quality at matched compute.

VAR fixes the generation-order problem by changing what is being generated. Instead of predicting image tokens one by one in space, VAR predicts a whole image at increasing resolutions. Step 1: predict a 1x1 token (the overall image "summary"). Step 2: predict a 2x2 grid of tokens (coarser features). Step 3: predict a 4x4 grid. Step K: predict the final (H/8)x(W/8) grid.

Each scale attends to all previous scales (causally in "scale order") and parallel within its own scale. The order problem disappears: the whole image at scale k is produced in one transformer pass.

Warm-up0/3

1. In a decoder attention layer with a causal mask, what does the mask do to the attention scores?Warm-up

A 5 by 5 grid of attention scores for the sentence The cat sat on mat. Rows are query positions 1 to 5, columns are key positions 1 to 5, and each cell is labelled s with its row and column numbers. — Layout only: every cell holds one raw score s(row, column). The sentence is an illustrative example.

2. A DDPM sampler produces an image by running its denoiser many times. How are those runs organized?Warm-up

A row of boxes: pure noise on the left, four boxes labelled run 1, run 2, run 3 and run T in the middle, and a finished image on the right. Dashed links between them carry question marks. — Schematic. The question marks stand for how consecutive runs are connected.

3. How does a GPT-style language model generate a passage of text?Warm-up

A prompt made of three word chips, Once upon a, above a row of six empty dashed slots marked with question marks, which are the tokens of the passage to be generated. — Illustrative prompt, typed by hand. The slots are the tokens the model has to produce.

Check: The Problem0/1

1. Why did raster-order autoregressive image models such as PixelCNN scale worse than GPT does on text?Explain

Schematic of raster order on an 8x8 grid.

The Concept

VQ-VAE Multi-Scale Tokenizer

VAR needs a multi-scale discrete tokenizer. For an image x, it produces a sequence of progressively higher-resolution token grids:

x -> encoder -> latent f
f -> tokenize at 1x1: token grid z_1 of shape (1, 1)
f -> tokenize at 2x2: token grid z_2 of shape (2, 2)
...
f -> tokenize at (H/p)x(W/p): token grid z_K of shape (H/p, W/p)

Each z_k uses the same codebook (typical size 4096-16384). The tokenization at each scale is not independent. It is trained so that summing the residuals at each scale reconstructs f:

f ≈ upsample(embed(z_1), target_size) + ... + upsample(embed(z_K), target_size)

This is a residual VQ variant. Scale k captures what scales 1..k-1 missed. Decoder takes the sum of all scale embeddings and produces the image.

The multi-scale VQ tokenizer is trained once (like VQGAN) and then frozen. All the generative work is done by the autoregressive model on top.

Next-Scale Prediction

The generative model is a transformer that sees tokens from all previous scales and predicts the tokens at the next scale.

Input sequence structure:

[START, z_1 tokens, z_2 tokens, z_3 tokens, ..., z_K tokens]

Position embeddings encode both scale index and spatial position within the scale. Attention is causal in scale order: token at scale k, position (i, j) can attend to all tokens at scales 1..k and to tokens at scale k itself that come earlier in whatever intra-scale order is used (VAR uses fixed positional attention with no intra-scale causality, so all positions within a scale are predicted in parallel).

Training loss: at each scale k, predict the tokens z_k given all prior-scale tokens. Cross-entropy loss on the discrete VQ codes. Same structure as GPT except the "sequence" is now scale-structured.

Generation

At inference:

generate z_1 = sample from p(z_1)                    # 1 token
generate z_2 = sample from p(z_2 | z_1)              # 4 tokens in parallel
generate z_3 = sample from p(z_3 | z_1, z_2)         # 16 tokens in parallel
...
decode: f = sum of embed-and-upsample scales 1..K
image = VAE_decoder(f)

For K = 10 scales, generation is 10 transformer forward passes. Each pass produces its entire scale in parallel, with no per-token autoregression within a scale. For a 256x256 image this is roughly 10 passes vs DiT's 28-50.

Why Next-Scale Wins Over Next-Token

Three structural wins:

Coarse-to-fine aligns with natural image statistics. Human visual perception and image datasets both exhibit scale-dependent regularities: low-frequency structure is stable and predictable; high-frequency detail is conditional on low-frequency content. Next-scale prediction exploits this.
Parallel generation within scale. Unlike GPT-style token AR, VAR produces all tokens at a scale in one step. Effective generation length is log-scale instead of linear.
No generation order bias. Tokens at scale k see all of scale k-1; there is no "left-of" or "above" bias that forces early tokens to commit before late context is available.

Scaling Law

Tian et al. demonstrated that VAR follows a power-law scaling curve for FID on ImageNet, the same way GPT does for perplexity. The paper presents this as the first image model to scale that cleanly. The practical payoff is that you can predict how a bigger VAR will do from compute, instead of guessing per architecture.

Relationship to Diffusion

VAR and diffusion share the same data-compression story: both break the generation problem into a sequence of easier subproblems.

Diffusion: gradually add noise, learn to undo one step.
VAR: gradually add resolution, learn to predict the next scale.

They are different axes through the problem. Both yield tractable conditional distributions. Tian et al. report that VAR is faster at inference (fewer passes, all parallel within a scale) and beats DiT on class-conditional ImageNet. Text-conditional VAR (VARclip, HART) is an active research direction.

Check: The Concept0/23

1. What is VAR (Visual AutoRegressive modeling)?Define

Real output of main.py's generate() (seed 0, first trial). Cell tint is the codebook value. The predictor is a histogram, not a transformer.

2. What does next-scale prediction mean?Define

Four 8x8 grayscale images of the ring pattern, each decoded from the first 1, 2, 3 and 4 scales of the toy tokenizer, getting sharper left to right, with their reconstruction errors printed below, and the original ring beside the label target. — Real values from main.py's tokenizer on the ring pattern (training image 2). Reconstruction errors are computed in the script.

3. What does a multi-scale VQ tokenizer output for an image?Define

A flow from an 8x8 ring pattern image, through a box labelled multi-scale VQ tokenizer, to a dashed box marked output with a question mark, then a decoder box that produces the reconstructed image. — Schematic. The ring is a pattern from main.py's library; the tokenizer output is what the question asks about.

4. In VAR, what is 'scale k'?Define

Four token grids named z_1, z_2, z_3 and z_4 with shapes (1, 1), (2, 2), (4, 4) and (8, 8), taken from the toy tokenizer run on a ring pattern, with the note that K is 4 and k runs over 1 to 4. — Real tokens from main.py's tokenize_multiscale on the ring pattern. Cell tint is the codebook value; numbers are token indices.

5. What does 'parallel-within-scale' mean in VAR?Define

Scales 1 and 2 are the real tokens of the ring pattern from main.py's tokenizer. The 4x4 grid is the unknown.

6. What does causal-across-scales attention allow?Define

A left-to-right strip of five blocks labelled START, z_1 (1 token), z_2 (4 tokens), z_3 (16 tokens) and z_4 (64 tokens), with a marker above the z_3 block for one query token at scale 3 and the question of which blocks it may attend to. — Schematic of the toy's scale sizes. START is one slot ahead of the first scale, as in the lesson's input sequence.

7. In VAR's multi-scale VQ tokenizer, what do the tokens at scale k encode?Define

The 8x8 ring pattern on the left and, to its right, the 4x4 grid of token indices z_3 that the toy tokenizer produces at scale 3, with a note that what was quantized to get them is the question. — Real tokens from main.py's tokenize_multiscale on the ring pattern.

8. What did Tian et al. show about how VAR scales?Define

Empty plot axes: training compute increasing to the right on the horizontal axis, and FID on ImageNet, lower is better, on the vertical axis, with a note that the measured values are in the Tian et al. paper and not drawn here. — Axes only. No data is plotted because the lesson gives no numbers for the curve.

9. What is HART?Define

A list of five papers from the lesson's Further Reading with year and authors: VQ-VAE 2017, VQGAN 2021, DiT 2022, VAR 2024 and HART 2024, with the HART row highlighted and its full paper title printed below. — Years, names and the HART title are copied from the lesson's Further Reading list.

10. What does a scale position embedding encode?Define

A strip of 22 small token cells: one START cell, one z_1 cell, four z_2 cells and sixteen z_3 cells, with the eleventh cell, inside z_3, highlighted as this token, and a note that a position embedding is added to every token. — Schematic with the toy's first three scale sizes. Which cell is highlighted is arbitrary.

11. Which three structural wins does the Concept section give for next-scale prediction over next-token prediction?Explain

Schematic. Step counts are computed from the grid sizes: 4 x 4 = 16 tokens versus scales 1, 2 and 4.

12. Why does the decoder sum the upsampled scale embeddings instead of reading only the finest scale?Explain

Four 8x8 heatmaps, one per scale of the toy tokenizer on a ring pattern, each showing that scale's upsampled embedding with its minimum and maximum value, joined by plus signs, followed by decoder output equals a question mark and a small copy of the input image. — Real values from main.py: each scale's embedding, upsampled to 8x8. Orange is positive, blue negative.

13. What is the division of labor between the tokenizer and the transformer in VAR?Explain

A flow: image, multi-scale VQ tokenizer, token grids z_1 to z_K, transformer. Under the tokenizer the note weights during the transformer's training with a question mark, and under the transformer the note what it predicts with a question mark. — Schematic. The question marks are what the question asks about.

14. How does attention work across scales in the VAR transformer?Explain

A 3 by 3 table of blocks: rows are query tokens at scale 1, 2 and 3 with 1, 4 and 16 tokens, columns are key tokens at the same three scales, and every block holds a question mark. — Schematic. Each question mark asks whether those queries may attend to those keys.

15. For a 256x256 image with K = 10 scales, how many forward passes does VAR need compared with DiT?Explain

Two boxes: VAR with K equal to 10 scales, and DiT, the diffusion transformer, with 28 to 50 denoising steps, both for one 256x256 image, each with a note forward passes with a question mark. — The numbers are the ones in the lesson's Generation section.

16. How does VAR's route to image generation relate to diffusion's?Explain

Two rows of boxes joined by arrows. The first row, Diffusion, runs pure noise, step 1, step 2, dots, image. The second row, VAR, runs START, scale 1, scale 2, dots, image. — Schematic of the two step sequences, with no claim yet about what changes at each step.

17. Your 10-scale tokenizer reconstructs images poorly, but a predictor trains easily on its tokens. Which change addresses this?Analyze

A pipeline of image, a 10-scale residual VQ tokenizer marked as reconstructing images poorly, tokens, and a predictor marked as training easily on these tokens, with the question of which part to change. — Schematic of the situation in the question; no measurements are shown.

18. A variant predicts the tokens inside each scale one at a time, left to right. Which VAR benefit does it lose?Analyze

Schematic of the variant in the question, drawn on the toy's scale sizes.

19. A bug lets tokens at scale 2 attend to scale 3 during training. What goes wrong?Analyze

A 4 by 4 table of attention between scale blocks. Rows are queries at scales 1 to 4 and columns are keys at scales 1 to 4. Cells on and below the diagonal say yes, those above say no, except the cell for scale 2 queries and scale 3 keys, which is outlined and marked bug. — Schematic of the lesson's causal-across-scales mask with the one wrong cell from the question.

20. A teammate trains each scale to encode the full latent f independently, then sums the scale embeddings in the decoder. What goes wrong?Analyze

Four 8x8 gray images, each a pooled and upsampled copy of the whole ring pattern at scales 1 to 4, joined by plus signs, followed by decoder sums the four scale embeddings, output equals a question mark, and a small copy of the input image. — Computed in the script: average-pooled and upsampled copies of the ring pattern. Quantization is left out to keep the picture simple.

21. You must serve 512x512 class-conditional images under a tight latency budget, and a DiT baseline needs 50 denoising steps. Which VAR property helps most?Design

A request box for 512x512 class-conditional images under a tight latency budget, a DiT baseline box with 50 denoising steps, and a VAR box with passes marked by a question mark. — Schematic. The 512x512 size, the class conditioning and the 50 steps are the ones in the question.

22. A team must cut generation from 10 passes to about 6 for a 256x256 image, with the same VAR design. Which tokenizer choice fits?Design

Two horizontal bars on one scale, today at 10 passes per image filled in orange, and a target outlined at about 6 passes, for a 256x256 image with the same VAR design. — Schematic. Bar lengths are proportional to the 10 and about 6 passes in the question.

23. Before a large training run, how can a team estimate how a bigger VAR would score on ImageNet?Design

Three boxes for small VAR runs, each with FID measured, beside a highlighted box for the planned large VAR run whose FID is a question mark, with a note that the dataset is ImageNet. — Schematic. The number of small runs shown is arbitrary.

Build It

In main.py you will:

Build a tiny multi-scale residual VQ tokenizer on 8x8 grayscale "images" drawn from a five-pattern library (solid, gradient, ring, checker, cross).
Fit a stand-in for the transformer: one conditional histogram per scale, keyed on a summary of the previous scales' tokens.
Sample by running 4 passes (scales 1, 2, 4, 8), drawing every position at a scale in parallel, then decode by summing the upsampled scale embeddings.
Print reconstruction MSE on train and validation patterns, plus a check of how many prior tokens each scale conditions on.

This is a toy, and the predictor is not a transformer. What it does show is the scale-ordered conditioning and the parallel-within-scale sampling, which are the parts that stay the same when you swap in a real model.

Check: Build It0/8

1. In `main.py`, what does 'parallel within a scale' amount to?Explain

A code excerpt from main.py's generate function showing, for each scale, the context key, the table lookup for the probabilities, and the line that builds the flat array of sampled tokens. — Exact lines copied from main.py.

2. In `main.py` the scales are (1, 2, 4, 8). How many tokens does one generated sample contain across all scales?Apply

Grids are drawn at their true cell counts; cell sizes shrink so they fit.

3. In the scale-ordered attention check printed by `main.py`, how many prior tokens does the 8x8 scale attend to?Apply

The loop from main.py that prints the scale-ordered attention check, above the first three lines it prints, for scales 0, 1 and 2, which attend to 0, 1 and 5 prior tokens, and a last line for the 8x8 scale left as dots. — Code copied from main.py. The three printed lines are real output; the last line is withheld.

4. A learner sets CODEBOOK = 128 in `main.py` and runs it on the 64 training images. What happens?Apply

Two code excerpts from main.py: CODEBOOK set to 128 with the start of fit_codebook and its check on the number of samples, and the loop in train_codebooks that pools the residuals at each scale and fits a codebook of that size. — Exact lines copied from main.py, with CODEBOOK edited to 128 as in the stem.

5. In `main.py`, what does `context_key` use to summarize each earlier scale for the predictor?Apply

Code copied from main.py. The grids are the real toy tokens of the ring pattern.

6. In `tokenize_multiscale`, a token from the 2x2 scale is upsampled to 8x8 before it is subtracted. How many pixels does each of its tokens cover?Apply

Code copied from main.py. The 2x2 tokens are the real scale-2 tokens of the ring pattern.

7. In `main.py`, the 4x4 scale works on an 8x8 image. What does its tokenizer actually quantize?Apply

Real tokens of the ring pattern from main.py's tokenize_multiscale for scales 1 and 2.

8. The toy's generated images show no spatial structure inside a scale. Which cause explains this?Analyze

Three decoded 8x8 samples from the toy VAR that look like noise, and below them the five pattern types in the training library: solid, gradient, ring, checker and cross. — Real samples from main.py's generate (seed 0, three trials). Library patterns are redrawn from the formulas in make_patterns.

Ship It

This lesson produces outputs/skill-var-tokenizer-designer.md, a skill for designing a multi-scale tokenizer. It covers the number of scales, scale ratios, codebook size, residual sharing, and decoder architecture.

Author's Note

I'd run main.py first, because the whole thing finishes in moments and prints reconstruction MSE for the train and validation patterns. The images are 8x8 and drawn from five patterns, the scales are 1, 2, 4, 8, and the codebook has 16 entries. That is small enough that you can hold the full token pyramid in your head.

Two things differ from the Concept section. The toy fits a separate codebook per scale, with k-means on the residual left by the scales before it, instead of sharing one codebook. And the predictor is not a transformer. It is a conditional histogram per scale, keyed on the mean token index of each earlier scale. So "parallel within a scale" here means every position at a scale is drawn independently from the same distribution. Real VAR gets spatial structure within a scale from the transformer. The toy only demonstrates the ordering and the parallel draw.

My bet is that the residual loop in tokenize_multiscale is the part worth reading twice. Each scale average-pools what is left, snaps it to the codebook, upsamples the result, and subtracts it. detokenize_multiscale just adds the upsampled embeddings back up. Once that reads as obvious, the rest of VAR is a sequence model over those grids.

If I were taking this past the toy, I'd start with the codebook size exercise, but with smaller sizes. The scale-1 codebook is fit on only 64 pooled values, so 512 and up raise a ValueError in fit_codebook. Try 4, 8, 16, 32 and 64 instead. In this setup it is the cheapest knob to turn, and it shows the reconstruction versus prediction tradeoff directly.

Exercises

Scale count ablation. Train VAR with 4, 6, 8, 10 scales. Measure reconstruction quality vs number of autoregressive passes. More scales = finer residuals = better quality but more passes.
Codebook size. Train tokenizers with codebook sizes 512, 4096, 16384. Larger codebooks give better reconstruction but harder prediction. Find the knee.
Parallel-within-scale check. For a trained VAR, measure the attention pattern explicitly. Within scale k, does the model attend to cross-scale positions but not intra-scale? Verify the mask implementation.
VAR vs DiT scaling. For the same ImageNet class-conditional task, train VAR and DiT at matched param budgets (e.g., 33M, 130M, 458M). Plot FID vs compute. The paper reports VAR ahead of DiT; see whether you can reproduce that at small scale.
Text conditioning. Extend VAR to take a text embedding (CLIP pooled) as an extra conditioning input via adaLN. This is the HART recipe. How much does FID improve on text-aligned sampling?

Key Terms

Term	What people say	What it actually means
VAR	"Visual AutoRegressive"	Image generation by next-scale prediction over a pyramid of VQ token grids
Next-scale prediction	"Predict coarser, then finer"	The model predicts tokens at increasing resolution scales, conditioning on all previous scales
Multi-scale VQ tokenizer	"Residual VQ"	VQ-VAE that produces K token grids of increasing resolution, with decoder summing all scales
Scale k	"Pyramid level k"	One of K resolution levels, from 1x1 at k=1 up to (H/p)x(W/p) at k=K
Parallel-within-scale	"One forward per scale"	All tokens at scale k are predicted in one transformer pass, not autoregressively
Causal-across-scales	"Scale-ordered attention"	Token at scale k can attend to all of scales 1..k but not scales k+1..K
Residual VQ	"Additive tokenization"	Each scale's tokens encode the residual left by lower scales; decoder sums all scale embeddings
VAR scaling law	"Image GPT scaling"	FID follows a predictable power law in compute, like language models' perplexity
HART	"Hybrid VAR + text"	Text-conditional VAR variant combining MaskGIT-style iterative decoding with VAR's scale structure
Scale position embedding	"(scale, row, col) triple"	Positional encoding carries both the scale index and spatial coordinates within the scale