BuildPython

Chameleon and Early-Fusion Token-Only Multimodal Models

Most VLMs keep images and text apart. Visual tokens come from a vision encoder, pass through a projector, and only meet text inside the LLM. The two vocabularies never overlap. Chameleon (Meta, May 2024) asked what happens if they do. Train a VQ-VAE that turns an image into discrete tokens from a shared vocabulary, and a multimodal document becomes one sequence: text tokens and image tokens interleaved, one autoregressive loss. A side effect is that the model can emit text and image tokens alternately in a single inference call. This lesson reads the early-fusion thesis and builds a toy version end to end.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 05, Phase 8 (Generative AI)

Learning Objectives

Explain why a shared vocabulary + single loss changes what the model can do.
Describe how a VQ-VAE tokenizes an image into a discrete sequence compatible with a transformer's next-token objective.
Name Chameleon's training-stability tricks: QK-Norm, dropout placement, LayerNorm ordering.
Compare Chameleon vs BLIP-2's Q-Former approach and describe when each is the right choice.

The Problem

Adapter-based VLMs (LLaVA, BLIP-2, Qwen-VL) treat text and image as two different things. A text token goes through embed(text_token); an image goes through visual_encoder(image) → projector → ... pseudo_tokens. The model has two input paths that merge partway in.

Three consequences:

The LLM can only consume images, not emit them. Output is text only.
Mixed-modality documents (alternating paragraphs and images, as in an article) are awkward: you either parse the multimodal input outside the model or chain generations.
Distributional mismatch. Visual tokens and text tokens live in different regions of the hidden space, creating subtle alignment issues.

Chameleon rejects the premise: images are just sequences of discrete tokens from a shared vocabulary. Train the model on interleaved documents, one loss, one autoregressive decoder, and you unlock mixed-modality generation for free.

Warm-up0/3

1. In LLaVA, how do image features reach the language model?Warm-up

A photo on the left and a text prompt reading What is in this photo, both leading into one circle containing a question mark, which leads to a language model box on the right. — Typed schematic. The prompt text is an example.

2. In a VAE from Phase 8, what does the encoder produce that the decoder then uses to rebuild the image?Warm-up

A 7x7 toy image of a plus sign goes into an encoder, then a dashed box containing a question mark, then a decoder that outputs a rebuilt plus sign. — Typed schematic with a 7x7 toy image.

3. A language model's final layer turns a hidden vector into one score per vocabulary entry. What does softmax then produce?Warm-up

Four-word vocabulary and scores are made up for illustration.

Check: The Problem0/2

1. Why can an adapter VLM such as LLaVA not produce images as output?Explain

An adapter VLM drawn as a photo into a vision encoder, a projector, and a language model. A prompt reading Draw a cat also feeds the language model. Below the model is a dashed box reading output: question mark. — Typed schematic. The request text is an example.

2. A team attaches a vision encoder and projector to an LLM. It describes photos well but never returns a picture, even when asked to draw one. What is the cause?Analyze

A chat: the user sends a photo and asks what is in it; the model describes a mug on a desk. The user asks Draw a cat; the model replies with a text description of a cat. — Typed chat, illustrative.

The Concept

VQ-VAE as image tokenizer

The tokenizer is a vector-quantized variational autoencoder. The architecture:

Encoder: CNN + ViT that maps image to a spatial feature map, say 32x32 features of dim 256.
Codebook: a learned vocabulary of K vectors (Chameleon uses 8192), also dim 256.
Quantization: for each spatial feature, look up the nearest codebook entry by L2 distance. Replace the continuous feature with the integer index.
Decoder: CNN that takes quantized features back to pixels.

Training: VAE reconstruction loss + commitment loss + codebook loss. The codebook indices form a discrete alphabet for images.

For Chameleon: one image becomes 32*32 = 1024 tokens drawn from a vocabulary of 8192. Concatenate with text tokens (from the LLM's BPE vocabulary, say 32000). Final vocabulary: 40192. The transformer sees one sequence, one loss.

The shared vocabulary

Chameleon's vocabulary combines text tokens, image tokens, and modality separators. Each token has a single ID. The input embedding layer maps every ID to a D-dim hidden vector. The output projection maps hidden back to vocab logits. Softmax picks the next token, whatever modality.

Separators matter: <image> and </image> tags bracket the image-token sequence. At generation time, if the model emits <image>, downstream software knows the next 1024 tokens are VQ indices to send to the decoder for pixel rendering.

Mixed-modality generation

Inference is next-token prediction in the shared vocabulary. Example prompt: "Draw a cat and describe it." Chameleon emits:

<image> 4821 1029 2891 ... (1024 image tokens) </image>
The cat is orange, sitting on a windowsill...

The model picks the order on its own. It may produce image then text, text then image, or interleave them. Same decoder, same loss.

Compare to adapter VLMs where generation is text-only. Chameleon reopens the question of which modalities a model can output.

Training stability: QK-Norm, dropout, LayerNorm ordering

Early-fusion training is unstable at scale. Chameleon's paper documents three tricks:

QK-Norm. Apply LayerNorm to the query and key projections inside attention, before the dot product. This prevents logit magnitude explosion at depth, and multiple post-2024 large models use it.
Dropout placement. Dropout after every residual-add, not just after attention and MLP. More regularization required when gradients from image tokens can dominate.
LayerNorm ordering. Pre-LN on the residual branch (standard), plus an extra LN on the skip connection of the last block. Stabilizes final-layer gradient flow.

Without these tricks, 34B-param Chameleon training diverged at multiple checkpoints. With them, it converges. The training recipe is as much of the contribution as the architecture.

The tokenizer's reconstruction ceiling

VQ-VAE is lossy. At 8192 codebook entries and 1024 tokens per 512x512 image, reconstruction PSNR caps around 26-28 dB. This is enough for recognizable image gen but visibly worse than continuous-space diffusion (Stable Diffusion 3 achieves 32+ dB).

The tokenizer is the bottleneck. Better tokenizers (MAGVIT-v2, IBQ, SBER-MoVQGAN) lift the ceiling. Emu3 (Lesson 12.12) achieves SDXL-quality generation via a better tokenizer alone.

Chameleon vs BLIP-2 / LLaVA

Chameleon (early fusion, shared vocab):

One loss, one decoder.
Generates mixed-modality output.
Tokenizer is the quality ceiling.
Expensive: VQ-VAE decoder per generated image on inference path.

BLIP-2 / LLaVA (late fusion, separate towers):

Vision in, text out only.
Reuses pretrained LLM.
No tokenizer bottleneck for understanding.
Cheap: single forward pass.

Pick by task. If you need image generation, Chameleon family. If you only need understanding, adapter-VLM is simpler and reuses more pretrained compute.

Fuyu and AnyGPT

Fuyu (Adept, 2023) is a related approach: skip the separate vision encoder entirely, feed raw image patches through the LLM's input projection as if they were tokens, no tokenizer. Simpler than Chameleon, loses the shared-vocab output generation.

AnyGPT (Zhan et al., 2024) extends Chameleon to four modalities: text, image, speech, music. Same VQ-VAE trick for each, shared transformer. Any-to-any generation. Covered more in Lesson 12.16.

Check: The Concept0/23

1. What is early fusion in a multimodal model?Define

A photo and a short sentence, a cat on a mat, both lead to a circle with a question mark, which leads into a transformer box labelled layer 1 to layer L. — Typed schematic. The caption text is an example.

2. What is a VQ-VAE as Chameleon uses it?Define

A pipeline: an image goes into an encoder, giving a small grid of colored feature cells, then a box labelled codebook, then a decoder. — Typed schematic. Cell colors and sizes are illustrative.

3. What is a shared vocabulary in Chameleon?Define

A legend of text, image and separator colors above a row: the word cat, the image tag, two small colored 2x2 swatches for image patches, and the closing image tag. — Typed schematic. The patch swatches are illustrative.

4. What does QK-Norm do inside attention, and what does it prevent?Define

Diagram of one attention head: x passes through three projections W_q, W_k, W_v to give q, k and v; q and k go into a dot product box, then softmax, which then multiplies v to give the output. — Typed schematic of standard scaled dot-product attention (scale factor omitted).

5. What is mixed-modality generation?Define

A prompt reading Draw a cat and describe it goes into a model box. Below the model is a dashed output area with a question mark. — The prompt is the lesson's own example. The output box is left open on purpose.

6. What does the codebook size K of a VQ-VAE set?Define

A blue feature swatch on the left and a codebook of eight colored swatches numbered 0 to 7, labelled codebook, toy with eight entries. — Typed schematic. The colors are illustrative stand-ins for vectors.

7. What is the tokenizer ceiling?Define

A row: Image, Encode, tokens, Decode, Image. A dashed bracket spans Encode to Decode labelled tokenizer ceiling, with a question mark below it. — Typed schematic of the VQ-VAE round trip. The dashed span marks the term from the lesson.

8. How many tokens does Chameleon's VQ-VAE produce for one image, and from what size codebook?Define

A 512 by 512 image goes into a box labelled VQ-VAE with encoder and codebook. Two dashed boxes with question marks are labelled image tokens and codebook entries. — Typed schematic. The image size is the lesson's example.

9. Which three losses train Chameleon's VQ-VAE tokenizer?Define

A pipeline: image, encoder, z_e, codebook, z_q, decoder, rebuilt image. A dashed box reads training loss: question mark. — Typed schematic. z_e is the encoder output and z_q the chosen codebook entry.

10. Why do the <image> and </image> separators matter at generation time?Explain

Typed from the lesson's example. The ids are illustrative.

11. Why does a single loss over the shared vocabulary let the model choose the order of text and image output?Explain

The prompt Draw a cat and describe it, then three rows of output blocks: image then text; text then image; and text, image, text, image. — Typed from the lesson's list of orders (image then text, text then image, interleaved).

12. What does Chameleon's 'dropout placement' trick change?Explain

One transformer block: input x goes to an attention box and a plus node, then an MLP box and a second plus node, to the output. Residual lines skip over attention and over the MLP. — Typed schematic. Normalization layers are left out here.

13. What does Chameleon's LayerNorm-ordering trick add on top of standard Pre-LN?Explain

Three identical boxes labelled block 1, block 2 and block 3. Each has a skip line across the top and, below it, LN then a layer then a plus node. — Typed schematic. 'layer' stands for attention or MLP.

14. What reconstruction quality ceiling does the tokenizer impose, compared to continuous-space diffusion?Explain

Two rows. Top: image, discrete VQ tokens, decode, PSNR question mark. Bottom: image, continuous latents, decode, PSNR question mark. — Typed schematic. Top: Chameleon-style tokenizer. Bottom: Stable Diffusion 3 style.

15. Why is an adapter VLM like LLaVA cheaper at inference than Chameleon?Explain

Two side-by-side cards: Adapter VLM (LLaVA family) and Chameleon family (early fusion). Under each is a dashed box reading inference cost: question mark. — Typed schematic.

16. How does Fuyu's approach differ from Chameleon's?Explain

Two rows. Chameleon: photo, VQ-VAE tokenizer, shared-vocab ids, transformer. Fuyu: photo, a dashed box with a question mark, then an LLM. — Typed schematic.

17. How does AnyGPT extend Chameleon's early-fusion approach?Explain

Left: Chameleon, with text and image inputs feeding a shared transformer. Right: AnyGPT, a dashed box reading modalities: question mark. — Typed schematic.

18. A prompt has 200 text tokens followed by one image encoded as a 32x32 grid, wrapped in <image> and </image>. How many positions does the sequence use?Apply

A row of four boxes: 200 text tokens, the image tag, a 32 x 32 image grid, and the closing image tag. Below is the line positions used by the whole sequence, followed by a question mark. — Typed schematic. Box widths are not scaled to token counts.

19. Training a 34B early-fusion model, the loss diverges at several checkpoints and attention logit magnitudes grow with depth. Which change targets this directly?Analyze

Two schematic plots. Left: loss against training step, falling then jumping up sharply twice. Right: attention logit size against layer depth, rising steeply. — Typed schematic. Shapes only; no measured values.

20. A Chameleon-style model's images are recognizable but consistently softer than Stable Diffusion 3 outputs, and scaling the transformer barely helps. Where is the likely limit?Analyze

A plot of image sharpness against transformer size. An orange curve for a Chameleon-style model is nearly flat and low; a dashed blue line for Stable Diffusion 3 runs well above it. — Typed schematic of the stem. No measured values.

21. A product only has to answer questions about uploaded screenshots, must launch cheaply, and wants to reuse a strong pretrained LLM. Which design fits?Design

Typed from the question stem.

22. A team wants one model that writes a recipe and generates a photo for each step in a single call, and can afford a VQ-VAE decode per image. Which family fits, and what caps image quality?Design

Typed from the question stem.

23. You want to add speech output to a text-and-image early-fusion model, following the lesson's approach. What do you do?Design

Typed schematic. Token ids are left out.

Use It

code/main.py builds a toy end-to-end early-fusion model:

A tiny VQ-VAE-style quantizer that maps an 8x8 grayscale image to 4 codebook indices (K=16).
A shared vocabulary of (text ids 0..31) + (image ids 32..47) + (separators 48, 49).
A toy autoregressive decoder (bigram table) trained on synthetic captions + image-token sequences.
Sampling loop that emits alternating text + image tokens given a prompt.

The code intentionally keeps the transformer tiny (bigrams) so you can trace the signal flow end to end.

Check: Use It0/5

1. In main.py, CODEBOOK[i] = [(i*7 + 3*j) % 8 for j in range(4)], and quantize_patch scans entries in order, replacing the best only when the distance is strictly smaller. What does quantize_patch([6, 1, 4, 7]) return?Apply

Code excerpt from main.py defining the codebook formula, an image offset of VOCAB_TEXT, and quantize_patch, which keeps the best entry only when d is strictly smaller. Below it is the call quantize_patch([6, 1, 4, 7]) followed by a question mark. — Lines copied from main.py; the docstring and type hints are omitted and the CODEBOOK line is wrapped for width.

2. With VOCAB_TEXT = 32 and VOCAB_IMG = 16, what does render([40, 48, 5, 49]) return?Apply

Code excerpt from main.py: constants VOCAB_TEXT, VOCAB_IMG, IMG_OFFSET, SEP_OPEN and SEP_CLOSE, then the loop in render() with its if, elif and else branches. Below it is the call render([40, 48, 5, 49]) followed by a question mark. — Lines copied from main.py; the function header and the join are omitted.

3. In make_dataset, a caption-first sequence for 'red' is synthesize_caption('red') + [SEP_OPEN] + image_to_tokens(...) + [SEP_CLOSE]. How many tokens is it?Apply

The expression synthesize_caption('red') + [SEP_OPEN] + image_to_tokens(img) + [SEP_CLOSE] with len(tokens) = question mark. Below it, the 8x8 red image as a grid of numbers 7, 8 and 9, and code excerpts of synthesize_caption and image_to_tokens. — Lines excerpted from main.py (docstrings, type hints and inner loops omitted).

4. generate(bigram, [1, 5], max_len=30) breaks out of its loop early when `nxt == SEP_CLOSE and any(t < VOCAB_TEXT for t in out)`. When does that happen?Apply

Six lines from generate() in main.py (the sampling line shown as dots) with the if line highlighted: it breaks when nxt equals SEP_CLOSE and any token in out is below VOCAB_TEXT. Below are the call generate(bigram, [1, 5], max_len=30) and VOCAB_TEXT = 32. — Lines copied from main.py; the sampling line is elided.

5. In the toy's sampled output, captions are followed by plausible image tokens, yet the image never matches the caption. Which property of the toy explains this?Analyze

Output of generate(bigram, [1, 5], max_len=30) after seed 42 training, as render() prints it.

Ship It

This lesson produces outputs/skill-tokenizer-vs-adapter-picker.md. Given a product spec (understand only vs understand + generate, required image quality, cost budget), it picks between Chameleon-family (early fusion) and LLaVA-family (late fusion) and justifies with quantitative rules of thumb.

Author's Note

I'd run main.py and read the second block of output before anything else. It prints the vocabulary layout: text ids 0 to 31, image ids 32 to 47, <image> as 48 and </image> as 49, 50 ids in total. That printout is the early-fusion idea in five lines. The decoder never asks which modality a token belongs to, because the id already says.

Be careful about what the toy proves. The codebook is a fixed arithmetic pattern, not a trained one, and the quantizer is just a nearest-entry lookup by L2 distance. So the tokenizer ceiling from the concept section is a claim you have to take from the text here, since this code has no reconstruction step to measure. The same goes for QK-Norm and the other stability tricks. A bigram count table has no attention to normalize.

What I'd look at next is the sampled output in the fourth block. The generation loop only conditions on the previous token, so it can produce plausible-looking mixed output without ever matching an image to its caption. In this toy it couldn't match them anyway: the red, green and gray images quantize to the same codes. Watching that fail on a table of counts makes it clearer why Chameleon needs a real transformer, and why the tokenizer and the training recipe matter as much as the shared vocabulary. The fifth exercise is the natural follow-up once you can see the loop's limits.

Exercises

Chameleon uses K=8192 codebook entries and 1024 tokens per 512x512 image. Estimate the compression ratio vs a 24-bit RGB image. Is it lossy? How lossy?
A 4K image (3840x2160) at the same VQ-VAE density produces how many image tokens? Can a Chameleon-style model generate a 4K image in one inference call? What breaks first: context, tokenizer quality, or KV cache?
Implement QK-Norm in pure Python. Given a 64-dim query and key, show the dot product before and after LayerNorm. Why is magnitude control important at depth?
Read Chameleon Section 2.3 on training stability. Describe the exact failure mode the paper observed at 34B without QK-Norm. What was the "norm explosion" signature?
Extend the toy decoder to emit a mixed-modality response given a text-only prompt. Measure how often the model picks image-first vs text-first given training-data distribution 60% text-first / 40% image-first.

Key Terms

Term	What people say	What it actually means
Early fusion	"Unified tokens"	Images converted to discrete tokens sharing the transformer's vocabulary from step one
VQ-VAE	"Image tokenizer"	CNN + ViT + codebook that maps images to integer indices the transformer can predict
Shared vocabulary	"One dictionary"	A single token ID space covering text + image + modality separators
QK-Norm	"Attention stabilizer"	LayerNorm applied to query and key before their dot product, prevents norm blowup
Mixed-modality generation	"Text + image output"	Inference that autonomously produces interleaved text and image tokens in one pass
Codebook size	"K entries"	Number of discrete vectors the VQ-VAE can quantize to; trades compression for fidelity
Tokenizer ceiling	"Reconstruction limit"	Best PSNR achievable by decoding VQ tokens; bounds the model's image quality