TAI BUI
Back to curriculum
BuildPython

LLaVA and Visual Instruction Tuning

LLaVA (April 2023) became the template most open vision-language models copied. It replaced BLIP-2's Q-Former with a 2-layer MLP, replaced Flamingo's gated cross-attention with plain token concatenation, and trained on 158k visual-instruction turns that GPT-4 wrote from text-only captions. LLaVA-1.5 added academic-task data, LLaVA-NeXT added AnyRes, and LLaVA-OneVision folded image, multi-image, and video into one recipe. This lesson reads the recipe, builds the projector, and looks at why the simple design won.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 02 (CLIP), Phase 11 (LLM Engineering — instruction tuning)

Learning Objectives

  • Build a 2-layer MLP projector that maps ViT patch embeddings (dim 1024) to an LLM's embedding dim (dim 4096).
  • Walk the LLaVA two-stage recipe: (1) projector alignment on 558k caption pairs, (2) visual instruction tuning on 158k GPT-4-generated turns.
  • Construct a LLaVA-format prompt with the image token placeholder, system prompt, and user/assistant turns.
  • Explain why the community moved from Q-Former to MLP despite Q-Former's token-budget win.

The Problem

BLIP-2's Q-Former (Lesson 12.03) compresses an image to 32 tokens. It is clean, efficient, and good on benchmarks. It also has two problems.

First, the Q-Former is trainable but its loss is not the final task. Stage 1 trains ITC+ITM+ITG and stage 2 trains LM loss, so the queries learn an intermediate representation that the LLM then has to decode. Information gets lost in the bottleneck.

Second, the Q-Former has 188M params and has to be co-designed with the target LLM. Change the LLM and you retrain the Q-Former. Change the vision encoder and you retrain it again. Every combination becomes its own R&D project.

LLaVA's answer was almost embarrassingly simple. Take the ViT's 576 patch tokens, pass each through a 2-layer MLP (1024 → 4096 → 4096), and put all 576 into the LLM's input sequence. There is no bottleneck and no stage-1 pretraining on odd objectives. You train the MLP on a direct LM loss.

The data came from a second insight: use text-only GPT-4 to generate the instructions. Feed it the COCO captions and bounding boxes for an image, and ask for conversations, descriptions, and complex reasoning questions. That gives 158k instruction-response turns with no human annotation.

The result was a VLM that trained in about a day on one 8xA100 node and shipped as an open checkpoint the community could extend. Many forks followed.

Warm-up0/3
1. What does CLIP's contrastive training teach the vision encoder?Warm-up
A batch of three image and caption pairs (a sun, a house, a tree with short captions). The images go into an image encoder, the captions into a text encoder, and both feed a contrastive loss box.
Hand-drawn pairs, illustrative.
2. What does instruction tuning change about a pretrained LLM?Warm-up
A text input, Translate cat to French, goes into a box labelled pretrained LLM, and a dashed box with a question mark shows the output.
The prompt is a typed example, not from the lesson.
3. Why would you freeze a pretrained network's weights while training a new layer attached to it?Warm-up
A chain of boxes: input, pretrained layers, a highlighted new layer, and loss. A dashed line shows gradients flowing back from the loss toward the input.
Schematic, no weights shown.
Check: The Problem0/2
1. Which two problems with BLIP-2's Q-Former motivated LLaVA's simpler design?Explain
The BLIP-2 pipeline: image, ViT, a highlighted Q-Former, and the LLM, with a note that LLaVA's simpler design replaced the Q-Former.
Schematic from the stem.
2. Why does LLaVA not need a stage-1 objective such as ITC+ITM+ITG the way BLIP-2 does?Explain
Two pipelines. BLIP-2: image, ViT, Q-Former, LLM, with stage 1 objective ITC+ITM+ITG. LLaVA: image, ViT, MLP projector, LLM, with the note stage 1 objective none of these.
Schematic; token counts are left out on purpose.

The Concept

The architecture

LLaVA-1.5 at 7B:

  • Vision encoder: CLIP ViT-L/14 @ 336 (frozen during stage 1, optionally unfrozen stage 2).
  • Projector: 2-layer MLP with GELU activation, 1024 → 4096 → 4096.
  • LLM: Vicuna-7B (later Llama-3.1-8B).

Forward pass on an image + text prompt:

img -> ViT -> 576 patches of dim 1024
patches -> MLP -> 576 tokens of dim 4096
prompt: system + "<image>" placeholder + user question
replace <image> token with the 576 projected tokens
feed the full sequence to the LLM
decode response

The image occupies 576 tokens of the LLM context. At 2048 context, that leaves 1472 tokens for text. At 32k context, it is a rounding error.

Stage 1: projector alignment

Freeze ViT. Freeze LLM. Train only the 2-layer MLP. Dataset: 558k image-caption pairs (LAION-CC-SBU). Loss: language modeling on the caption, conditioned on the projected image tokens.

A single epoch at batch 128 takes a few hours. The projector learns to map ViT space into LLM space, with no task-specific supervision.

Stage 2: visual instruction tuning

Unfreeze the projector (still trainable). Unfreeze the LLM (usually fully, sometimes LoRA). Train on 158k visual-instruction turns.

The instruction data is the trick. Liu et al. generated it by:

  1. Take a COCO image.
  2. Extract the text description (5 human captions + bounding-box list).
  3. Send to GPT-4 with three prompt templates:
    • Conversation: "Generate a back-and-forth dialogue between a user and assistant about this image."
    • Detailed description: "Give a rich, detailed description of the image."
    • Complex reasoning: "Ask a question that requires reasoning about the image, then answer it."
  4. Parse GPT-4's output into (instruction, response) pairs.

GPT-4 never sees the image, only the text description, so it invents plausible image content. That adds some noise, but 158k turns was enough to unlock dialogue.

Why the community copied this

  • No stage-1-specific losses to tune. LM loss throughout.
  • Projector trains in hours, not days.
  • LLM can be swapped (LLaVA-Llama2, LLaVA-Mistral, LLaVA-Llama3) by retraining just the projector.
  • Visual-instruction data pipeline uses GPT-4 and is cheap to regenerate for a new domain.

LLaVA-1.5 and LLaVA-NeXT

LLaVA-1.5 (October 2023) added:

  • Academic-task data (VQA, OKVQA, RefCOCO) mixed into instruction tuning.
  • Better system prompt.
  • 2048 → 32k context.

LLaVA-NeXT (January 2024) added:

  • AnyRes: split high-res images into a 2x2 or 1x2 grid of 336x336 crops, plus one global low-res thumbnail. Each crop becomes 576 tokens; total around 2880 visual tokens per image. OCR and chart tasks improved a lot.
  • Better instruction data mixture with ShareGPT4V (high-quality GPT-4V captions).
  • Stronger base LLMs (Mistral-7B, Yi-34B).

LLaVA-OneVision

Lesson 12.08 covers OneVision in depth. Short version: same projector, but trained with a curriculum that covers single-image, multi-image, and video in one model with shared visual-token budget.

The comparison to Q-Former

Q-Former (BLIP-2)MLP (LLaVA)
Visual tokens per image32576 (base) or 2880 (AnyRes)
Trainable params188M + LM~21M + LM
Stage 1 lossITC+ITM+ITGLM only
LLM drop-inRequires retrainSwap with minimal retrain
Multi-imageAwkwardNatural (concat)
VideoAwkwardNatural (per-frame concat)
Token budgetSmallLarge

MLP wins on simplicity and token flexibility. Q-Former wins on token budget. By late 2023 the token budget was no longer the binding constraint (LLM contexts grew to 32k-128k+) and simplicity dominated.

The prompt format

A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image> Describe this image in detail. ASSISTANT: The image shows ...

<image> is a placeholder token. Before tokenization, it is replaced with the 576 visual tokens (or 2880 with AnyRes). The LLM sees a longer sequence than plain text would give it, and it copes because stage 1 taught it what projected image tokens look like.

Parameter economy

LLaVA-1.5-7B breakdown:

  • CLIP ViT-L/14 @ 336: 303M (frozen stage 1, often unfrozen stage 2).
  • Projector (2x linear): ~21M trainable.
  • Llama-7B: 7B.
  • Total: 7.3B params. Trainable during stage 2: full 7B + 21M projector.

Training cost for stage 2: ~20 hours on 8xA100. This is the number that matters: one day, one node, reproducible. That is a big part of why LLaVA spread.

Check: The Concept0/29
1. What is visual instruction tuning in the LLaVA recipe?Define
One training example drawn as three fields: a hand-drawn scene with a house, a sun and a tree labelled image; the instruction Describe this image in detail; and a response that starts The image shows.
The instruction and the response opening come from the lesson's prompt example; the picture is hand-drawn.
2. In LLaVA-1.5, what does the 2-layer MLP projector map?Define
A left-to-right pipeline: image, CLIP ViT-L/14 at 336, a highlighted 2-layer MLP projector with GELU, and the LLM Vicuna-7B.
Component names from the lesson's LLaVA-1.5 architecture list; no sizes drawn.
3. In the prompt 'USER: <image> Describe this image in detail. ASSISTANT:', what happens to <image> before the sequence reaches the LLM?Define
The prompt line USER, image placeholder, Describe this image in detail, ASSISTANT. An arrow leads down to a dashed question-mark box labelled what the LLM receives at the image placeholder position.
4. During Stage 1 (projector alignment), which components are frozen and which train?Define
Stage 1 setup: a hand-drawn image with the caption a house under a sun, then three boxes ViT, MLP projector and LLM in a row, and a note that the loss is language modeling on the caption.
Caption text is illustrative; the loss description comes from the lesson.
5. Where did LLaVA's 158k instruction-response turns come from?Define
A card for the LLaVA instruction data of 158k turns with a sample instruction and response opening, and a dashed arrow from a dashed box labelled source with a question mark.
Sample wording from the lesson's prompt example.
6. How does LLaVA-NeXT's AnyRes handle a high-resolution image?Define
A large hand-drawn high-resolution image with an arrow marked with a question mark pointing into a CLIP ViT-L/14 at 336 box.
Hand-drawn picture, no grid shown.
7. What does it mean that the vision encoder is frozen in LLaVA training?Define
A training step as four boxes: image plus text, a highlighted CLIP encoder, a projector and an LLM. Under the CLIP encoder the word frozen with a question mark.
Schematic of the question, no markers on the other boxes.
8. What is ShareGPT4V in the LLaVA family of recipes?Define
Under the heading LLaVA-NeXT, January 2024, three boxes: AnyRes, a highlighted ShareGPT4V with its arXiv id 2311.12793, and Mistral-7B and Yi-34B.
Names and the citation come from the lesson; nothing is described.
9. What is VQA?Define
A hand-drawn scene with a house that has two windows, a sun and a tree, next to the typed question How many windows does the house have, with the answer 2.
Hand-drawn scene and typed example, not from a dataset.
10. What is the Prismatic VLMs paper?Define
A list of five references from the lesson with arXiv ids: two Liu et al. papers, ShareGPT4V, a highlighted Karamcheti et al. Prismatic VLMs entry, and LLaVA-OneVision.
Titles copied from the lesson's Further Reading links; descriptions omitted.
11. Liu et al. prompted GPT-4 with three templates to build the instruction data. Which set is correct?Define
An arrow from a highlighted box GPT-4 plus 3 prompt templates to a box instruction data. The template names are not shown.
Flow from the stem; the three names are the question.
12. For LLaVA-1.5-7B, which parameter breakdown matches the Parameter economy section?Define
LLaVA-1.5-7B drawn as three boxes: the vision encoder CLIP ViT-L/14 at 336, a highlighted projector of 2 linear layers, and a language model labelled 7B. The only size drawn is the 7B from the model name; no encoder or projector sizes.
Names from the lesson; encoder and projector sizes left out.
13. What did LLaVA-1.5 (October 2023) add on top of the original recipe?Define
A timeline of three boxes: original LLaVA with the two-stage recipe, a highlighted LLaVA-1.5 from October 2023, and LLaVA-NeXT from January 2024, with a note asking what LLaVA-1.5 added.
Dates from the lesson.
14. Why does LLaVA train only the projector on captions before touching the LLM?Explain
Two boxes in sequence: first, train the projector on captions; then, the LLM is trained too. A note asks why in this order.
Order as given in the stem.
15. How do the trainable parts differ between stage 1 and stage 2?Explain
Two rows of the same three boxes, ViT, MLP projector and LLM, one labelled stage 1 with image-caption data and one labelled stage 2 with GPT-4 instruction data. No box is marked.
Schematic of the question; markers are left out.
16. GPT-4 never sees the image when it writes LLaVA-Instruct. What is the consequence?Explain
A hand-drawn image on the left with a red crossed dashed link to a highlighted GPT-4 box, which outputs LLaVA-Instruct with 158k turns. A note says GPT-4 never sees the image.
Schematic of the stem.
17. Why does the MLP projector handle multi-image and video inputs naturally?Explain
Four small frame tiles f1, f2, f3 and fN with dots between, an arrow into a highlighted MLP projector then LLM box, and a note asking what goes into the LLM sequence.
Schematic of a video as frames.
18. What is the main tradeoff between the MLP projector and the Q-Former?Explain
Two pipelines in rows: Q-Former from BLIP-2 (image, ViT, Q-Former, LLM) and MLP from LLaVA (image, ViT, MLP, LLM). No numbers are drawn.
Schematic; the tradeoff is the question.
19. Why did the community settle on MLP projectors over the Q-Former by late 2023?Explain
An arrow labelled by late 2023 from a Q-Former box (BLIP-2 style) to a highlighted MLP projector box (LLaVA style), with a note asking why the community settled on the MLP.
Schematic of the stem.
20. Which string is a correctly formed LLaVA-format prompt for one image question, before <image> is expanded?Apply
Five loose chips in no order: What is in this picture, system prompt, ASSISTANT colon, image placeholder, USER colon.
The chips are the pieces; the order is the question.
21. The projector maps 1024 to 4096 to 4096 with biases. Roughly how many trainable parameters is that?Apply
Two boxes for the projector. Layer 1: weights 1024 by 4096 and a bias of 4096. Layer 2: weights 4096 by 4096 and a bias of 4096. A note asks for the total number of trainable parameters.
Shapes from the stem.
22. A 576-token image goes into a 2048-token context window. How many tokens are left for text?Apply
A bar for a 2048-token window. The first 576 tokens are an orange block labelled image 576; the remainder is labelled text with a question mark. Bar scaled to the window size.
Block widths are to scale.
23. You serve LLaVA-NeXT with a 2x2 AnyRes grid on a model with a 2048-token window, and requests fail. What is the likely cause?Analyze
A 2 by 2 grid of crops plus a thumbnail goes into a LLaVA-NeXT model with a 2048-token window, and the result box says fail.
Schematic of the stem.
24. A model trained only through stage 1 writes caption-style text and ignores instructions like 'ask a reasoning question'. What is missing?Analyze
A chat for a model trained through stage 1 only. The user turn asks for a reasoning question about the image; the assistant replies with the caption-style text a house with a tree and a sun.
Typed illustration of the stem, not real model output.
25. A developer inserts the ViT's 1024-dim patch vectors directly at the <image> position and skips the projector. What goes wrong?Analyze
ViT patches as 1024-dimensional vectors with a red arrow marked no projector going straight to the image placeholder in the LLM input sequence USER, image, question.
Schematic of the stem.
26. Chart and OCR questions fail with a single 336x336 view. Which change does the lesson credit with improving them, and at what cost?Analyze
A chart with the title Revenue by quarter in dollars M and four bars Q1 to Q4 on the left; on the right the same chart shrunk to a 336 by 336 view, where the title and labels turn into thin gray smears. A note says labels and numbers are too small.
Hand-drawn illustration of the stem, not a real chart.
27. Your team swaps Vicuna for Mistral as the LLM and keeps the same CLIP encoder. Which step does the Why the community copied this section say is enough to get a working model?Design
Two rows. Today: CLIP, projector, Vicuna. After the swap: the same CLIP encoder, a projector, and Mistral. A note asks what is enough to get a working model.
Schematic of the stem.
28. You need a VLM for satellite imagery. You have captions and object boxes but no annotators. Which plan fits the lesson?Design
Goal: a VLM for satellite imagery. Three boxes: have image captions, have object boxes, and do not have annotators.
Resources as given in the stem.
29. A product must compare four photos in one query within an 8k window. Which design fits, and what does it cost?Design
Four hand-drawn photos, photo 1 to photo 4, with an arrow to a box for an 8k window. A note says one query compares all four photos.
Hand-drawn photos.

Use It

code/main.py is a stdlib-only walkthrough of the forward pass, with no training. It has three demos:

  1. The 2-layer MLP projector at toy scale (16 patches of dim 16, mapped 16 → 32 → 24), with random weights, GELU written by hand, and a printed parameter count.
  2. A prompt builder. It writes the system prompt, a <image> placeholder, the user turn, and an empty assistant slot, then reports how many visual tokens the placeholder stands for (576). It does not splice real embeddings in.
  3. A context-budget printout showing what share of a 2k, 8k, 32k, and 128k window the visual block takes, followed by an AnyRes table that prints the visual token count for five tile grids.
Check: Use It0/4
1. Demo 1 projects 16 patches of dim 16 through 16 -> 32 -> 24 with biases. What parameter count does it print?Apply
Terminal output of demo 1: ViT out 16 patches of dim 16, MLP 16 to 32 to 24, a params line replaced by a question mark, and output 16 visual tokens of dim 24.
Real output of main.py demo 1; the params value is masked, and the sample-token line is left out.
2. Demo 3 prints the AnyRes cost of a 672x336 (1x2) image. How many visual tokens is that?Apply
Terminal output of demo 3: the header, a source note that tile_tokens equals 576, and the row for 672x336 (1x2) with its value replaced by a question mark.
Header line is real output; the tile_tokens note is from main.py's source; the 1x2 value is masked and the other rows are not shown.
3. build_llava_prompt is called with image_tokens=576. What does the returned raw_prompt contain at the image position?Apply
Code: a user string, a call to build_llava_prompt with image_tokens equal to 576, and a last line reading the raw_prompt field with a question mark for what sits at the image position.
First two lines are from main.py's demo_prompt; the last line is the question, typed.
4. You run an AnyRes 2x4 grid (1344x672) plus thumbnail through a LLaVA-style model with a 4096-token window. What happens?Apply
A 2 by 4 grid of 336 crops plus a thumbnail tile, a note that each costs 576 tokens, and a box for a model window of 4096 tokens.
Hand-drawn grid; tile cost from the lesson's code.

Ship It

This lesson produces outputs/skill-llava-vibes-eval.md. Given a LLaVA-family checkpoint, it runs a 10-prompt vibes-eval suite (3 captioning, 3 VQA, 2 reasoning, 2 refusal) and reports a human-readable scorecard. It is a smoke test for whether the projector and LLM are connecting, not a benchmark.

Author's Note

I'd run main.py first and read the AnyRes table before anything else. Exercise 3 asks for the token count of a 1344x672 image, and the table has a row for that grid, so you can check your arithmetic against it. Then compare the 2x2 row to the roughly 2880 tokens quoted for LLaVA-NeXT above. Counting the thumbnail is the part people forget.

Keep in mind how little the script does. The projector weights are random and nothing trains, so stage 1 and stage 2 exist only in the prose. The prompt builder returns a string with <image> in it and a number for how many tokens that placeholder expands to. If you want to feel the recipe, the next step is to splice the projected tokens into a real LLM's embedding sequence at the placeholder position and run the alignment loss on a handful of captions.

My bet is that the context-budget printout is the best argument for the MLP over the Q-Former. At a 2k window the visual block takes a large share of the sequence. At 32k or 128k it barely registers, and that shift is what made the extra tokens affordable.

Exercises

  1. Compute the trainable-parameter count for the 2-layer MLP projector at 1024 → 4096 → 4096. With GELU and bias, what fraction of LLaVA-1.5-7B does it represent?

  2. Construct a LLaVA prompt for a "refusal" case where the image contains a private individual. Write the expected assistant response. Why should LLaVA refuse this zero-shot and what training data would be needed to reinforce the refusal?

  3. Read the AnyRes section of the LLaVA-NeXT blog. Compute the visual token count for a 1344x672 image at AnyRes. Compare to base 576 tokens at 336x336.

  4. The LLaVA stage-1 projector is trained with LM loss on captions. What happens if you skip stage 1 and go straight to stage 2 (visual instruction tuning)? Cite the Prismatic VLMs ablation (arXiv:2402.07865) for the answer.

  5. LLaVA-Instruct-150k uses GPT-4 with COCO captions to generate instructions. For a new domain (medical X-rays, satellite imagery), describe the four-step data pipeline to generate domain instructions. What could go wrong at each step?

Key Terms

TermWhat people sayWhat it actually means
Projector"MLP bridge"2-layer MLP with GELU mapping ViT dim to LLM dim
Image token" placeholder"Prompt marker replaced by N projected visual tokens before inference
Visual instruction tuning"LLaVA stage 2"Training on GPT-4-generated (image, instruction, response) triplets
Stage 1 alignment"Projector pretraining"Freeze ViT and LLM, train projector with LM loss on captions
AnyRes"Multi-crop tiling"Split high-res image into a tile grid and concatenate each tile's visual tokens
LLaVA-Instruct"GPT-4-generated"158k instruction-response pairs synthesized from COCO captions + GPT-4
Vision encoder freeze"Backbone locked"CLIP weights do not update in stage 1, sometimes not in stage 2 either
ShareGPT4V"Better captions"1M dense captions generated by GPT-4V, used for higher-quality alignment
VQA"Visual question answering"Task of answering a free-form question about an image
Prismatic VLMs"Design-space paper"Karamcheti 2024 ablation systematically testing projector and data choices

Further Reading

Code

Ships With