BuildPython

From CLIP to BLIP-2 — Q-Former as Modality Bridge

CLIP aligns images and text but cannot caption, answer questions, or hold a conversation. BLIP-2 (Salesforce, 2023) fixed that with a small trainable bridge: 32 learnable query vectors cross-attend over a frozen ViT's features, and the result slots into a frozen LLM's input stream. 188M parameters of bridge connected an 11B LLM to a ViT-g/14. Adapter-based VLMs through 2026, including MiniGPT-4 and InstructBLIP, descend from it, and LLaVA is a close cousin. This lesson reads the Q-Former's architecture, walks through its two-stage training, and builds a toy that projects visual tokens to an LLM's embedding dim.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 02 (CLIP), Phase 7 (Transformers)

Learning Objectives

Explain why a trainable bottleneck between a frozen vision encoder and frozen LLM beats end-to-end finetuning in cost and stability.
Implement a cross-attention block where a fixed set of learnable queries attend to external image features.
Walk through BLIP-2's two-stage pretraining: representation (ITC + ITM + ITG) then generative (LM loss with frozen decoder).
Compare Q-Former to the simpler MLP projector used in LLaVA and argue when each choice wins.

The Problem

You have a frozen ViT that produces 256 patch tokens of dim 1408 per image, and a frozen 7B LLM that expects token embeddings of dim 4096. The obvious bridge is a linear layer from 1408 to 4096. It works, but it feeds all 256 patch tokens into the LLM's context, which costs 256 extra tokens per image. Over a batch of 32 images that is 8192 tokens spent on the visual modality alone.

The BLIP-2 question: can you compress those 256 tokens into far fewer (say 32) and still give the LLM enough to caption, answer questions, and reason about the image? And can you train that bridge without touching either frozen backbone, so the training cost is just the bridge's parameters?

The answer is a Q-Former: 32 learnable "query" vectors that cross-attend to the ViT's patch tokens and produce a 32-token visual summary for the LLM. It has 188M parameters and is trained with contrastive, matching, and generative objectives before it ever meets the LLM.

Warm-up0/3

1. In a standard cross-attention layer, where do the queries, keys, and values come from?Warm-up

Typed schematic with placeholder tokens. No wiring is drawn.

2. What does CLIP's contrastive loss do with a batch of image-text pairs?Warm-up

A three by three grid. Rows are images 1 to 3 and columns are the captions a dog on grass, a red bus and a bowl of soup. The diagonal cells are marked paired in data and the six other cells are marked not paired. — Typed schematic with illustrative captions. It shows only which image and text were paired in the batch.

3. What does the context window of a transformer LLM limit?Warm-up

Typed schematic. The token split is illustrative.

Check: The Problem0/2

1. Why is a plain linear-layer bridge from a frozen ViT to a frozen LLM expensive in context?Explain

A row of three boxes labelled frozen ViT, linear layer and frozen LLM. Below it a context window bar with a solid text prompt part, a dashed part labelled image tokens with a question mark, and an empty dashed part. — Typed schematic. Bar widths are not to scale.

2. How does BLIP-2 compress a 256-token image representation without touching the frozen backbones?Explain

A 16 by 16 grid of 256 patch tokens beside a frozen ViT box, a dashed box with a question mark that compresses to fewer tokens than 256, and a frozen LLM box. — Typed schematic.

The Concept

Learnable queries

The core trick: instead of letting the LLM's text tokens attend to image patches, introduce a new set of 32 learnable query vectors Q and let them attend to the patches. The queries are parameters of the model. They are learned during training, and the same 32 are used for every image.

After cross-attention, each query holds a compressed summary of the image. You can imagine one asking "what is the main object", another "what is the background", another "how many things are there". The queries do not literally specialize on semantic labels, though. They learn whatever encoding makes the downstream losses drop.

Architecture

The Q-Former is a small transformer (12 layers, ~100M params) with two paths:

Query path: 32 query vectors flow through self-attention (among themselves), then cross-attention over the frozen ViT's patch tokens, then an FFN.
Text path: a BERT-like text encoder that shares the self-attention and FFN weights with the query path. Cross-attention is disabled for the text path.

At training time both paths run. The queries and text interact through the shared self-attention, so the queries can condition on text for tasks that need it (ITM, ITG). At inference time for VLM handoff, only the queries flow through, yielding 32 visual tokens.

Two-stage training

BLIP-2 pretrains in two stages.

Stage 1 is representation learning, with no LLM. Three losses run together:

ITC (image-text contrastive): CLIP-style contrastive loss between pooled query tokens and the text CLS token.
ITM (image-text matching): a binary classifier that asks whether an image-text pair is a match, trained on hard-negative-mined pairs.
ITG (image-grounded text generation): a causal LM head on the text, conditioned on the queries. It forces the queries to encode content the text can be generated from.

Only the Q-Former trains. The ViT is frozen, and no LLM is involved.

Stage 2 is generative learning. Attach a frozen LLM (OPT-2.7B or Flan-T5-XL, for example), project the 32 query outputs to the LLM's embedding dim with a small linear layer, and prepend them to the text prompt. Train only the linear projection and the Q-Former on LM loss over the concatenated prompt + image + caption sequence.

After stage 2, the Q-Former plus projection is the full visual adapter. At inference: image → ViT → Q-Former → linear projection → prepended to text → frozen LLM emits output.

Parameter economics

BLIP-2 with ViT-g/14 (1.1B, frozen) + OPT-6.7B (6.7B, frozen) + Q-Former (188M, trained) comes to 8B total with 188M trained. The Q-Former is about 2.4% of the stack. Training cost reflects that: days on a handful of A100s versus weeks for end-to-end.

On quality, BLIP-2 matches or beats Flamingo-80B on zero-shot VQA while being 50x smaller. The bridge works.

InstructBLIP and the instruction-aware Q-Former

InstructBLIP (2023) gives the Q-Former one more input: the instruction text itself. At cross-attention time the queries see both the image patches and the instruction, so they can specialize per instruction ("count the cars", "describe the mood") instead of learning one fixed summary. It showed benchmark gains on held-out tasks.

MiniGPT-4 and the projector-only approach

MiniGPT-4 kept the Q-Former but trained only the output linear projection and froze everything else. That is cheap, and the price is quality: the queries were BLIP-2's, not tuned for MiniGPT-4. Good for fast iteration, not the best architecture.

Why LLaVA went simpler

LLaVA (2023, Lesson 12.05) replaced the Q-Former with a plain 2-layer MLP that projects each ViT patch token into LLM space. For a 24x24 grid that is 576 tokens per image, all fed to the LLM. Compression is worse, but the LLM gets to attend over raw patches. At the time this was controversial. By late 2023 it was dominant, because visual instruction data (LLaVA-Instruct-150k) showed the MLP could be trained to preserve enough signal. The tradeoff is that LLaVA's context fills faster, but it scales naturally to multi-image and video.

By 2026 the field had split. The Q-Former survives where token budget matters (long video, many images), and the MLP projector dominates where raw quality per token is the priority.

Gated cross-attention: Flamingo, the ancestor

Flamingo (Lesson 12.04) predated BLIP-2 and used the same cross-attention idea, but at each frozen LLM layer rather than as a single bridge. BLIP-2 showed you can compress down to the input layer only and still get a working model. Gemini and Idefics combine both: interleaved input tokens plus optional gated cross-attention for in-context few-shot.

The 2026 descendants

Q-Former: BLIP-2, InstructBLIP, MiniGPT-4, and most video-language models for token budget reasons.
Perceiver resampler: Flamingo's variant (Lesson 12.04); the Idefics family, Eagle, OmniMAE.
MLP projector: LLaVA, LLaVA-NeXT, LLaVA-OneVision, Cambrian-1.
Attention pool: VILA, PaliGemma.

These four are all valid choices. What decides between them is whether you are constrained on token budget or on quality per token.

Check: The Concept0/30

1. What is a Q-Former?Define

A three-row name card for the Q-Former: the term, what people say (Querying transformer), and where it was introduced (BLIP-2, Li et al., arXiv 2301.12597). — Typed from the lesson's Key Terms table and reading list.

2. What are the learnable queries in a Q-Former?Define

A single row of orange chips labelled q1 to q8 followed by an ellipsis chip, under the heading the query vectors of one Q-Former. — Typed schematic. The number of queries is not shown.

3. In the Q-Former, which side supplies the queries and which supplies the keys and values of cross-attention?Define

Two source boxes, learnable queries in blue and ViT patch tokens in orange, beside a cross-attention layer with three unconnected inputs named queries, keys and values. — Typed schematic. No wiring is drawn.

4. What does the ITC loss train the Q-Former to do?Define

Typed schematic with illustrative captions.

5. What is ITM in BLIP-2's first training stage?Define

Three image-text pairs (image A with a dog on grass, image B with a red bus, image C with two hikers) go into a box labelled ITM, which has one output box with a question mark. — Typed schematic with illustrative pairs. The output is left as a question mark.

6. What does ITG force the Q-Former's queries to do?Define

Typed schematic with an illustrative caption.

7. What is BLIP-2's two-stage pretraining?Define

A row of the four BLIP-2 parts ViT, Q-Former, projection and LLM, above two dashed boxes labelled stage 1 and stage 2 whose contents are not shown. — Typed schematic.

8. What does it mean that the ViT and the LLM are frozen backbones in BLIP-2?Define

An image tile feeds three boxes in a row labelled ViT, bridge and LLM, then a loss box. The ViT and the LLM are captioned backbone and the middle box is captioned between them. — Typed schematic.

9. What does the projection head do after the Q-Former in stage 2?Define

Typed schematic.

10. What is a Perceiver resampler?Define

Four boxes naming bridge designs from the lesson: Q-Former, Perceiver resampler (highlighted), MLP projector and Attention pool, with a note citing Perceiver IO, arXiv 2107.14795. — Typed from the lesson's list of bridge designs and reading list.

11. What is shared between the Q-Former's query path and its text path?Explain

A column of three blocks labelled self-attention, cross-attention and FFN under the heading one Q-Former layer, with two input boxes, queries and text tokens, drawn at the left with no arrows. — Typed schematic. No wiring is drawn.

12. Does each of the 32 queries specialize on a human-readable question such as "what is the main object"?Explain

An image tile beside a query chip q1 with the quoted example question what is the main object, followed by chips q2 to q32 with no text. — Typed schematic. The quoted question is the stem's example.

13. During BLIP-2's Stage 1 (representation learning), which components are trained?Explain

A row of four BLIP-2 parts labelled ViT, Q-Former, projection and LLM, with the heading stage 1: representation learning and the losses ITC, ITM and ITG listed below. — Typed schematic.

14. Why does stage 1 include ITG in addition to ITC and ITM?Explain

Three boxes labelled ITC, ITM and ITG under the heading stage 1 losses, run together, with ITG highlighted. — Typed schematic.

15. In stage 2, what is trained and with which loss?Explain

A chain of boxes ViT, Q-Former, projection and LLM ending in a loss box, with a text input chip below it. — Typed schematic.

16. What fraction of BLIP-2's 8B total parameters is the trained Q-Former, and what does that imply for training cost?Explain

Three equal-width boxes in a row: ViT-g/14 image encoder, Q-Former trained, and OPT-6.7B LLM, headed BLIP-2 stack with 8B parameters in total. Sizes are not shown. — Typed schematic from the lesson. Box widths are equal and do not show parameter counts.

17. How does InstructBLIP extend the base Q-Former?Explain

An image tile feeds a Q-Former box whose output goes to an LLM box. Two instruction chips, count the cars and describe the mood, sit below with no arrows. — Typed schematic. The two instructions are the lesson's examples; no arrows are drawn for them.

18. Which pairing correctly matches a BLIP-2 Stage 1 loss to its function?Explain

A Q-Former box with three arrows to three boxes labelled ITC, ITM and ITG. Their functions are not shown. — Typed schematic.

19. Why did LLaVA replace the Q-Former with a 2-layer MLP despite the Q-Former's token-budget advantage?Explain

A ViT box leads to a dashed grey Q-Former box, marked replaced by, then to a highlighted 2-layer MLP box and an LLM box, with a note that LLaVA in 2023 used the MLP in place of the Q-Former. — Typed schematic.

20. What does LLaVA's MLP projector give up and gain relative to a Q-Former?Explain

Two rows, each with a ViT box, a bridge (Q-Former for BLIP-2 style, 2-layer MLP for LLaVA style) and an empty dashed bar for how many image tokens reach the LLM's context, marked with a question mark. — Typed schematic.

21. Which bridge family does the Perceiver resampler belong to?Explain

Twelve model names in a grid with no grouping (BLIP-2, LLaVA, Idefics family, VILA, InstructBLIP, Eagle, PaliGemma, LLaVA-NeXT, MiniGPT-4, OmniMAE, LLaVA-OneVision, Cambrian-1), and a highlighted box below labelled Perceiver resampler. — Typed from the lesson's 2026 descendants list, shuffled into one pool.

22. What tradeoff does MiniGPT-4's approach to the Q-Former involve?Explain

A pipeline of four boxes for MiniGPT-4: ViT, Q-Former, projection and LLM, with a note that which parts are trained is the question. — Typed schematic.

23. How does Flamingo's use of cross-attention differ from BLIP-2's?Explain

Two panels, Flamingo and BLIP-2, each with a vision features box and a stack of four frozen LLM layers, with no arrows between them. — Typed schematic. No wiring is drawn.

24. A 10-minute video is sampled at 1 FPS to 60 frames. How many visual tokens does an MLP projector (576 per frame) put into the context?Apply

A strip of video frames f1 to f5, an ellipsis and f60 under the heading 10-minute video sampled at 1 FPS to 60 frames, beside a 24 by 24 patch grid for one frame labelled 576 tokens with an MLP projector. — Typed schematic. Frame thumbnails are placeholders.

25. A teammate says BLIP-2 has 8B parameters, so training it must cost as much as training any 8B model. Which assumption is wrong?Analyze

A BLIP-2 box labelled 8B parameters next to a speech bubble in which a teammate says that with 8B parameters training costs as much as any 8B model. — Typed schematic of the stem's claim.

26. A VLM built with an MLP projector runs out of context when a chat contains 20 images. Which diagnosis fits?Analyze

Typed schematic.

27. An implementation samples a fresh random set of 32 query vectors for every image at inference. Why is that a bug?Analyze

A code box showing queries = torch.randn(32, d) inside a loop over images at inference, and three images each with their own differently coloured row of query vectors, each labelled its own 32 with 8 drawn. — Typed illustration. Not code from the lesson.

28. You are building a single-image chat model with plenty of visual instruction data and a large context window, and quality per token is the priority. Which bridge fits best?Design

A four-row brief: images per chat is one, context window is large, visual instruction data is plenty, and the priority is quality per token. — Typed from the stem.

29. Your pipeline feeds 10 images into an 8k-token LLM, and the ViT yields 576 patches per image. Which choice fits?Design

Typed schematic from the stem's numbers.

30. You are building a video QA model that samples 360 frames from an hour of footage into an LLM with a 32k-token context. Which bridge fits?Design

Typed schematic. Thumbnails are placeholders.

Use It

code/main.py builds a stdlib Q-Former-style cross-attention at toy scale:

Simulate 64 image patch tokens (dim 16).
Instantiate 8 query vectors (dim 16).
Run scaled-dot-product cross-attention (Q from queries, K/V from patches).
Project to a toy LLM dim (24) with a linear layer.
Output 8 LLM-ready visual tokens.

The math is pure Python, nested loops over vectors. The sizes are scaled down from the 256 patches and 32 queries above, but the shapes and the mechanism are the same. Instead of the full weight matrix, the script prints a per-query summary: the entropy of the attention weights against the uniform baseline, and the top five patches each query pulled from. A second demo copies a few patch vectors in as the queries and checks that each query lands on its own patch.

Check: Use It0/4

1. In `demo_untrained` of main.py (8 queries, 64 patches of dim 16, LLM_DIM = 24), what comes out of the final projection?Apply

A code panel showing the constants of main.py (64 patches, 8 queries, dims 16, LLM_DIM 24) and the lines in demo_untrained that call cross_attention, build W_out and call linear_project, ending with a print of the projected shape marked with a question mark. — Constants (two per line here) and code lines are from the lesson's main.py. The final print line is added here for the question and has no output shown.

2. In `cross_attention`, each entry of `attn_weights` belongs to one query. How long is it and what does it sum to?Apply

Four lines of main.py inside cross_attention: for each q in Q it builds logits against every key, applies softmax to get weights, and appends weights to attn_weights. — Code lines are copied from the lesson's main.py.

3. In `demo_biased`, query 3 is a copy of `patches[48]` and all three weight matrices are identity. Which patch does it attend to most?Apply

An 8 by 8 grid numbering the 64 patches 0 to 63 with eight patches outlined: 5, 17, 33, 48, 60, 2, 11 and 27, where 48 is orange, next to the code lines for favored and queries and a note that query 3 is a copy of patches[48]. — Favored list and code are from the lesson's main.py (the queries line is wrapped onto two lines here).

4. A reader runs the untrained toy, sees mean entropy far below the uniform baseline, and concludes the queries have learned to focus. What is wrong with that conclusion?Analyze

A bar chart of the entropy of each of the eight queries in the untrained demo, from about 0.000 to 0.747, all far below a dashed line at the top marking the uniform baseline, with the mean entropy 0.199. — Values come from a real run of main.py. The baseline is drawn as a line without its number.

Ship It

This lesson produces outputs/skill-modality-bridge-picker.md. Given a target VLM configuration (vision encoder token count, LLM context budget, deployment constraints, quality target), it recommends Q-Former vs MLP vs Perceiver resampler with a short justification and a parameter-count estimate for each bridge.

Author's Note

I'd run main.py and read the entropy lines before anything else. The first demo uses random, unscaled weights and nothing is trained, yet the attention comes out sharply peaked: in my run most queries put nearly all their weight on a single patch, and the mean entropy sits far below the uniform baseline. Low entropy by itself tells you nothing about learning. The model has to earn its attention pattern against a loss, and this toy has no loss.

The second demo is the one that shows the mechanism. Queries are copies of patches 5, 17, 33, 48, 60, 2, 11 and 27, with identity weights, and each query's top patch is its own source patch. That is the whole idea of cross-attention here: a query is a question, and the patches it matches best get pulled into its summary.

Keep the toy in proportion. It has no text path, no ITC/ITM/ITG, and no stage 2, so the "learnable" queries are just random vectors. The TAKEAWAYS printout at the end still says 32 queries and 256 patches, while the code runs 8 and 64. If you want to feel why the bridge matters, do exercise 5 and count tokens for a video.

Exercises

Implement the cross-attention block in PyTorch. Verify that with 32 queries and 256 keys/values, the attention-weight matrix is 32 x 256 and each row sums to 1 after softmax.
In BLIP-2 stage 1 the Q-Former runs three losses simultaneously: ITC, ITM, ITG. Write the forward signature for each in pseudo-code. Which one requires the text encoder path to be active?
Compare parameter counts: Q-Former (12 layers, 768 hidden) vs a 2-layer MLP projector (1408 → 4096, two layers). At what LLM scale does the 188M Q-Former cost pay back in training efficiency?
Read Section 3.2 of the BLIP-2 paper (arXiv:2301.12597) on how the Q-Former is initialized. Explain why initializing from BERT-base (not random) accelerates convergence.
For a 10-minute video at 1 FPS sampled to 60 frames, compute the per-frame token cost at (Q-Former → 32 tokens/frame) vs (MLP projector → 576 tokens/frame). Which fits into a 128k-token LLM context window?

Key Terms

Term	What people say	What it actually means
Q-Former	"Querying transformer"	Small transformer with 32 learnable query vectors that cross-attend to frozen ViT features
Learnable queries	"Soft prompt for vision"	A fixed set of parameters that serve as the query side of cross-attention; learned per model, shared across all inputs
Cross-attention	"Q from here, K/V from there"	Attention where query, key, and value come from different sources; how the queries pull from ViT patches
ITC	"Image-text contrastive"	CLIP-style loss applied to Q-Former pooled queries vs text CLS
ITM	"Image-text matching"	Binary classifier on hard-negative-mined pairs; forces the queries to discriminate fine-grained mismatches
ITG	"Image-grounded text generation"	Causal LM loss where text is generated conditioned on queries; forces queries to encode text-decodable content
Two-stage pretraining	"Representation then generative"	Stage 1 trains Q-Former alone (ITC/ITM/ITG); Stage 2 attaches frozen LLM and trains only the projection + Q-Former
Frozen backbone	"Do not finetune"	The vision encoder and LLM weights are fixed; only the bridge trains
Projection head	"Linear to LLM dim"	Final linear layer mapping Q-Former output to the LLM's embedding dimension
Perceiver resampler	"Flamingo's version"	Similar learnable-query cross-attention, used by Flamingo at every layer rather than as a single bridge