Flamingo and Gated Cross-Attention for Few-Shot VLMs

BLIP-2 feeds 32 visual tokens into a frozen LLM's input layer. That works for one image per prompt. Now try interleaving many images with text: "here is image A, caption it; here is image B, caption it; now here is image C, caption it." The LLM's self-attention would have to handle image tokens and text tokens in a single stream, and deciding which positions can attend to which images gets fussy.

Flamingo's answer is to leave the LLM's input stream alone. Insert extra cross-attention layers between existing LLM blocks. Text tokens still flow through the LLM's causal self-attention as usual, and every few blocks they also cross-attend to image features through a new gated layer. The gate starts at zero, so at step zero the new layers are no-ops and the model behaves like the pretrained LLM. As training progresses the gate opens and visual information starts flowing.

The second question Flamingo had to answer: how do you handle a variable number of images (0, 1, or many) per prompt? A Perceiver resampler, a small cross-attention module, takes whatever number of patches you have and produces a fixed number of visual latent tokens. The LLM's cross-attention layer sees the same shape however many images are in the prompt.

The frozen LLM

Flamingo starts with a frozen Chinchilla 70B LLM. None of the 70B weights change, and the existing text self-attention and FFN run as they did before.

Perceiver resampler

For each image in the prompt, the ViT produces N patch tokens. The Perceiver resampler has K fixed learnable latents (Flamingo uses K=64). Each resampler block has two sub-steps:

1. Cross-attention: the K latents attend over the N patch tokens (Q from latents, K/V from patches).
2. Self-attention + FFN within the latents.

After 6 resampler blocks, the output is K=64 visual tokens of dim 1024, whatever the ViT's patch count. A 224x224 image (196 patches) and a 480x480 image (900 patches) both exit as 64 resampler tokens.

For video, the resampler is applied temporally: each frame's patches produce 64 latents, and a temporal positional encoding lets the model distinguish t=0 from t=N. The full video becomes T * 64 visual tokens.

Gated cross-attention

Between every M layers of the frozen LLM (Flamingo uses M=4), insert a new gated cross-attention block:

x_after_llm_block = llm_block(x_before)
cross = cross_attn(x_after, resampler_output)
gated = tanh(alpha) * cross + x_after
x_before_next_block = gated

• alpha is a learnable scalar initialized to zero.
• tanh(0) = 0, so at init the gated branch contributes zero.
• As alpha moves away from zero, the cross-attention contribution grows smoothly.
• Because of the residual connection, even a wide-open gate does not overwrite the LLM's text representation. It adds visual information on top.

This is the design choice that matters most in Flamingo. Visual conditioning is additive, gated, and zero at initialization, so on text-only inputs a Flamingo at step 0 behaves like Chinchilla 70B.

Masked cross-attention for interleaved inputs

In a prompt like " caption A caption B ?", each text token should only see images that came before it in the sequence. The cross-attention mask enforces this: the text token at position t attends only to resampler tokens of image i_t, the most recent image before position t. "Sees only the last preceding image" and "sees all preceding images" are both valid choices, and Flamingo chose the former.

In-context few-shot learning

A Flamingo prompt looks like:

<image1> A photo of a cat. <image2> A photo of a dog. <image3> A photo of a

The model sees the completion pattern and outputs "bird" (or whatever image3 shows), with no gradient steps. The frozen LLM's in-context learning carries through the gated cross-attention. That is the punchline of the paper.

Training data

Flamingo trained on three datasets:

1. MultiModal MassiveWeb (M3W): 43M web pages with interleaved images and text, in reading order.
2. Image-Text Pairs (ALIGN + LTIP): 4.4B pairs.
3. Video-Text Pairs (VTP): 27M short video clips.

OBELICS (2023) is an open reproduction of the interleaved web corpus, which Idefics, Idefics2, and most open "Flamingo-like" models train on.

OpenFlamingo and Otter

OpenFlamingo (2023) is the open reproduction. The architecture is the same (Perceiver resampler + gated cross-attention on a frozen LLaMA or MPT), with checkpoints at 3B, 4B, and 9B. Quality lags Flamingo because of the smaller base LLM and less data.

Otter (2023) builds on OpenFlamingo with instruction tuning on MIMIC-IT (a dataset of multimodal instructions), which shows gated cross-attention works for instruction following too.

The descendants

• Idefics / Idefics2 / Idefics3: Hugging Face's gated cross-attention lineage, getting simpler over time. Idefics2 dropped the resampler for direct patch tokens with adaptive pooling.
• Flamingo-to-Chameleon transition: by 2024 many teams had moved to early fusion (Lesson 12.11). Flamingo-style gated cross-attention still shows up in production where the backbone has to stay frozen.
• Gemini's interleaved input: conceptually inherits Flamingo's interleaved-format flexibility, though the mechanism is proprietary.

Comparison to BLIP-2

Pick BLIP-2 for single-image VQA on a budget. Pick Flamingo/Idefics2 for interleaved, few-shot, or multi-image reasoning.

code/main.py runs three demos:

1. A Perceiver resampler on 36, 196, and 900 random patch tokens, each reduced to 8 latents with pure Python cross-attention. The output shape is the same for all three.
2. A gated cross-attention step with alpha = 0, where the output equals the input (LLM unchanged), then alpha = 2.0, where the visual contribution is mixed in (tanh(2.0) is about 0.96).
3. An interleaved-mask builder that produces the 2D attention mask for a sequence of images and text, where each text token sees only the most recent preceding image.

This lesson produces outputs/skill-gated-bridge-diagnostic.md. Given an open VLM's config (resampler Y/N, cross-attn frequency, gate scheme), it identifies the Flamingo lineage elements and explains the freezing strategy. It helps when a fine-tune has degraded text performance, where the likely cause is a gate that got too wide too fast.

1. Compute Flamingo-9B's visual parameter count: 9B LLM + 1.4B gated cross-attention layers + 64M resampler. What fraction of total params is trained?

2. Implement the gated residual y = tanh(alpha) * cross + x in PyTorch. Show experimentally that with alpha=0, y==x exactly at init.

3. Read OpenFlamingo Section 3.2 (arXiv:2308.01390) on how they handle multiple images in a batch when each prompt has a different image count. Describe the padding strategy.

4. Why does Flamingo's cross-attention mask let a text token attend to only the most recent preceding image rather than all preceding images? Read the Flamingo paper Section 2.4 and explain the tradeoff.

5. In-context few-shot: construct a prompt with 4 examples of "image → color of main object" for a new Flamingo variant. Describe the expected accuracy pattern as you vary the number of examples from 0 to 8.

• Alayrac et al.: Flamingo (arXiv:2204.14198): the original paper.
• Awadalla et al.: OpenFlamingo (arXiv:2308.01390): open reproduction.
• Laurençon et al.: OBELICS (arXiv:2306.16527): interleaved web corpus.
• Jaegle et al.: Perceiver IO (arXiv:2107.14795): the general Perceiver architecture.
• Li et al.: Otter (arXiv:2305.03726): instruction-tuned Flamingo descendant.
• Laurençon et al.: Idefics2 (arXiv:2405.02246): modern simplification of the Flamingo approach.