TAI BUI
Back to curriculum
LearnPython

Open-Weight VLM Recipes: What Actually Matters

The 2024-2026 open-weight VLM literature is a forest of ablation tables. Apple's MM1 tested 13 combinations of image encoder, connector, and data mix. Allen AI's Molmo showed that detailed human captions beat GPT-4V distillation. Cambrian-1 compared 20+ encoders. Idefics2 formalized the five-axis design space. Prismatic VLMs compared 27 training recipes on a controlled benchmark. Across all of that, a few results hold from paper to paper: the image encoder matters more than the connector, the data mixture matters more than either, and detailed human captions beat distilled synthetic data. This lesson reads the tables so you don't have to.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 05 (LLaVA baseline)

Learning Objectives

  • Name the five-axis VLM design space: image encoder, connector, LLM, data mix, resolution schedule.
  • Read an MM1 / Idefics2 / Cambrian-1 ablation table and predict which knob moves a given benchmark.
  • Pick a recipe (encoder, connector, data, resolution) for a new VLM given a compute budget and task mix.
  • Explain why detailed human captions beat GPT-4V distillation at the same token count.

The Problem

Many open-weight VLMs exist, and most of the gap between a good one and a state-of-the-art one is not architecture. It is data, resolution schedule, and encoder choice. Knowing which knob to turn first when your model underperforms saves you from an expensive wrong guess.

The 2023 wave (LLaVA-1.5, InstructBLIP, MiniGPT-4) ran on caption-pair pretraining plus LLaVA-Instruct-150k. That is a good baseline, and it topped out around MMMU 35%.

The 2024 wave (MM1, Idefics2, Molmo, Cambrian-1, Prismatic VLMs) ran exhaustive ablations, and the results were surprising and practical.

Warm-up0/3
1. Which vision encoder did the LLaVA-1.5 baseline use?Warm-up
A photo goes into a vision encoder drawn as a dashed question box, then a 13B language model, then a text answer. A row below lists the training data LLaVA-Inst-150k and MMMU 35.3.
Row for LLaVA-1.5 from main.py (LLM 13B, data, MMMU). The encoder box is hidden. The flow is a typed schematic.
2. You want to know which of four design changes made a model's accuracy rise. Which experimental setup tells you the most reliably?Warm-up
A previous model, four design changes labelled A to D applied together, and a new model whose accuracy rose, with a dashed box asking which change caused the rise.
Typed schematic of the stem.
3. A photo of a street sign is shrunk to a tiny thumbnail before a model sees it. Which task gets hurt the most?Warm-up
A drawn street photo with a white sign reading ELM ST on a gray post, a large brown dog, and a sun in a blue sky, with an arrow to a small dashed box labelled tiny thumbnail.
Illustrative drawing, not a real photo.

The Concept

The five-axis design space

Idefics2 (Laurençon et al., 2024) named the axes:

  1. Image encoder. CLIP ViT-L/14, SigLIP SO400m/14, DINOv2 ViT-g/14, InternViT-6B. Encoders differ in patch size, resolution, and pretraining objective.
  2. Connector. MLP (2-4 layers), Q-Former (32 queries + cross-attn), Perceiver Resampler (64 queries), C-Abstractor (convolutional + bilinear pooling).
  3. Language model. Llama-3 8B / 70B, Mistral 7B, Phi-3, Gemma-2, Qwen2.5. LLM size is the dominant param cost.
  4. Training data. Caption pairs (CC3M, LAION), interleaved (OBELICS, MMC4), instruction (LLaVA-Instruct, ShareGPT4V, PixMo, Cauldron).
  5. Resolution schedule. Fixed 224/336/448, AnyRes, native dynamic. Ramped during training or constant.

Every production VLM makes a choice on each axis. Most of the variance in MMMU scores comes from axes 1, 4, and 5, not from which connector you picked.

Axis 1: encoder > connector

MM1 Section 3.2 showed that swapping from CLIP ViT-L/14 to SigLIP SO400m/14 added 3+ points MMMU, while swapping the connector from MLP to Perceiver Resampler added less than 1 point. Idefics2 replicated it: SigLIP > CLIP, and Q-Former ≈ MLP ≈ Perceiver at the same token count.

Cambrian-1's "Cambrian Vision Encoders Match-Up" (Tong et al., 2024) ran 20+ encoders on a vision-centric benchmark (CV-Bench). The top of the leaderboard is a mix of DINOv2 and SigLIP, CLIP sits in the middle of the pack, and ImageBind and ViT-MAE score lower. The gap from CLIP ViT-L to DINOv2 ViT-g/14 is ~5-9 points on CV-Bench (9 in the condensed table in main.py).

The 2026 default encoder for open VLMs is SigLIP 2 SO400m/14 for semantic + dense features, sometimes concatenated with DINOv2 ViT-g/14 features (Cambrian's "Spatial Vision Aggregator" does this).

Axis 2: connector design is a wash

MM1, Idefics2, Prismatic, and MM-Interleaved reached the same conclusion: at a fixed visual-token count, connector architecture barely matters. A 2-layer MLP on mean-pooled patches performs within 1 point of a 32-query Q-Former at the same token budget.

Token count does matter. More visual tokens means more LLM compute and better performance up to a point, then diminishing returns. 64 tokens per image is too few for OCR. 576-1024 tokens is the sweet spot for most open VLMs, and 2048+ helps only for documents and charts.

Q-Former vs MLP is a cost question, not a quality question. Q-Former caps tokens at 32-64 regardless of image resolution, while MLP emits all patch tokens. For high-res inputs, Q-Former saves LLM context. For low-res inputs, the difference is noise.

Axis 3: LLM size sets the ceiling

Doubling the LLM from 7B to 13B adds 2-4 points on MMMU across the VLM papers. By 70B most benchmarks saturate. The VLM's multimodal reasoning ceiling is the LLM's text reasoning ceiling. The visual encoder can only feed it, not reason for it.

That is why Qwen2.5-VL-72B and Claude Opus 4.7 lead on MMMU-Pro and ScreenSpot-Pro: the language brain is huge. Clever connector design will not turn a 7B VLM into a 70B one.

Axis 4: data, where detailed human captions beat distillation

Molmo + PixMo (Deitke et al., 2024) is the 2024 result to read first. Allen AI had human annotators describe images in 1-3 minute dense speech-to-text passes, yielding 712K densely-captioned images. There is no GPT-4V distillation anywhere in the training data.

Molmo-72B beat Llama-3.2-90B-Vision on 11 of 11 benchmarks. The delta comes from caption quality, not architecture. Detailed human captions contain 5-10x more information per image than short web captions, and they stay factually grounded where GPT-4V distillation hallucinates.

ShareGPT4V (Chen et al., 2023) and Cauldron (Idefics2) followed the same playbook with mixed human + GPT-4V captions. For the 2026 frontier the ordering is caption density, then caption quantity, then distillation convenience.

Axis 5: resolution and its schedule

Idefics2's ablations: 384 -> 448 adds 1-2 points. 448 -> 980 with image splitting (AnyRes) adds another 3-5 on OCR benchmarks. Flat resolution training plateaus at medium accuracy, while resolution ramping (start 224, finish 448 or native) trains faster and ends higher.

Cambrian-1 ran a resolution vs tokens trade-off. At fixed compute you can have more tokens at lower resolution or fewer tokens at higher resolution. Higher resolution wins for OCR, and lower-res-more-tokens wins for general scene understanding.

The 2026 production recipe: train Stage 1 at 384 fixed, then Stage 2 with dynamic resolution up to 1280 for OCR-heavy tasks.

The Prismatic controlled comparison

Prismatic VLMs (Karamcheti et al., 2024) controlled all the axes. Same 13B LLM, same instruction data, same evaluation, and only one axis varies at a time. Results:

  • Per-image visual-token count explains ~60% of variance.
  • Encoder choice explains ~20%.
  • Connector architecture explains ~5%.
  • Everything else (data mix, scheduler, LR) the remaining ~15%.

This is a rough decomposition, but it is the cleanest answer to "what should I ablate first" in the literature.

A picker for 2026

Given the evidence, here is the default open-VLM recipe for a new project in 2026:

  • Encoder: SigLIP 2 SO400m/14 at native resolution with NaFlex, concatenated with DINOv2 ViT-g/14 for dense features if you need segmentation/grounding.
  • Connector: 2-layer MLP on patch tokens. Skip Q-Former unless you are token-constrained.
  • LLM: Qwen2.5 / Llama-3.1 / Gemma 2, 7B for cost, 70B for quality, picked by target latency.
  • Data: PixMo + ShareGPT4V + Cauldron, topped up with task-specific instruction data.
  • Resolution: dynamic (min 256, max 1280 pixels per long side).
  • Schedule: Stage 1 alignment (projector-only), Stage 2 full fine-tune, Stage 3 task-specific fine-tune.

Each default above traces back to an ablation in the papers listed under Further Reading.

Check: The Concept0/27
1. In a LLaVA-style VLM, which component maps the vision encoder's patch features into the LLM's embedding space?Define
An image goes into a vision encoder, which outputs a 2 by 2 grid of four patch features. A dashed question box sits between those features and a row of four embedding squares that feed the LLM.
Typed schematic.
2. Which list gives the five axes of the VLM design space that Idefics2 named?Define
A card with the MM1-3B row from main.py showing its MMMU 38.6, CV-B 59.0 and DocVQA 62.0, and below it a dashed box asking for the five axes of the design space that Idefics2 named.
Scores from the MM1-3B entry of RECIPES in main.py. The design fields of the row are left off.
3. In VLM design, what is an ablation?Define
Three run cards, run 1 with an MLP connector, run 2 with a Q-Former and run 3 with a Perceiver Resampler, each with an unknown MMMU score.
Typed schematic; no scores shown.
4. What is a detailed human caption?Define
A drawn street photo with a sign reading ELM ST, a brown dog and a sun, beside a text box holding a few sentences that describe the scene.
Illustrative drawing and description, both typed for this quiz.
5. In VLM training data, what does distillation mean?Define
A drawn desk with a mug and a laptop beside a record with an image line, a caption line reading A white mug sits on a wooden desk beside an open laptop, and a made-by line holding a dashed question box.
Typed illustration of one training pair; the caption and the drawing are made up for this quiz.
6. What is AnyRes (dynamic resolution) used for?Define
A flow from an image into a dashed box labelled AnyRes 672 with the note used for: ?, then a vision encoder, then an LLM. The title says the resolution field of the LLaVA-NeXT row in main.py reads AnyRes 672.
Resolution field from the LLaVA-NeXT entry of RECIPES in main.py. The flow is a typed schematic.
7. What is a resolution ramp?Define
A chart with training progress from start to end on the horizontal axis and input resolution on the vertical axis, with ticks at 224 and 448 pixels and a dashed box labelled resolution schedule with a question mark.
Typed schematic; 224 and 448 are the values named in the lesson and in main.py's notes. No curve is drawn.
8. What does a vision-centric benchmark such as CV-Bench measure?Define
A bar chart of nine recipes' CV-Bench scores from main.py: LLaVA-1.5 56.0, LLaVA-NeXT 58.5, Idefics2-8B 60.0, MM1-3B 59.0, MM1-30B 64.0, Molmo-7B-D 65.0, Molmo-72B 73.0, Cambrian-1-8B 67.8 and Prismatic-7B default 58.0.
Scores as printed by main.py (they are condensed figures, read the structure, not the decimals).
9. What is PixMo?Define
A card citing Deitke et al., Molmo and PixMo, arXiv 2409.17146, and a card with the Molmo-7B-D row from main.py: 7B LLM, AnyRes 672, MMMU 45.3, CV-Bench 65.0, DocVQA 92.4.
Citation from the lesson's Further Reading; scores from main.py. The training-data field of the row is left out.
10. In MM1's ablations, what did swapping CLIP ViT-L/14 for SigLIP SO400m/14 add to MMMU, compared with swapping an MLP connector for a Perceiver Resampler?Explain
Two cards, swap 1 from CLIP ViT-L/14 to SigLIP SO400m/14 and swap 2 from an MLP to a Perceiver Resampler connector, each with an arrow down to a dashed box for the unknown change in MMMU.
Typed schematic of the stem.
11. What does the Prismatic VLMs controlled comparison say explains benchmark variance across VLM designs?Explain
A flow from 27 training recipes to benchmark scores to a dashed box for the share of variance that each factor explains.
Typed schematic; the 27 recipes are from the lesson; no shares shown.
12. What is the key result of Molmo and PixMo for caption data?Explain
Two model cards. Molmo-72B shows an LLM of 72B, MMMU 54.1, CV-B 73.0, DocVQA 93.5 and unknown training captions. Llama-3.2-90B-Vision shows a 90B LLM and no scores.
Molmo-72B numbers from main.py; the Llama card is from the stem and the lesson.
13. Which visual-token budget is the sweet spot for most open VLMs, and what goes wrong below it?Explain
An image goes through a vision encoder to N visual tokens and then an LLM whose compute grows with N. Below, the LLM input sequence is a bar with a block of visual tokens, N unknown, followed by the text prompt.
Typed schematic; block widths are not to scale.
14. What did Cambrian-1's 20+ encoder comparison on CV-Bench find?Explain
A pool box listing CLIP ViT-L, DINOv2 ViT-g/14, SigLIP, ImageBind, ViT-MAE and 15 or more others, with an arrow to a CV-Bench card.
Typed schematic; names are those in the stem's options and the lesson.
15. What did Idefics2's resolution ablations find?Explain
Three squares of increasing size labelled smaller, larger and largest, standing for increasing input resolution, with a dashed box for the unknown gain.
Typed schematic; the real resolutions are not drawn.
16. Why does going from a 7B to a 13B LLM help reliably, while going from 7B to 70B eventually saturates?Explain
A vision encoder box with an arrow into three LLM boxes of growing height labelled 7B, 13B and 70B, and a note that benchmark gain per step up is unknown.
Typed schematic; box heights are a size cue only.
17. Why can a Q-Former save LLM context on high-resolution inputs when an MLP connector cannot?Explain
A low-resolution input drawn as 36 patches and a high-resolution input drawn as 144 patches, each followed by two cards, an MLP connector and a Q-Former, whose output token counts are unknown.
Typed schematic. The grids are illustrative sizes, not a real model's.
18. At fixed compute, which trade-off did Cambrian-1 find between resolution and token count?Explain
Two configurations: a small image with a fine 6 by 6 token grid, labelled lower resolution and more tokens, and a larger image with a coarse 3 by 3 grid, labelled higher resolution and fewer tokens. A note says both use the same compute and are scored on OCR and general scene understanding.
Typed schematic; grid sizes are illustrative.
19. Why do detailed human captions beat GPT-4V distillation at the same token count?Explain
A drawn street photo beside two caption boxes, a detailed human caption and a GPT-4V caption, each drawn as three gray bars standing for hidden text of the same length, with the training result marked unknown.
Typed schematic; the caption text is hidden.
20. Why does the 2026 default recipe skip the Q-Former unless you are token-constrained?Explain
Patch tokens go to either a 2-layer MLP or a Q-Former, then to the LLM, with a note that the MMMU score for each connector is unknown.
Typed schematic.
21. You are building a 7B document-reading VLM and its OCR scores lag. The training data is already fixed. Which change should you try first?Apply
A card with the Prismatic-7B default row from main.py: SigLIP SO400m/14, MLP-2, 7B LLM, resolution 336, training data marked fixed, DocVQA 70.0, flagged as lagging.
Row from main.py used as a stand-in for the stem's 7B model; the stem gives no numbers, and its data field is replaced by 'fixed'.
22. Using the 60/20/5/15 Prismatic split, what share of the variance would you cover by ablating only the two largest factors?Apply
A bar from 0 to 100 split into four blocks of widths 60, 20, 5 and 15, with a note asking what share the two largest cover.
Block widths come from the stem's 60/20/5/15; names are left off.
23. A team swaps its MLP connector for a Perceiver Resampler at the same token count, expecting about +5 MMMU, and gets under 1. Which assumption was wrong?Analyze
Two bars of MMMU gain after swapping an MLP connector for a Perceiver Resampler at the same token count: expected about 5, measured under 1.
Figures from the stem; the measured bar length is illustrative.
24. A VLM trained on GPT-4V captions confidently describes objects that are not in the image. Which ingredient from the lesson explains this?Analyze
A drawn desk with a mug and a laptop and no animal, beside a model reply that mentions a gray cat sleeping next to them, with a note that the training data was captions written by GPT-4V.
Typed illustration of the stem; the drawing and the reply are made up for this quiz.
25. You must ship a 7B VLM that reads dense invoices and charts under a fixed compute budget. Which first move fits the evidence best?Design
A drawn invoice page with text lines and a small bar chart, beside a compute-budget bar marked fixed, a card saying LLM size 7B, and a note asking for the first change to try.
Illustrative drawing; the page content is not real text.
26. You want to know how much of a gain comes from better captions rather than a better encoder. Which design isolates the caption effect?Design
A bar labelled total gain of the new model, with two dashed boxes below it for the unknown part from better captions and the unknown part from better encoder, and a note asking how to design runs to measure the caption share.
Typed schematic; the split is unknown.
27. A new 2026 project needs segmentation and grounding in addition to general understanding. Which encoder setup does the lesson's picker favor?Design
The street drawing with an orange pixel mask over the dog and a green box around the sign, labelled segmentation and grounding, the dense tasks the project needs.
Illustrative drawing of the two tasks; no model output.

Use It

main.py encodes condensed versions of these ablation tables as plain Python data and prints them: a table of canonical open VLMs with MMMU, CV-Bench, and DocVQA scores, the Prismatic-style axis-impact breakdown, the encoder-swap rows, and the data-mix rows. There is no parser and no numpy, just dicts and print tables.

On top of the tables sits a small picker. Give it a parameter budget and a task profile (ocr, agent, reasoning, or equal weights), and it filters recipes by LLM size, scores each one as a weighted sum of the three benchmarks, and prints the top three with their encoder, data, and resolution. The numbers are rounded and condensed, so read the structure of the evidence, not the decimals.

Check: Use It0/6
1. You run the picker in main.py with a 10B budget and the ocr profile. Molmo-72B, which has the best scores in the table, never appears. Why?Apply
The nine-row recipe table printed by main.py, with LLM size and MMMU, CV-B and DocVQA scores, and below it the call pick_recipe(10, ocr) with a note that Molmo-72B is not in the output.
Table captured from running main.py; the note restates the stem.
2. In the encoder-swap table (7B LLM, same data), CLIP scores 56.0 on CV-Bench and 70.0 on DocVQA, while DINOv2 scores 65.0 and 52.0. Which encoder fits a product that only reads scanned documents?Apply
A small table with two rows from the encoder-swap table: CLIP with CV-Bench 56.0 and DocVQA 70.0, and DINOv2 with CV-Bench 65.0 and DocVQA 52.0. Below it a card saying the product reads scanned documents only.
Numbers from the stem, matching main.py's encoder-swap rows.
3. A breakdown's six weights sum to 115, and you rebase them to sum to 100. What happens to a weight of 23?Apply
A bar of total 115 with a 23 segment, and below it a shorter bar of total 100 next to a dashed box asking what the 23 becomes.
Typed schematic; bars are drawn to the same scale (115 and 100 units); the rebased share is not drawn.
4. You run the picker at a 10B budget with the ocr profile and again with the agent profile. What do you see?Apply
Two panels. The first calls pick_recipe(10, ocr) with weights mmmu 0.4, cv 0.3, doc 1.2. The second calls pick_recipe(10, agent) with weights mmmu 1.0, cv 1.2, doc 0.8. Each ends with the top three printed marked unknown.
Weights read from pick_recipe in main.py.
5. A teammate says DINOv2 is a worse encoder than CLIP because it scored 52.0 on DocVQA against CLIP's 70.0. What is wrong with that claim?Analyze
A short quote, DINOv2 is a worse encoder than CLIP, above two bars of DocVQA: CLIP 70.0 and DINOv2 52.0.
Values from the stem and main.py's encoder-swap rows.
6. The picker returns the same top three for the ocr and agent profiles. What is the most sensible reading?Analyze
Two weight panels, profile ocr with weights 0.4, 0.3, 1.2 and profile agent with weights 1.0, 1.2, 0.8, with an equals sign between them for the identical top three, and the score formula below.
Weights read from pick_recipe in main.py.

Ship It

This lesson produces outputs/skill-vlm-recipe-picker.md. Given a target task mix, a compute budget, and a latency target, it emits a full recipe (encoder, connector, LLM, data mix, resolution schedule) with a citation to the ablation behind each choice. That way you don't rebuild the Idefics2 ablation table at the start of every VLM project.

Author's Note

I'd run main.py and read the encoder-swap table before the prose. It holds the LLM at 7B and the data fixed, so the SigLIP row against the CLIP row is the cleanest look at axis 1 in the file. It also shows something a single MMMU number hides: DINOv2 scores 65.0 on CV-Bench and only 52.0 on DocVQA, while CLIP lands at 56.0 and 70.0. Which encoder is better depends on which benchmark you care about.

The picker is the part to poke at. It filters on LLM size only, then sums the three benchmarks with per-task weights. Run it at a 10B budget with the ocr profile and again with agent, and Molmo-7B-D, Cambrian-1-8B, and Idefics2-8B come back in the same order both times. With only nine recipes in the table, the weights have little room to reorder anything. That is a limit of the toy, and it tells you more about the table than about OCR versus agents.

I'd also treat the axis-impact percentages with suspicion. The six weights in the code sum to 115, and the final line prints a note about rebasing to 100 without actually doing it. The body's 60/20/5/15 split is the Prismatic summary I'd trust more as a rough guide to what to ablate first.

Exercises

  1. Read MM1 Section 3.2. For a fixed 2B LLM at budget 50M images, which encoder wins? Would the answer flip at 13B LLM? Why?

  2. Cambrian-1 finds that concatenating DINOv2 + SigLIP outperforms either alone on vision-centric benchmarks but adds no signal on MMMU. Predict which benchmarks gain and which stay flat.

  3. Your target is a mobile UI agent on a 2B LLM. Pick encoder, connector, resolution, and data mix. Justify each choice with a specific ablation table.

  4. Molmo ships 4B and 72B models. The 4B is competitive with closed 7B VLMs; the 72B beats Llama-3.2-90B-Vision on 11/11 benchmarks. What does that tell you about the LLM-size plateau hypothesis?

  5. Design an ablation table to isolate data-mix quality from encoder quality on a 7B VLM. How many training runs minimum? Propose the four axis settings.

Key Terms

TermWhat people sayWhat it actually means
Ablation"Turning one knob"Training multiple runs that differ in exactly one design-space axis, holding everything else constant
Connector"Bridge" / "projector"Trainable module that maps vision encoder output into the LLM's token space (MLP, Q-Former, Perceiver)
Detailed human caption"Dense caption"A multi-sentence human-written description (typically 80-300 tokens) richer than a web alt text
Distillation"GPT-4V captions"Training data generated by a stronger proprietary VLM; convenient but prone to inherited hallucination
AnyRes / dynamic res"High-res path"Strategy to feed images larger than the encoder's native resolution via tiling or M-RoPE
Resolution ramp"Curriculum"Training schedule that starts low-resolution and increases, speeding alignment learning
Vision-centric bench"CV-Bench / BLINK"Evaluation that stresses fine-grained visual perception rather than language-heavy reasoning
PixMo"Molmo's data"Allen AI's 712K densely-captioned image dataset; human speech transcribed into dense captions

Further Reading

Code

Ships With