TAI BUI
Back to curriculum
BuildPython

LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model

Before LLaVA-OneVision (Li et al., August 2024) the open-VLM world split into separate lineages: LLaVA-1.5 for single images, multi-image models like Mantis and VILA, video models like Video-LLaVA and Video-LLaMA. Each won its own benchmark and did poorly on the others. LLaVA-OneVision argued that one curriculum could train a single model for all three scenarios, and that skills transferring between them (single-image skills exported to video, multi-image reasoning exported to single-image) beat the sum of specialists. The recipe is simple: a visual-token budget that stays roughly constant across scenarios, plus an explicit curriculum that moves from single-image to OneVision (multi-image) to video. This lesson reads the budget, the curriculum, and the emergent behaviors.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 05 (LLaVA), Phase 12 · 06 (any-resolution)

Learning Objectives

  • Design a visual-token budget that holds constant across single-image, multi-image, and video inputs.
  • Order a training curriculum that transfers skills from single-image to video without catastrophic forgetting.
  • Explain why a single model beats specialists at the same parameter count when curriculum is done right.
  • Name the three emergent capabilities reported by LLaVA-OneVision: multi-camera reasoning, set-of-mark prompting, iPhone-screenshot agent.

The Problem

Image, multi-image, and video stress a model in different ways.

Single-image wants high-resolution tokens (AnyRes, ~2880 visual tokens) to catch OCR and fine detail. Budget per sample: one image, 2880 tokens.

Multi-image wants several images at moderate resolution (~576 tokens each) so reasoning across images fits in context. Budget per sample: 4-8 images, 576 each, 2300-4600 tokens.

Video wants many frames at low resolution (~196 tokens per frame after pooling) to capture temporal dynamics. Budget per sample: 8-32 frames, 196 each, 1600-6200 tokens.

Train separate models and you pick one budget per model. Train one model and the budget has to scale across scenarios without blowing the context window.

Before OneVision, the default answer was to train one scenario and ignore the others. Video-LLaVA retrofitted video onto an image model with extra training stages. LLaVA-NeXT added multi-image support with tiling. Neither handled all three cleanly.

Warm-up0/3
1. In LLaVA, how do the vision encoder's patch features become input the language model can use?Warm-up
Pipeline: an image goes through a vision encoder, which produces a small grid of patch features; a dashed question-mark box sits between the patch features and the language model, which also receives a text prompt.
Typed schematic, not captured output.
2. A 384-pixel tile with 14-pixel patches gives a 27x27 patch grid. An AnyRes setup with 4 tiles plus 1 thumbnail produces how many patch tokens before any pooling?Warm-up
Five boxes in a row, four labelled tile and one labelled thumbnail, each marked with a 27x27 patch grid, and a dashed question-mark box for the total patch tokens before pooling.
Typed schematic. The 27x27 grid per view is the figure the stem gives.
3. A network is fine-tuned only on a new task and loses much of what it learned on its earlier task. What is this problem called?Warm-up
Two panels joined by an arrow labelled fine-tune on new task only. Before, the earlier-task skill bar is nearly full and the new-task bar nearly empty. After, the earlier-task bar is short and the new-task bar nearly full. A dashed box asks for the name of the problem.
Typed schematic. Bar lengths are illustrative, not measured.
Check: The Problem0/2
1. In LLaVA-OneVision, what is a scenario?Define
Four small photos and a question box enter a box labelled LLaVA-OneVision, one set of weights, with a bracket under the whole row labelled scenario with a question mark.
Typed schematic of one sample.
2. Why can't one fixed visual-token budget serve single-image, multi-image, and video equally well?Explain
Three rows of input, a single photo, four photos and a strip of ten video frames, each with an arrow into one box labelled LLM, one fixed budget.
Typed schematic. Image and frame counts are illustrative.

The Concept

The OneVision token budget

LLaVA-OneVision picks a unified visual-token budget of approximately 3000-4000 tokens per sample and allocates it differently per scenario:

  • Single image: AnyRes-9 (3x3 tiles + thumbnail), each tile at 384 with 729 patches, aggressive bilinear pooling 2x2 → 182 per tile. Total: 9 * 182 + 182 = 1820 tokens. Or AnyRes-4 at 729-per-tile = 2916 + 729.
  • Multi-image: each image at moderate resolution (384, no tiling), 729 tokens with no pooling. Budget 6 images → 4374 tokens.
  • Video: 32 frames at 384 resolution with aggressive 3x3 bilinear pool → 81 tokens per frame. Total: 32 * 81 = 2592 tokens.

Total tokens stay in a similar range across the three. The encoder produces different geometry per scenario, but the LLM consumes about the same budget each time, so its context cost is predictable.

The three-stage curriculum

LLaVA-OneVision trains in three stages:

  1. Single-image SFT (stage SI). All data is single-image-plus-text, with high-resolution AnyRes input. This teaches perception, OCR, and fine-grained understanding. It uses LLaVA-NeXT data plus OneVision-specific single-image data.
  2. OneVision SFT (stage OV). Mix single-image, multi-image, and video (uniformly sampled frames) under the unified token budget. This teaches the model to handle heterogeneous batch shapes. No weight reset: it continues from stage SI.
  3. Task transfer (stage TT). Continue with a target task mix, typically heavier on multi-image or video depending on the product. Optional fine-tune for deployment.

Order matters. Training video-first or multi-image-first gives worse image performance than single-image-first on the same data, and the paper ablates this explicitly.

Why curriculum works

Single-image training builds the perceptual base. Patch tokens carry fine-grained visual features, and the LLM learns to integrate them with text. Multi-image and video add structural problems (which image is which, what happened first) that are hard to learn without that base.

Train all scenarios from scratch together and the model underfits perception, because each batch holds little single-image data, while it overfits structure on the abundant multi-image and video data. You get a model that follows cross-image reasoning patterns but is visually shallow.

Curriculum ordering gives you perception from stage SI, then compositional and temporal reasoning from stage OV, with neither one crowding out the other.

Emergent cross-scenario skills

The LLaVA-OneVision paper reports three emergent capabilities:

  1. Multi-camera reasoning. Trained on multi-image and video separately, the model is asked at inference to reason about a multi-camera driving scene. It integrates the views even though it never saw that exact format in training.
  2. Set-of-mark prompting. The user annotates objects in an image with numbered marks, and the model reasons about "what is mark 3 doing relative to mark 7." It was trained on neither marks nor annotation, and picked the skill up from spatial grounding combined with multi-image reference.
  3. iPhone-screenshot agent. The user provides a screenshot of an iPhone screen and asks the model to plan the next click. Training covered UI screenshots, video of user workflows, and multi-image before/after pairs, and the skill generalizes to this agent use case.

None of these are trained tasks. They emerge from the curriculum's compositional structure.

Visual-token pooling

The token budget requires pooling. OneVision uses bilinear interpolation on the 2D patch grid: 24x24 = 576 patches becomes 12x12 = 144 (2x factor) or 8x8 = 64 (3x factor). Pooling happens in patch-grid space, not token space, to preserve locality.

The pooling factor per scenario is a hyperparameter. Less pooling means more tokens and a richer representation. More pooling means fewer tokens, so more frames or images fit.

LLaVA-OneVision-1.5

The 2025 follow-up (LLaVA-OneVision-1.5, arXiv 2509.23661) is "fully open" in training data, model weights, and code. It closes the gap with proprietary models on some benchmarks and makes the recipe reproducible. Same curriculum, more data, better base LLM, no architecture change.

Contrast with Qwen2.5-VL

Qwen2.5-VL (Lesson 12.09) makes different choices. It uses M-RoPE and dynamic FPS instead of fixed pooling, so its budget scales with the input: a 1-minute video uses more tokens than a 5-second video. LLaVA-OneVision fixes the budget and scales the pooling. Both work. One trades predictability for configurability, the other the reverse.

Check: The Concept0/24
1. What does the visual-token budget count?Define
A row of small squares in a box labelled LLM input and output for one sample: ten blue squares, four grey squares and three dim squares, with a dashed question-mark box asking what the budget counts.
Typed schematic. Square counts are illustrative.
2. What does bilinear pooling do in LLaVA-OneVision?Define
A 6x6 grid of patch tokens with an arrow labelled bilinear pooling to a dashed box asking how many tokens come out, then an arrow to the LLM.
Typed schematic. The 6x6 grid is illustrative.
3. What makes a capability an emergent skill in this lesson?Define
A box labelled training mix (single, multi-image, video) leads to a box labelled LLaVA-OneVision, then to a dashed box asking for the capability.
Typed schematic, not captured output.
4. What does AnyRes-9 describe?Define
A photo labelled one high-resolution image, an arrow to a box labelled AnyRes-9, and a dashed box asking what the encoder sees.
Typed schematic, not captured output.
5. What does task transfer mean for a unified model?Define
Single-image training data feeds a box labelled shared backbone, one set of weights, which leads to a dashed box labelled task transfer with a question mark.
Typed schematic, not captured output.
6. What are LLaVA-OneVision's three training stages, in order?Define
A base model box, an arrow to a dashed box asking for the three training stages in order, and an arrow to a box labelled OneVision, trained model.
Typed schematic, not captured output.
7. Which of these is NOT one of the three emergent capabilities reported for LLaVA-OneVision?Define
A box labelled LLaVA-OneVision with an arrow to three dashed question-mark boxes standing for its reported emergent capabilities.
Typed schematic, not captured output.
8. Why does OneVision keep total visual tokens in a similar range across the three scenarios?Explain
Three horizontal bars of nearly equal length for single image, multi-image and video, each with an arrow into a box labelled LLM, and a dashed box asking why the totals are kept similar.
Typed schematic. Bar lengths show the stem's premise of similar totals and are illustrative.
9. How does LLaVA-OneVision get from a 27x27 patch grid per video frame down to 81 tokens per frame?Explain
A 27x27 grid of patch cells, an arrow to a dashed question-mark box, then an arrow to a box saying the LLM sees 81 tokens per frame.
The 27x27 grid is drawn to scale in cells.
10. What does stage OV do with the weights produced by stage SI?Explain
A box labelled stage SI with a weights chip, a dashed arrow with a question mark to a box labelled stage OV that asks what happens to the weights.
Typed schematic, not captured output.
11. What goes wrong if all scenarios are trained together from scratch, with no single-image first stage?Explain
A random-initialization box leads to a box labelled one training phase, single plus multi-image plus video, then to a dashed box asking for the result.
Typed schematic, not captured output.
12. What does the paper's ablation show about training video-first or multi-image-first instead of single-image-first?Explain
Three rows labelled single-image first, video first and multi-image first, each followed by then the rest, same data, and a dashed box asking for the image benchmarks.
Typed schematic, not captured output.
13. Why can one OneVision model beat separate specialists at the same parameter count?Explain
Three separate specialist boxes for single-image, multi-image and video on the left, versus one large OneVision model box handling all three, with a dashed box asking why it can beat the specialists.
Typed schematic, not captured output.
14. Set-of-mark prompting was trained on neither marks nor annotation. Which combination does the lesson say it was picked up from?Explain
A drawn scene with two objects marked 3 and 7, the prompt what is mark 3 doing relative to mark 7, a box for LLaVA-OneVision, and a dashed box asking what the skill was picked up from.
Typed scene for illustration. The prompt wording is the lesson's.
15. How does OneVision's handling of variable input size differ from Qwen2.5-VL's?Explain
An input box labelled images or video with arrows to two model boxes, OneVision and Qwen2.5-VL, each followed by a dashed box asking for the visual tokens.
Typed schematic, not captured output.
16. What does LLaVA-OneVision-1.5 change relative to the original recipe?Explain
A box for LLaVA-OneVision, arXiv 2408.03326, a dashed arrow labelled what changed to a box for LLaVA-OneVision-1.5, arXiv 2509.23661.
Typed schematic. Arxiv ids are from the lesson's references.
17. How many visual tokens does single-image AnyRes-9 use in the body's budget list, and how many does a 6-image multi-image sample use with no pooling?Apply
Two boxes: single image with AnyRes-9, and multi-image with no pooling, each asking for the number of visual tokens, with six small image rectangles under the second.
Typed schematic, not captured output.
18. A 24x24 patch grid is pooled with a 3x factor on the patch grid. How many tokens remain?Apply
A 24x24 grid of patch cells, an arrow labelled pool on the patch grid, and a dashed box asking how many tokens remain.
The 24x24 grid is drawn to scale in cells.
19. A team trains a VLM on a mix of all three scenarios from the first step. It follows cross-image reasoning patterns but misreads small text in single images. Which ingredient is missing?Analyze
A training box mixing single, multi-image and video from the start leads to a trained model; two boxes below say it follows cross-image patterns but misreads small text, and a dashed box asks for the missing ingredient.
Typed schematic, not captured output.
20. A teammate flattens the patches into a sequence and averages every 4 consecutive tokens to meet the budget. What is wrong with this compared with OneVision's approach?Analyze
A 6x6 grid of patches numbered 1 to 36, an arrow labelled read row by row, and a strip of the first 16 flattened tokens with a line under each group of four consecutive tokens.
Typed schematic. The 6x6 grid is illustrative.
21. A serving team sees LLM cost grow with video length, so a 1-minute clip costs far more than a 5-second clip. Which design choice explains this?Analyze
A bar chart of LLM cost per request: a short bar for a 5-second clip and a tall bar for a 1-minute clip, with a dashed box asking for the design choice behind it.
Typed schematic. Bar heights are illustrative; the stem gives only that the longer clip costs far more.
22. Your product compares 6 photos per request under a 4096-token budget. Unpooled at 729 tokens each, the images cost 4374. What is the most direct fix?Design
A budget bar with a marker at the 4096-token limit, six image tiles each marked 729, a line saying unpooled they cost 4374, and a dashed box asking for the most direct fix.
Typed schematic. The figures 729, 4374 and 4096 are the stem's; the bar is scaled so its full width is 4374.
23. Your product is 80% single-image, 10% multi-image with 2-4 images, and 10% video with 8-16 frames, under a 4096-token budget. Where should the tokens saved on multi-image go?Design
Three bars showing traffic share: single-image 80 percent, multi-image 10 percent with 2 to 4 images, video 10 percent with 8 to 16 frames, and a dashed box asking where the saved tokens go.
Typed schematic. Shares are the stem's.
24. You fine-tune a OneVision-style model for a product that is mostly video. Which curriculum fits?Design
A bar showing a mostly-video traffic share and a dashed box asking which curriculum fits.
Typed schematic. The bar length is illustrative.

Use It

code/main.py is a curriculum and budget planner for a OneVision-style VLM. Give it a token budget per sample and a target scenario mix (say 40% single-image, 30% multi-image, 30% video) and it:

  • Allocates resolution, pooling factor, and frames per scenario.
  • Checks that each scenario fits within the shared budget.
  • Reports expected token count, LLM FLOPs, and which scenarios are under-tokenized.
  • Prints a stage-by-stage training schedule.

Use it to plan a OneVision fine-tune or to sanity-check the per-request cost of a VLM deployment.

Check: Use It0/4
1. In main.py, plan_single_image is called with a budget of 1500. The loop tries 9 tiles first, with (res, patch, pool) options (384, 14, 2), (384, 14, 1), (336, 14, 2) in that order. What does it return?Apply
A code panel with the per_tile_tokens and anyres_tokens helpers and the plan_single_image search loop from main.py, ending with the call plan_single_image(1500) and a question mark for its result.
Real source lines from main.py; the return dict and the fallback return are elided.
2. In main.py, plan_video is called with a budget of 2000. It tries 32, 16, then 8 frames, and for each tries (384, 3), (384, 2), (336, 2). What does it return?Apply
A code panel with the per_tile_tokens helper and the plan_video search loop from main.py, ending with the call plan_video(2000) and a question mark for its result.
Real source lines from main.py; the return dict and the fallback return are elided.
3. In main.py, plan_multi_image is called with a budget of 1000. It tries 8, 6, 4, then 2 images, and for each tries (384, 2), (384, 1), (336, 2). What does it return?Apply
A code panel with the per_tile_tokens helper and the plan_multi_image search loop from main.py, ending with the call plan_multi_image(1000) and a question mark for its result.
Real source lines from main.py; the return dict and the fallback return are elided.
4. A teammate points to the planner's printed line "video first underperforms by 2-4 MMMU" as proof from your run. What is wrong with that?Analyze
A terminal panel showing one line printed by main.py, between omitted output: ordering matters, stages in reverse (video first) underperform by 2-4 MMMU per LLaVA-OneVision ablation.
Real line from a run of main.py, wrapped to fit; the output before and after it is omitted.

Ship It

This lesson produces outputs/skill-onevision-budget-planner.md. Given a target task distribution and a per-sample budget, it emits the AnyRes factor, per-frame pooling, video frame count, and curriculum stage weights. Use it when you train or fine-tune a unified-scenario VLM.

Author's Note

I'd run main.py first and read the output against the numbers above. With a 4096 budget it picks AnyRes-9 at 2x pooling for single-image (1690 tokens), 8 images for multi-image (1352), and 32 frames at 3x pooling for video (2592). Video lands at the 81-tokens-per-frame figure from the video bullet. The other two come out lower than the 1820 and 4374 in the budget list, because the planner floors the 27x27 patch grid when it pools (13x13 = 169 per tile) and takes the first option that fits, which is 8 images rather than 6.

The planner also says the target is a spread under 30% of the budget, and its own run reports 30.3%. I'd leave that alone and look at why: the multi-image plan stops at 8 images and uses a third of the budget, while video uses most of it. If you want the three closer together, the lever is the candidate lists in plan_multi_image, not the budget.

The curriculum section is the part I'd trust least from the code alone. The stage weights for SI and OV are hardcoded (1.0/0/0 and 0.5/0.3/0.2), and only the TT row reads your mix. The "2-4 MMMU" ordering penalty is a string it prints, not something it computes. So treat the table as a starting point for your own ablation, and check the ordering claim on your data before relying on it.

Exercises

  1. Your product supports 80% single-image, 10% multi-image (2-4 images), 10% video (8-16 frames). Design the token budget. Where would you put the extra budget you save from not doing heavy multi-image?

  2. Read LLaVA-OneVision Section 4.3 (emergent capabilities). Propose a fourth emergent skill the curriculum would likely unlock but the paper did not report.

  3. Swap the curriculum order: train multi-image first, then single-image, then video. Predict which benchmarks degrade and why.

  4. The paper reports video benchmarks trained on only 8 frames per sample. Does that generalize to 30-second videos at inference? What breaks first, the token budget or the temporal reasoning?

  5. Bilinear pooling of 24x24 patches to 12x12 is a 4x reduction per dim. Implement the pooling in stdlib Python and verify that the mean over each 2x2 block matches the bilinear output.

Key Terms

TermWhat people sayWhat it actually means
OneVision scenario"Single-image, multi-image, or video"One of three input shapes the unified VLM handles; the budget stays constant across
Token budget"How many tokens per sample"Total visual tokens the LLM sees per training / inference sample, typically 3000-4000
Curriculum"Training order"Stage ordering (single-image → multi-image → video) chosen for emergent transfer
Bilinear pooling"Token shrink"Applying bilinear interpolation to the patch grid (2D) to reduce token count while preserving locality
Emergent skill"Not trained, still works"Capability that appears at inference without matching training data, due to curriculum composition
AnyRes-k"k-tile setup"k sub-tiles of fixed resolution plus one thumbnail, typical k ∈ {4, 9}
Task transfer"Cross-scenario generalization"Skills learned on single-image that apply to video (and vice versa) via shared backbone

Further Reading

Code

Ships With