LearnPython

InternVL3: Native Multimodal Pretraining

Open VLMs before InternVL3 mostly followed one recipe: take a text LLM trained on trillions of text tokens, bolt on a vision encoder, then fine-tune the seams. It works, but it carries alignment debt. The LLM spent its pretraining budget on pure text, so it has to learn how visual tokens relate to its text reasoning after the fact, without forgetting the text. InternVL3 (Zhu et al., April 2025) rejects that approach with one pretraining run that interleaves text and multimodal data from step one. The 78B model matches Gemini 2.5 Pro on MMMU-Pro with open weights. This lesson reads the case for native pretraining and what changes when you make it.

TIME: ~120 minutesPREREQUISITES: Phase 12 · 05, Phase 12 · 07 (recipes)

Learning Objectives

Explain why post-hoc VLM training accumulates alignment debt, citing the three measurable symptoms (catastrophic forgetting, answer drift, visual-text inconsistency).
Describe InternVL3's native pretraining corpus mix and why the ratio of text : interleaved : caption matters.
Compare V2PE (variable visual position encoding) to Qwen2-VL's M-RoPE.
Name the Visual Resolution Router (ViR) and Decoupled Vision-Language (DvD) deployment optimizations.

The Problem

Post-hoc VLM training is the default. LLaVA, BLIP-2, Qwen-VL, and Idefics all take an already-pretrained LLM (Llama, Vicuna, Qwen, Mistral) and add vision. The training stages typically look like:

Frozen LLM + frozen vision encoder + trainable projector, trained on caption pairs to align embeddings.
Unfreeze LLM, train on instruction data (LLaVA-Instruct, ShareGPT4V).
Optional task-specific fine-tune.

Three symptoms of alignment debt show up:

Catastrophic forgetting. The post-hoc VLM forgets text-only skills. GSM8K scores drop 5-10 points. Hellaswag scores drop. Pure-text agents regress.
Answer drift. Small phrasings of the same visual question get different answers. The vision encoder connects to the LLM with weaker bindings than the LLM's own tokens.
Visual-text inconsistency. The VLM can describe an image correctly and then answer a question that contradicts its own description. Visual tokens do not take part in the LLM's internal consistency checks the way text does.

These symptoms are well documented. MM1.5 Section 4 quantifies them, and LLaVA-OneVision's ablations hint at them. Native pretraining is the proposed fix.

Warm-up0/3

1. What does it mean to freeze a layer during training?Warm-up

Six boxes in a row: input, layer 1, layer 2 marked frozen, layers 3 and 4 unmarked, and loss. Below, a line about training steps and a dashed box asking what happens to layer 2's weights. — Typed schematic, not captured. Layer names and count are illustrative.

2. A model is fine-tuned heavily on a narrow new dataset and then does worse on tasks it handled well before. What is the most likely cause?Warm-up

Typed schematic, not captured. Bar heights show direction only, there are no measured values.

3. Why does a transformer add positional encodings to its token inputs?Warm-up

A three-token sentence, the cat sat. Each token embedding has a plus sign and a position vector added to it (position 0, 1, 2), feeding a box of self-attention layers. A dashed box asks why the position vectors are added. — Typed schematic, not captured. Three-token sentence is illustrative.

Check: The Problem0/7

1. In a LLaVA-style post-hoc recipe, which stage order is standard?Define

A LLaVA-style model drawn as image, vision encoder, projector and LLM in a row, with two data boxes below (caption pairs, instruction data) and a dashed box asking for the standard training order. — Typed schematic, not captured. Names are the ones in the lesson's Problem section.

2. What is alignment debt in a VLM?Define

A terminal with the header and the alignment debt row of the post-hoc versus native table printed by main.py: alignment debt visible versus negligible. A dashed box asks what alignment debt is. — Captured from the lesson's main.py. The values are hardcoded strings in that file, not measurements.

3. Why does the first alignment stage of a post-hoc VLM train only the projector?Explain

Stage 1 drawn as a caption pair going through a vision encoder, a projector marked trained here, and an LLM, with a dashed box asking why only the projector is trained. — Typed schematic, not captured. Stage 1 setup as the stem and the lesson's Problem section describe it.

4. Which of these is NOT one of the three alignment-debt symptoms of post-hoc VLM training?Explain

A box for a post-hoc VLM (pretrained LLM, vision encoder, projector) with three arrows pointing to three dashed boxes, each labelled symptom with a question mark. — Typed schematic, not captured. The lesson says the debt shows up as three measurable symptoms.

5. What evidence shows that alignment debt is measurable rather than just a hypothesis?Explain

Two cards for InternVL3-8B (single-stage pretrain plus instruction tune) and Qwen2.5-VL-7B, with a dashed box asking what evidence makes alignment debt measurable. — Typed schematic, not captured. Model names and sizes come from the lesson's Single-stage vs multi-stage section.

6. A post-hoc VLM describes a chart correctly, then answers a follow-up in a way that contradicts its own description. Which symptom is this, and why does it happen?Analyze

A chat transcript: the user asks for a description of a chart, the model says revenue rises each year from 2021 to 2023, the user asks if revenue fell, and the model answers yes. A dashed box asks which symptom this is and why. — Typed example, not captured. The chart and numbers are illustrative.

7. After vision training, a post-hoc VLM drops 8 points on GSM8K and pure-text agents built on it regress. Which symptom is this, and which idea in the lesson addresses it?Analyze

Two bars for GSM8K before and after vision training, the second labelled GSM8K minus 8, with a card saying pure-text agents built on it regress, and a dashed box asking which symptom this is and which idea addresses it. — Typed schematic, not captured. Bars show direction and the stem's 8 points, with no absolute scores.

The Concept

Native multimodal pretraining

InternVL3 trains from scratch on a corpus that is native multimodal from step one. The mix is:

40% text-only data (FineWeb, Proof-Pile-2, etc.)
35% interleaved image-text data (OBELICS, MMC4-style)
20% paired image-caption data
5% video-text data

Vision tokens, text tokens, and cross-modal interactions feed the same loss from the first gradient step. There is no alignment pretraining, no projector freezing stage, and no catastrophic forgetting to recover from.

Training is a single stage for the base model. Instruction tuning follows, but the base model already treats visual tokens as first-class inputs.

V2PE (variable visual position encoding)

Qwen2-VL uses M-RoPE with a fixed axis allocation. InternVL3 introduces V2PE, where the position encoding varies per modality type (text, image, video) with learnable scaling. In practice:

Text tokens get 1D position (text index).
Image patches get 2D position (row, col).
Video frames get 3D position (time, row, col).

The three share the same RoPE frequency base, but the hidden-dim allocation per band is a learned parameter rather than a fixed split. That gives pretraining room to trade off temporal against spatial frequency resolution.

V2PE's ablation claim is 1-2 points on video benchmarks over M-RoPE at the same compute. Not a revolution, but cleaner.

Visual Resolution Router (ViR)

ViR is a deployment optimization. A photo with one object at low detail wastes tokens when encoded at 1280px native, so ViR uses a small classifier to predict the minimum resolution needed to answer the question before encoding.

The routing has three tiers: low-res (256 tokens), medium (576), high (2048+). For 60% of queries in production traffic, low or medium is sufficient. The net effect is 2-3x throughput at equal quality.

Decoupled Vision-Language deployment (DvD)

When you serve a large VLM, the vision encoder runs once per image, but the LLM runs autoregressively for each output token. The two have different bottlenecks: the encoder is bound by GPU memory bandwidth (conv + attention), the LLM by its KV cache. DvD splits them onto separate GPUs with streaming between.

For an 8B + 400M encoder model, DvD roughly doubles per-node throughput versus co-located serving.

Single-stage vs multi-stage quality

InternVL3's primary benchmark claim: at 78B params, match Gemini 2.5 Pro's MMMU-Pro. At 38B, match GPT-4o. At 8B, lead the open-8B leaderboard. All of that comes from a single-stage pretrain plus instruction-tune recipe.

The alignment-debt hypothesis is measurable. InternVL3-8B loses fewer text-benchmark points (MMLU, GSM8K) than Qwen2.5-VL-7B per unit of vision-benchmark gain. The model is more of a generalist because training was one piece, not two.

InternVL3.5 and InternVL-U

InternVL3.5 (August 2025) scales the recipe with the same native-pretrain approach, more data, and more params. MMMU improvements are incremental.

InternVL-U (2026) adds unified generation: image output via MMDiT heads on top of the same backbone. The "U" stands for "Understanding + generation," chasing Transfusion-style unified models (Lesson 12.13). One native-pretrain backbone supports both understanding and generation heads.

Trade-offs of native pretraining

Native pretraining is not free:

Compute. Training a new VLM from scratch costs about as much as training a text LLM, which means millions of GPU-hours. Post-hoc adaptation reuses existing LLM weights and saves most of that cost.
Data. Interleaved image-text corpora at scale are rare. OBELICS is 141M documents and MMC4 is 571M, while text alone ships at 15T tokens. Scarce multimodal pretraining data is a hard constraint.
Base-LLM reuse. Native pretraining gives up the option to drop in a new LLM later. Post-hoc lets you swap Llama-3.1 for Llama-4 by retraining only the adapter.

The bet InternVL3 makes is that alignment debt costs more than the lost reuse. The benchmarks back the claim, and the cost to produce keeps most labs from replicating it cheaply. Post-hoc VLMs will keep existing because they stay cheaper for most projects.

Check: The Concept0/19

1. What is native multimodal pretraining?Define

A two-column table with post-hoc examples LLaVA and Qwen-VL v1 on the left and native examples InternVL3, GPT-4o and Chameleon on the right, and a dashed box asking what native multimodal pretraining is. — Cells copied from the 'examples' row that main.py prints. In the real output the two cells run together on one line (the string is wider than the column), so the columns here are redrawn.

2. What is V2PE?Define

Typed schematic, not captured. Component placement is drawn from the lesson's description.

3. What does the Visual Resolution Router (ViR) do, and what gain is claimed for it?Define

A pipeline of boxes: image plus question, a box labelled ViR with a question mark, vision encoder, LLM, answer. A dashed box asks what ViR does and what gain is claimed. — Typed schematic, not captured. Order of components follows the lesson's Concept section.

4. What is an interleaved image-text corpus?Define

Four rows naming the data sources in InternVL3's mix: text-only (FineWeb, Proof-Pile-2), interleaved image-text highlighted with the label format question mark, paired image-caption, and video-text. — Names from the lesson's Concept section. Typed, not captured.

5. What is DvD (Decoupled Vision-Language deployment)?Define

Typed schematic, not captured. Sizes are the 8B + 400M encoder example in the lesson.

6. What does InternVL-U add to the native-pretrain backbone?Define

Text, image and video inputs go into a native-pretrain backbone that produces a text output. A dashed box below the backbone asks what InternVL-U adds. — Typed schematic, not captured. InternVL-U is dated 2026 in the lesson.

7. What corpus mix does InternVL3 train on from the first step?Explain

Typed schematic, not captured. Source names are the ones in the lesson's Concept section.

8. Why does InternVL3's mix keep 40% text-only data even though the goal is a vision-language model?Explain

A stacked bar with a text-only slice at 40% and the other sources at 60%, a goal line saying a vision-language model, and a dashed box asking why the 40% text-only slice is kept. — Typed schematic, not captured. The 60% is the remainder of 100%.

9. How does V2PE differ from Qwen2-VL's M-RoPE?Explain

A sentence fragment of three text tokens and four image patches with arrows to two cards, Qwen2-VL with M-RoPE and InternVL3 with V2PE, and a dashed box asking how V2PE differs from M-RoPE. — Typed schematic, not captured. Token strips are illustrative.

10. In V2PE, how are positions assigned to text, image and video tokens?Explain

Three kinds of input in one prompt: a row of text tokens, an image block cut into patches, and a row of three video frames over time. A dashed box asks how each kind of token gets a position. — Typed schematic, not captured. Prompt content is illustrative.

11. What does learning the hidden-dim allocation per band buy V2PE?Explain

A bar of three bands (band 1, band 2, band 3) with boundaries marked learned, and a dashed box asking what learning the allocation buys. — Typed schematic, not captured. Band widths are arbitrary.

12. Why does ViR classify a query before the image is encoded?Explain

Typed schematic, not captured. Boxes are not to scale in time.

13. Why does DvD put the vision encoder and the LLM on separate GPUs?Explain

Two model cards, a vision encoder of about 400M parameters and an LLM of 8B parameters, above two empty GPU cards named GPU 0 and GPU 1, with a dashed box asking why DvD splits the two across GPUs. — Typed schematic, not captured. Sizes are the 8B + 400M encoder example in the lesson.

14. Which cost does native pretraining impose compared with post-hoc adaptation?Explain

Two cards: native pretraining marked with alignment debt negligible, and a dashed card asking which cost against post-hoc. Below, a terminal with the header and alignment debt rows of the table main.py prints. — Captured from the lesson's main.py (hardcoded strings). Cards are a typed schematic.

15. Why can a post-hoc VLM swap its base LLM later while a native one cannot?Explain

Two cards, a post-hoc VLM and a native VLM, the post-hoc one with base LLM Llama-3.1 and the native one labelled trained natively. Under them, swap to Llama-4 is marked possible for post-hoc and not possible for native, and a dashed box asks why the difference. — Typed schematic, not captured. The swap example is from the lesson's Trade-offs section.

16. How does InternVL3.5 (August 2025) relate to InternVL3?Explain

Typed schematic, not captured. The arXiv number is from the lesson's Further Reading list.

17. A team freezes a text LLM, trains a projector on captions, then unfreezes the LLM for instruction data, and calls it native multimodal pretraining. What is wrong?Analyze

Typed schematic, not captured. The plan is the one in the stem.

18. A team has a small GPU budget, expects to move from Llama-3.1 to a newer base LLM within a year, and has little interleaved data. Which approach fits, and why?Design

Three constraint rows: a small GPU budget, an expected move from Llama-3.1 to a newer base LLM within a year, and little interleaved data, with a dashed box asking which approach fits and why. — Typed schematic, not captured. Constraints are the ones in the stem.

19. You already have a native-pretrained VLM backbone and now need image generation too, without training a second large model from scratch. Which design follows the lesson?Design

Three requirement rows: understands text, image and video inputs (have), generates images (need), and a budget that rules out a second large model from scratch, with a dashed box asking which design follows the lesson. — Typed schematic, not captured. Requirements are the ones in the stem.

Use It

code/main.py is a training-corpus mixer and ViR router simulator, stdlib only. It:

Takes a target corpus mix (%text, %interleaved, %caption, %video) and computes expected steps per modality.
Simulates ViR routing on a batch of queries (distribution: 50% low-detail, 30% medium, 20% high-detail) and reports average token count.
Reports DvD throughput estimates given encoder vs LLM FLOPs.
Prints a side-by-side of post-hoc vs native pretraining in params, compute, data, and expected alignment-debt symptoms.

Check: Use It0/8

1. What does interleaved data give a native model that caption pairs alone do not?Explain

Two panels: paired image-caption data drawn as one image with one caption, and interleaved image-text data with a question mark under the label adds. A dashed box asks what interleaved data gives that caption pairs alone do not. — Typed schematic, not captured. The caption pair is illustrative.

2. CorpusMix(40, 35, 20, 5) is normalized and steps(1_000_000) is called. How many video steps does it return?Apply

A terminal with the call CorpusMix(40, 35, 20, 5), normalize, and steps(1_000_000) for the video entry ending in a question mark, above the source of the steps method from main.py. — Method source captured from the lesson's main.py. The call lines are typed from the stem.

3. CorpusMix(80, 70, 40, 10) is normalized and steps(500_000) is called. How many text steps does it return?Apply

A terminal with the call CorpusMix(80, 70, 40, 10), normalize, and steps(500_000) for the text entry ending in a question mark, above the source of the normalize method from main.py. — Method source captured from the lesson's main.py. The call lines are typed from the stem.

4. vir_sim runs with tiers of 256, 576 and 2048 tokens at fractions 0.6, 0.3 and 0.1. What ratio does it report against the all-high-res baseline?Apply

A terminal with three RouterTier entries (256 tokens at 0.6, 576 at 0.3, 2048 at 0.1) and a call to vir_sim for the ratio ending in a question mark, above the source of vir_sim from main.py. — Function source captured from the lesson's main.py. The tier names and call lines are typed from the stem.

5. dvd_throughput is called with encoder_flops=1024, llm_flops=8 and llm_tokens=128. What speedup does it report?Apply

A terminal with a call to dvd_throughput with encoder_flops 1024, llm_flops 8 and llm_tokens 128 for the speedup ending in a question mark, above the source of dvd_throughput from main.py. — Function source captured from the lesson's main.py. The call lines are typed from the stem.

6. A team serves mostly scanned documents with OCR and sees a much smaller ViR gain than the toy's 2.9x. What best explains the gap?Analyze

Two bars for ViR throughput gain: the main.py toy at 2.9x and a team's traffic as a dashed shorter bar labelled much smaller with a question mark, under a line that the team serves mostly scanned documents with OCR. A dashed box asks what best explains the gap. — Typed schematic, not captured. Only the 2.9x is a stated value, the team's bar has no number.

7. The DvD toy reports about 1.3x while the prose claims roughly 2x. What explains the gap?Analyze

Two bars for the DvD speedup: the main.py toy at about 1.3x and the lesson's prose claim at roughly 2x, with a dashed box asking what explains the gap. — Typed schematic, not captured. Bar heights are proportional to 1.3 and 2.

8. Your target task is video-heavy and you want to raise the video share well above InternVL3's 5%. Which plan fits the lesson's reasoning?Design

Typed schematic, not captured. The 95% is the remainder of 100%.

Ship It

This lesson produces outputs/skill-native-vs-posthoc-auditor.md. Given a proposed VLM training plan, it audits whether to go native or post-hoc, flags alignment-debt risk, and recommends a corpus mix. Use it when you are sizing a new open-VLM project and need to pick a training strategy.

Author's Note

I'd run main.py before trusting the throughput numbers above. It is stdlib only and prints the corpus mix as step counts, so the 40/35/20/5 split becomes 200,000, 175,000, 100,000, and 25,000 steps out of 500,000. That is a useful way to feel what a 5% video share means in absolute terms.

The ViR simulation lands at roughly 710 average tokens per request against a 2048-token all-high-res baseline, about 2.9x. That is the top of the 2-3x range quoted above, and it depends on the 50/30/20 query mix the toy hardcodes. That mix sends 80% of traffic to low or medium tiers, more than the 60% quoted in the Concept section, so the toy flatters ViR. If your traffic skews toward documents and OCR, expect the ratio to shrink.

The DvD toy gives a smaller number than the prose: with 300 GFLOPs for the encoder and 8 GFLOPs per token over 128 tokens, it reports about 1.3x, not 2x. The LLM side dominates in that setup, so splitting the encoder off only hides a small share of the work. I'd read the roughly-2x claim as a statement about a balanced workload, and plug your own FLOPs in before believing it for your model.

One caution on the final table. Its GPU-hour and regression figures are hardcoded strings, not computed from anything, so treat them as a summary of the argument rather than measurements.

Exercises

Estimate the compute delta between InternVL3-8B (native pretrain) and LLaVA-OneVision-7B (post-hoc). Ratio of GPU-hours approximately? What explains the gap?
InternVL3 reports 40% text / 35% interleaved / 20% caption / 5% video. If your target task is video-heavy, propose a new ratio and argue why the base model still needs substantial text and caption data.
Read MM1.5 Section 4 on forgetting. Name the exact benchmark where post-hoc training showed the largest regression. How much did the regression cost?
ViR routes 60% of traffic to low-resolution encoding. What kinds of queries does it misroute (sends to low-res when high-res was needed)? Propose three router-failure modes.
DvD splits vision and LLM onto separate GPUs. Under what traffic pattern does DvD hurt throughput instead of helping?

Key Terms

Term	What people say	What it actually means
Native multimodal pretraining	"From scratch together"	Text + image + video tokens participate in the loss from step 1, not bolted on later
Alignment debt	"Post-hoc penalty"	Measurable regression in text skills and answer consistency that comes from bolting vision onto a frozen LLM
V2PE	"Variable visual pos encoding"	Per-modality learnable position encoding allocation; InternVL3's M-RoPE successor
ViR	"Resolution router"	Small classifier that picks minimum resolution needed per query before encoding, saving inference tokens
DvD	"Decoupled deployment"	Vision encoder on one GPU, LLM on another, with stream handoff; doubles throughput for large VLMs
InternVL-U	"Unified understanding + generation"	2026 follow-up that adds image-generation heads to the native-pretrain backbone
Interleaved corpus	"OBELICS / MMC4"	Documents with text and images in natural reading order; the raw material for native pretraining