TAI BUI
Back to curriculum
BuildPython

Any-Resolution Vision: Patch-n'-Pack and NaFlex

Real images are not 224x224 squares. A receipt is 9:16, a chart is 16:9, a medical scan might be 4096x4096, a mobile screenshot is 9:19.5. The pre-2024 VLM answer was to resize everything to a fixed square, which threw away the signal that makes OCR, document understanding, and high-resolution scene parsing work. NaViT (Google, 2023) showed you could pack variable-resolution patches into a single transformer batch with block-diagonal masking. Qwen2-VL's M-RoPE (2024) dropped absolute positional tables entirely. LLaVA-NeXT's AnyRes tiled high-resolution images into a base plus sub-images. SigLIP 2's NaFlex variant (2025) is now the default encoder for open VLMs that want a single checkpoint to serve every aspect ratio. This lesson implements patch-n'-pack end to end.

TIME: ~120 minutesPREREQUISITES: Phase 12 · 01 (ViT patches), Phase 12 · 05 (LLaVA)

Learning Objectives

  • Pack patches from a batch of variable-resolution images into one sequence and build the block-diagonal attention mask.
  • Pick between AnyRes tiling (LLaVA-NeXT), NaFlex (SigLIP 2), and M-RoPE (Qwen2-VL) for a given task.
  • Compute token budgets for OCR, charts, and photography without resizing.
  • Name the three failure modes of square-resize: squished text, cropped content, wasted tokens on padding.

The Problem

Transformers expect a sequence. A batch is a stack of sequences the same length. If your images are 224x224, you get 196 patch tokens every time, no padding needed. Train on 224, infer on 224, and resolution stops being your problem.

The world does not cooperate with this. Documents are portrait (8.5x11 inches, 2:3-ish). Chart screenshots are landscape (16:9). Receipts are tall and thin (1:3). Medical imaging ships at 2048x2048 or larger. Mobile device screenshots are 1170x2532 (0.46:1).

Three pre-2024 options, and why each one fails:

  1. Resize to a fixed square (224x224 or 336x336). The squish distorts text and faces, and the downscale wipes out chart labels and OCR content. This was standard practice until LLaVA-1.5.
  2. Crop to a fixed aspect ratio. You throw away most of the image, and choosing where to crop is its own vision problem.
  3. Pad to the longest side. This fixes the distortion but can spend 50% or more of the tokens on padding for portrait images, and attention cost grows quadratically in all those pad tokens.

The 2024-2025 answer is to let the transformer take patches at the image's native resolution, then pack a mixed batch into one sequence without spending compute on padding.

Warm-up0/3
1. A ViT splits a 224x224 image into 16x16 pixel patches. How many patch tokens does each image produce?Warm-up
A square image labelled 224 by 224 pixels with one 16 by 16 pixel patch outlined in its top-left corner, drawn to scale, feeding a ViT box whose output token count is a question mark.
Schematic. The outlined patch is drawn to scale; the rest of the image is not divided.
2. LLaVA-1.5 runs a CLIP encoder at 336x336 with 14-pixel patches. How many visual tokens does one image give the language model?Warm-up
A square 336 by 336 pixel image with one 14 pixel patch outlined to scale, passing through a CLIP box into a language model box, with the number of visual tokens per image shown as a question mark.
Schematic. The outlined patch is drawn to scale; the image is not divided.
3. A transformer must stack token sequences of different lengths into one batch tensor. What does a standard pipeline do?Warm-up
Three rows of token cells with different lengths, stacked on the left and heading toward a single batch tensor of shape batch, length, dim, whose common row length is marked with a question mark.
Typed schematic. The lengths 5, 3 and 7 are illustrative.
Check: The Problem0/3
1. Which failure of square-resize affects text and faces most directly?Explain
A tall portrait document page with a face and lines of text, an arrow labelled resize to 336 by 336, and a dashed square holding a question mark for the result.
Schematic.
2. Why is padding a portrait image up to its longest side a poor fix for square-resize distortion?Explain
A narrow portrait receipt, 600 by 1500 pixels, centered inside a dashed square whose side equals the receipt's long side, representing the padded encoder input.
Schematic; the receipt and the dashed square are drawn to scale. 600 x 1500 is the receipt in main.py's workload.
3. A pipeline pads a batch of portrait receipts to their longest side. Text reads fine, but attention cost is far above what the content justifies. Which failure mode is this, and what removes it?Analyze
Three portrait receipts, each centered in a dashed square canvas whose side is the receipt's long side, with notes that text reads fine and attention cost is far too high, and a question mark for the failure mode and fix.
Typed schematic.

The Concept

NaViT and patch-n'-pack

NaViT (Dehghani et al., 2023) showed this works at scale. The idea is mechanical:

  1. For each image in the batch, compute its native patch grid at a chosen patch size (say 14).
  2. Flatten each image's patches into its own variable-length sequence.
  3. Concatenate all images' patches into one long sequence for the batch.
  4. Build a block-diagonal attention mask so image A's patches only attend within image A.
  5. Carry per-patch position information (2D RoPE or fractional position embeddings).

A batch of three images at 336x336 (576 tokens), 224x224 (256 tokens), and 448x336 (768 tokens) becomes one 1600-token sequence with a 1600x1600 block-diagonal mask. There is no padding, and the transformer handles arbitrary aspect ratios.

NaViT also introduced fractional patch dropping during training: drop 50% of patches at random across the batch, which regularizes and speeds up training. SigLIP 2 inherited this.

AnyRes (LLaVA-NeXT)

LLaVA-NeXT's AnyRes is the pragmatic alternative. Given a high-resolution image and a fixed encoder (CLIP or SigLIP at 336), tile the image:

  1. Pick a grid layout from a predefined set, such as (1x1), (1x2), (2x1), (1x3), (3x1), or (2x2), that best fits the image's aspect ratio.
  2. Tile the full image into the grid; each tile becomes a 336x336 crop.
  3. Also produce a thumbnail: the whole image resized to 336x336 as a global-context token.
  4. Encode every tile through the frozen 336-encoder. Concatenate the tile tokens + thumbnail tokens.

For a 672x672 image at 2x2 grid plus thumbnail: 4 * 576 + 576 = 2880 visual tokens. Expensive, but the LLM sees both local detail and global context. (The picker in main.py scores only aspect ratio, so it would give this square image a 1x1 grid and 1,152 tokens. LLaVA-NeXT's own selection also weighs resolution, which is what produces the 2x2 layout.)

Reach for AnyRes when your encoder is frozen and only supports one resolution. The cost is token count on large images: a 1344x1344 image at 4x4 grid is 9216 + 576 ≈ 9800 tokens, which fills most of an 8k LLM context.

M-RoPE (Qwen2-VL)

Qwen2-VL introduced Multimodal Rotary Position Embedding. Instead of NaViT's fractional positions or AnyRes's tile-and-thumbnail, each patch carries a 3D position (temporal, height, width). The query/key rotations handle arbitrary H, W, and temporal length.

M-RoPE ships native dynamic resolution without retraining. At inference you feed any HxW image and the patch embedder produces H/14 x W/14 tokens. Each token gets its (t=0, r=row, c=col) position, and RoPE rotates attention with the right frequencies. Qwen2.5-VL and Qwen3-VL continue this. InternVL3's V2PE is the same idea with variable encoding per modality.

Unlike AnyRes, M-RoPE costs O(H x W / P^2) tokens at native resolution, with no multiplicative tile overhead. Unlike NaViT, it still expects a single image per forward. Batching across resolutions still needs patch-n'-pack on top.

NaFlex (SigLIP 2)

NaFlex is the SigLIP 2 checkpoint's native-flex mode. A single model serves multiple sequence lengths (256, 729, 1024 tokens) at inference. Internally it uses NaViT-style patch-n'-pack during training and absolute fractional positions per patch. The selling point is one checkpoint where you pick the token budget at inference based on the task.

For a semantic task (classification, retrieval), 256 tokens. For OCR or chart understanding, 1024 tokens. No retraining.

The packing mask

The block-diagonal mask is where implementations tend to stumble. For a packed sequence of length N_total covering images i=0..B-1 with lengths n_i, the mask M of shape (N_total, N_total) is 1 if both indices fall in the same image's block, else 0. You can build it from a cumulative length list:

offsets = [0, n_0, n_0+n_1, ..., N_total]
M[i, j] = 1 iff there exists b where offsets[b] <= i < offsets[b+1] and offsets[b] <= j < offsets[b+1]

This is one line in PyTorch with torch.block_diag or an explicit gather. FlashAttention's variable-length path (cu_seqlens) skips the mask entirely and attends within sequences using the cumulative-length tensor directly, which is ~10x faster than a dense mask for typical batches.

Token budgets

Pick your strategy by task:

  • OCR / documents: 1024-4096 tokens. SigLIP 2 NaFlex at 1024, or AnyRes 3x3 + thumbnail.
  • Charts and UI: 729-1024 tokens at 384-448 native. Qwen2.5-VL dynamic resolution with max pixels cap.
  • Natural photos: 256-576 tokens is usually enough for the downstream LLM. Spend tokens where content density is high.
  • Video: 64-128 tokens per frame after spatial pooling, 2-8 FPS. Lesson 12.17 covers this.

The 2026 production rule: pick a per-task max-pixels cap, encode at native aspect ratio up to that cap, pack the batch, and skip padding. Qwen2.5-VL exposes min_pixels and max_pixels for exactly this knob.

Check: The Concept0/24
1. What is patch-n'-pack?Define
Two images of different shapes, one with a 4 by 3 patch grid and one with a 2 by 5 patch grid, entering a patch-n'-pack box whose output is a question mark.
Typed schematic with illustrative grids (4 x 3 and 2 x 5).
2. What is the block-diagonal attention mask in a packed batch?Define
An empty 7 by 7 grid with rows and columns labelled A1 to A4 and B1 to B3, the attention mask M for a packed sequence of two images, with its cell contents marked by a question mark.
Typed schematic. The 4 and 3 patch counts are illustrative.
3. What is AnyRes in LLaVA-NeXT?Define
A wide high-resolution scene image on the left, an arrow into a box labelled AnyRes in LLaVA-NeXT, and an arrow to a question mark for what comes out of it.
Schematic.
4. What is M-RoPE?Define
A scene image divided into a 4 by 3 grid of patches with one patch outlined and labelled p, and an arrow to a box labelled M-RoPE applied to patch p, followed by a question mark.
Typed schematic. The 4 x 3 grid is illustrative.
5. What can a single SigLIP 2 NaFlex checkpoint do that a fixed-resolution model cannot?Define
Two boxes side by side, a fixed-resolution model and a SigLIP 2 NaFlex model, each one checkpoint, with an arrow from the NaFlex box to a question mark for what it can do that the fixed model cannot.
Schematic.
6. What is cu_seqlens in a packed batch?Define
A single row of nine token cells made of three groups with 3, 2 and 4 cells for images 1, 2 and 3, with the value of cu_seqlens shown as a question mark.
Typed schematic. The lengths 3, 2 and 4 are illustrative.
7. What do Qwen2.5-VL's min_pixels and max_pixels control?Define
A scene image going into a Qwen2.5-VL request box that lists min_pixels and max_pixels with unknown values, followed by an arrow to a question mark for what they control.
Typed schematic; no real values are shown.
8. What is a visual token budget?Define
A scene image going into a vision-language model box that shows a visual token budget with an unknown value, followed by a question mark.
Typed schematic; no value is shown.
9. Why does a packed batch need the block-diagonal mask?Explain
A receipt image A and a scene image B packed into one row of seven patch cells, A1 to A4 then B1 to B3, with patch A2 outlined as the query and a question mark for the keys it can see.
Typed schematic; the patch counts are illustrative.
10. What training technique did NaViT introduce alongside patch-n'-pack that SigLIP 2 later inherited?Explain
A row of twelve packed patch cells, then a NaViT box marked patch-n'-pack plus a question mark, with an arrow to a SigLIP 2 box that inherited the same unknown technique.
Schematic; the patch count is illustrative.
11. Why does AnyRes add a thumbnail of the whole image to its tile tokens?Explain
A scene image split into four tiles plus a separate whole-image thumbnail with an accent border, a bracket around the tiles and the thumbnail with an arrow into an LLM box, and the question of why the thumbnail is added.
Typed schematic.
12. Why does M-RoPE avoid the multiplicative tile overhead that AnyRes has?Explain
Top row: an image split into four tiles plus a thumbnail through a fixed encoder, with tokens multiplied. Bottom row: the same image at native size through a patch embedder, with its token count a question mark.
Typed schematic.
13. What sets Qwen2-VL's M-RoPE apart from NaViT-style patch-n'-pack?Explain
Two panels. The left is headed NaViT-style patch-n'-pack with its arXiv reference. The right is headed Qwen2-VL M-RoPE with its arXiv reference and holds a question mark for what sets it apart.
Schematic; the references come from the lesson's reading list.
14. Why does FlashAttention's cu_seqlens varlen path beat a dense block-diagonal mask on packed batches?Explain
A packed batch of three images feeding two attention routes side by side, a dense block-diagonal mask and the varlen cu_seqlens path, each with its relative speed marked by a question mark.
Typed schematic.
15. In NaViT's patch-n'-pack, how are images of 576, 256, and 768 patches combined into one batch?Apply
Three boxes with widths proportional to 576, 256 and 768 patches, a bracket under all three and an arrow down to a box asking how they form one batch for the transformer.
Schematic; box widths are proportional to the patch counts.
16. What does LLaVA-NeXT's AnyRes produce for a 672x672 image at a 2x2 grid, and what is the downside on very large images?Apply
A 672 by 672 image divided into a 2 by 2 grid, with a note that each tile is 336 by 336, and the total visual tokens and the downside on very large images marked with question marks.
Schematic.
17. A 448x336 image goes through Qwen2-VL's M-RoPE with 14-pixel patches. How many visual tokens does the patch embedder produce?Apply
A 448 by 336 pixel scene image with one 14 pixel patch outlined to scale, feeding a patch embedder box whose number of output tokens is a question mark.
Schematic. The outlined patch is drawn to scale; the image is not divided.
18. One NaFlex checkpoint serves several requests. Which request should get the 1024-token setting rather than 256?Apply
A SigLIP 2 NaFlex checkpoint box next to two bars, 256 tokens and 1024 tokens, the second four times as long, and the question of which request needs the larger setting.
Schematic; bar lengths are proportional to the token counts.
19. An AnyRes setup sends a 1344x1344 image as a 4x4 grid plus thumbnail to an 8k-context LLM, and almost no room is left for the question. What is the cause?Analyze
A 1344 by 1344 image divided into a 4 by 4 tile grid plus a thumbnail, feeding an LLM box marked 8k context, with almost no room left for the question and a question mark for the cause.
Schematic.
20. A team feeds mixed-resolution images to Qwen2-VL's M-RoPE and finds that each image still needs its own forward pass. What is missing?Analyze
Three images of different shapes, each with its own arrow into a separate forward pass box, and a question mark for what is missing.
Typed schematic.
21. A packed batch has 4,000 tokens, and a dense N x N block-diagonal mask makes attention slow and memory-heavy. Which change does the lesson point to?Analyze
A square N by N mask with five uneven blocks along its diagonal, N equals 4000 so 16,000,000 cells, noted as slow and memory-heavy, with a question mark for the change to make.
Typed schematic; the block sizes are illustrative.
22. Your VLM uses a vision encoder frozen at 336x336 that cannot be retrained, and users upload large, dense documents. Which design fits?Design
A vision encoder box marked frozen at 336 by 336 with a lock icon, a large page packed with small text lines, and a question mark for the design that fits.
Typed schematic.
23. One product must handle both photo classification and OCR from a single checkpoint, choosing the cost per request. Which design fits?Design
A scene photo for classification and a text document for OCR both feeding one checkpoint box, labelled cost chosen per request, with a question mark for the design that fits.
Typed schematic.
24. A product must read scanned invoices and also caption vacation photos, with one token budget per request. Which plan fits best?Design
A scanned invoice page and a vacation photo side by side, each followed by a question mark for its token budget.
Typed schematic.

Use It

main.py implements patch-n'-pack for a heterogeneous batch of images, working from (H, W) sizes and integer patch counts. It:

  • Takes a list of (H, W) image sizes.
  • Computes each image's patch sequence length at patch size 14.
  • Packs them into one sequence of total length sum(n_i).
  • Builds the block-diagonal attention mask (dense, for clarity).
  • Compares the packed cost vs square-resize and AnyRes tiling.
  • Prints a token budget table for a mixed batch (receipt, chart, screenshot, photo).

Run it. The numbers that drop out are the reason every 2026 open VLM uses patch-n'-pack.

Check: Use It0/4
1. In the toy batch of main.py, image A is 6x4 and image B is 4x8 at patch size 2. What does it print for cu_seqlens?Apply
Three lines copied from main.py that build image A of 6 by 4 and image B of 4 by 8 at patch size 2 and pack them, followed by the printed cu_seqlens line with its value hidden by a question mark.
Code lines copied from the lesson's main.py; the output is withheld here.
2. For the same toy batch (6 tokens and 8 tokens), how many cells of the 14x14 dense mask are 1?Apply
An empty 14 by 14 grid with the first 6 columns marked image A and the next 8 columns marked image B, and the number of cells equal to 1 shown as a question mark.
Schematic of the toy batch's dense mask; cell contents are withheld.
3. main.py's anyres_cost scores a grid as abs(image ratio - grid ratio) + 0.1 * (rows + columns). What does it return for a 672x672 image?Apply
An excerpt of the grid search loop in main.py that tries row and column counts from 1 to 3, skips more than 6 tiles, and scores each with the ratio difference plus 0.1 times rows plus columns, followed by the call for a 672 by 672 image whose result is a question mark.
Loop lines copied from the lesson's main.py; the call line is typed and its result is withheld.
4. What does main.py's square_cost(img, 336, 14) return for the 1170x2532 phone screenshot?Apply
A tall phone screenshot of 1170 by 2532 pixels beside the signature of square_cost from main.py with its body hidden and the result for this image marked with a question mark.
Signature copied from main.py; the body and result are withheld.

Ship It

This lesson produces outputs/skill-resolution-budget-planner.md. Given a mixed-aspect-ratio workload (OCR, charts, photos, video frames) and a total-token budget, it picks the right strategy (NaFlex, AnyRes, M-RoPE, or fixed-square) and emits a per-request configuration. Use this skill when you are sizing a VLM for a product, since it helps you catch a silent 10x token blowup before it eats your latency budget.

Author's Note

I'd run main.py first and read the toy batch before the budget table. Two tiny images, a 6x4 and a 4x8 at patch 2, make the packed length and the block-diagonal mask small enough to see in the terminal, and the cu_seqlens list printed next to it is the same offsets list the mask is built from.

The budget table is where I'd slow down. The native, square, and AnyRes columns are plain arithmetic on patch counts, and square-resize is a flat 576 tokens for every image no matter what it holds. My bet is that the gap between that flat 576 and the native column is the part worth staring at, because it shows where the token spend goes when the content is dense.

Keep the scope in mind, though. The code counts tokens. It does not encode patches, attach 2D or M-RoPE positions, or run attention, and the mask is a dense N x N list of lists, which is only sensible at toy size. For a real batch I'd use the cu_seqlens path, and I'd check the AnyRes grid picker before trusting its choices. It scores grids with a hand-written ratio-plus-penalty rule over 1 to 3 rows and columns (capped at 6 tiles), so it is a rough stand-in for LLaVA-NeXT's real layout selection.

Exercises

  1. A receipt is 600x1500 (1:2.5). At patch size 14, how many native-resolution tokens? How many after square-resize to 336? Which loses more OCR accuracy in practice?

  2. Build the block-diagonal mask for a batch of four images with lengths 256, 576, 729, 1024. Verify the attention matrix is 2585x2585 and has exactly 256^2 + 576^2 + 729^2 + 1024^2 non-zero entries.

  3. For a 1792x896 image at patch 14, compare: (a) square-resize to 336 then encode, (b) AnyRes 2x1 + thumbnail, (c) M-RoPE at native. Which uses fewest tokens? Which preserves most detail?

  4. Implement fractional patch dropping: given a packed sequence, drop 50% of tokens uniformly at random, and update the block-diagonal mask accordingly. Measure the mask's sparsity change.

  5. Read Section 3.2 of the Qwen2-VL paper (arXiv:2409.12191). Describe in two sentences what min_pixels and max_pixels control and why both bounds matter.

Key Terms

TermWhat people sayWhat it actually means
Patch-n'-pack"NaViT-style packing"Concatenate variable-length patch sequences from different images into one batch dimension
Block-diagonal mask"Packing mask"Attention mask that confines each image's patches to attend only to themselves, not neighbors in the pack
AnyRes"LLaVA-NeXT tiling"Split a high-res image into a grid of fixed-size tiles plus a global thumbnail; encode every tile with a fixed encoder
NaFlex"SigLIP 2 native-flex"Single SigLIP 2 checkpoint that serves 256/729/1024-token budgets at inference without retraining
M-RoPE"Multimodal RoPE"3D rotary position encoding (time, row, column) that handles arbitrary H, W, T without position tables
cu_seqlens"FlashAttention packing"Cumulative-length tensor the FlashAttention varlen path uses instead of a dense block-diagonal mask
min_pixels / max_pixels"Resolution bounds"Qwen2.5-VL per-request knobs capping token count on very small or very large inputs
Visual token budget"How many tokens per image"Rough count of patch tokens emitted per image; sets the LLM's prompt budget and attention cost

Further Reading

Code

Ships With