LearnPython

Vision Transformers and the Patch-Token Primitive

Before anything multimodal, an image has to become a sequence of tokens a transformer can eat. The 2020 ViT paper did it with 16x16 pixel patches, a linear projection, and a position embedding. Five years later the frontier models (Claude Opus 4.7 at 2576px native, Gemini 3.1 Pro, Qwen3.5-Omni) still start there. The encoder moved from ViT to DINOv2 to SigLIP 2, register tokens showed up, and positions became 2D-RoPE, but the primitive held. This lesson walks the patch-token pipeline end to end and builds it in stdlib Python, so the rest of Phase 12 has a concrete picture of what "visual tokens" means.

TIME: ~120 minutesPREREQUISITES: Phase 7 (Transformers), Phase 4 (Computer Vision)

Learning Objectives

Convert an HxWx3 image into a sequence of patch tokens with correct positional encoding.
Compute sequence length, parameter count, and FLOPs for a ViT of a given (patch size, resolution, hidden dim, depth).
Name the three upgrades that took ViT from 2020 research to 2026 production: self-supervised pretraining (DINO / MAE), register tokens, and native-resolution packing.
Pick between CLS pooling, mean pooling, and register tokens for a downstream task.

The Problem

Transformers operate on sequences of vectors. Text is already a sequence (bytes or tokens). An image is a 2D grid of pixels with three color channels, which is not. Flatten every pixel and a 224x224 RGB image becomes 150,528 tokens. Self-attention is quadratic in sequence length, so that is a non-starter.

Pre-2020 approaches bolted a CNN feature extractor onto the front. ResNet produces a 7x7 feature map of 2048-dim vectors, and you feed those 49 tokens to a transformer. This works, but it inherits the CNN's biases (translation equivariance, local receptive fields) and loses the transformer's appetite for scale.

Dosovitskiy et al. (2020) asked the blunt question: what if we skip the CNN? Split the image into fixed-size patches (say 16x16 pixels), project each patch linearly into a vector, add a positional embedding, and feed the sequence to a vanilla transformer. At the time this was heresy, vision without convolutions. With enough data (JFT-300M, then LAION) it beat ResNet on ImageNet and kept improving.

By 2026 the ViT primitive is the foundation most vision work builds on. The vision tower of an open-weights VLM is some descendant of it (DINOv2, SigLIP 2, CLIP, EVA, InternViT). Nobody asks "should we use patches?" anymore. The live questions are patch size, resolution schedule, pretraining objective, and positional encoding.

Warm-up0/3

1. Why can't you just flatten a 224x224 RGB image into one token per pixel and feed it to a transformer?Warm-up

Schematic of the stem's setup: one token per pixel. Photo and token chips are typed.

2. Which inductive bias do convolutional networks build in that a vanilla transformer does not?Warm-up

An 8 by 8 photo-like input with a 3 by 3 window outlined, a 3 by 3 filter K, and a 6 by 6 output map with one cell highlighted. — Typed schematic of one convolution step with illustrative sizes.

3. Self-attention on its own treats its inputs as an unordered set. What do text transformers add so tokens can be told apart by position?Warm-up

Two rows of token chips: dog, bites, man and man, bites, dog. Both contain the same three tokens. A box labeled self-attention layer sits below them. A note says the two sentences mean different things. — Typed example sentences.

Check: The Problem0/1

1. Why did feeding ResNet feature maps into a transformer fall short of what scaling could offer?Explain

A photo goes into a ResNet box, which outputs a 7 by 7 map of 2048-dimensional vectors, which then goes into a transformer box. — Typed schematic of the lesson's pre-2020 recipe.

The Concept

Patches as tokens

Given an image x of shape (H, W, 3) and a patch size P, you carve the image into a grid of (H/P) x (W/P) non-overlapping patches. Each patch is a P x P x 3 cube of pixels. Flatten each cube to a 3 P^2 vector, then apply a shared linear projection W_E of shape (3 P^2, D) to map it into the model's hidden dimension D.

For the canonical ViT-B/16 config:

Resolution 224, patch size 16 → grid 14x14 → 196 patch tokens.
Each patch is 16 x 16 x 3 = 768 pixel values, projected to D = 768.
Add a learnable [CLS] token → sequence length 197.

The patch projection is mathematically identical to a 2D convolution with kernel size P, stride P, and D output channels. Production code implements it that way: nn.Conv2d(3, D, kernel_size=P, stride=P). Think of it as a linear projection, and write it as a convolution because that is the efficient form.

Positional embeddings

Patches have no inherent order, so the transformer sees them as a bag. Early ViTs added a learnable 1D positional embedding, one 768-dim vector per position, 197 of them. That works, but it ties the model to the training resolution: change the grid at inference and you have to interpolate the position table.

Modern vision backbones use 2D-RoPE (Qwen2-VL's M-RoPE, SigLIP 2's default) or factorized 2D positions. 2D-RoPE rotates the query and key vectors by the patch's (row, column) index, so the model reads relative 2D position off the rotation angle. There is no position table, and the model handles arbitrary grid sizes at inference.

CLS token, pooled output, and register tokens

What is the image-level representation? Three choices coexist:

[CLS] token. Prepend a learnable vector to the patch sequence. After all transformer blocks, the CLS token's hidden state is the image representation. Inherited from BERT, and used by the original ViT and CLIP.
Mean pool. Average the patch tokens' output hidden states. Used by SigLIP, DINOv2, and most modern VLMs.
Register tokens. Darcet et al. (2023) observed that ViTs trained without an explicit sink token develop high-norm "artifact" patches that hijack self-attention. Adding 4-16 learnable register tokens absorbs that load and improves dense-prediction quality (segmentation, depth). DINOv2 and SigLIP 2 both ship with registers.

The choice matters downstream. CLS is fine for classification. A VLM that feeds patch tokens into an LLM skips pooling entirely, and each patch becomes an LLM input token. Registers get discarded before handoff because they are scaffolding, not content.

Pretraining: supervised, contrastive, masked, self-distilled

The 2020 ViT was pretrained with supervised classification on JFT-300M. Other objectives quickly took over:

CLIP (2021): contrastive image-text on 400M pairs. Lesson 12.02.
MAE (2021, He et al.): mask 75% of patches and reconstruct pixels. Self-supervised, works on pure images.
DINO (2021) / DINOv2 (2023): student-teacher self-distillation with no labels and no captions. The 2023 DINOv2 ViT-g/14 is the strongest purely-visual backbone and the default for "dense features" use cases.
SigLIP / SigLIP 2 (2023, 2025): CLIP with a sigmoid loss, plus NaFlex for native aspect ratio. The dominant vision tower in 2026 open VLMs (Qwen, Idefics2, LLaVA-OneVision).

The objective decides what the backbone is good for. CLIP and SigLIP match images to text, DINOv2 gives dense visual features, and MAE is a starting point for finetuning.

Scaling laws

ViT scaling (Zhai et al. 2022) established that quality follows predictable laws in model size, data size, and compute. At fixed compute:

A bigger model with more data gives better quality.
Patch size trades sequence length against fidelity. Patch 14 (typical for DINOv2 and SigLIP SO400m) gives more tokens per image than patch 16, which helps OCR and dense tasks and costs speed.
Resolution is the other big lever. Going from 224 to 384 to 512 almost always helps, at quadratic cost in FLOPs.

ViT-g/14 (1B params, patch 14, resolution 224 → 256 tokens) and SigLIP SO400m/14 (400M params, patch 14) are the two workhorse encoders for 2026 open VLMs.

Parameter count for a ViT

The full calculation lives in code/main.py. For ViT-B/16 at 224:

patch_embed = 3 * 16 * 16 * 768 + 768  =  591k
cls + pos    = 768 + 197 * 768          =  152k
block        = 4 * 768^2 (QKVO) + 2 * 4 * 768^2 (MLP) + 2 * 2*768 (LN)
             = 12 * 768^2 + 3k          =  7.1M
12 blocks    = 85M
final LN    = 1.5k
total       ≈ 86M

Ballpark a ViT this way before you load the checkpoint. The backbone size sets your VRAM floor in any downstream VLM.

2026 production config

Most open VLMs in 2026 ship SigLIP 2 SO400m/14 at native resolution (NaFlex). It has:

400M parameters.
Patch size 14, default resolution 384 → 729 patch tokens per image.
Mean pool for image-level tasks, with all 729 patches flowing into the LLM for VQA.
4 register tokens, discarded before LLM handoff.
2D-RoPE with image-level scaling for native aspect ratio.

Each choice in that config traces back to a paper you can read.

Check: The Concept0/30

1. In a vision transformer, what is a patch?Define

Typed schematic; the magnified region is drawn from the same toy scene.

2. How is the patch 'linear projection' implemented in production code?Define

Typed schematic of the setup in the stem.

3. What is the [CLS] token in a ViT?Define

A photo cut into patches feeds an input row with a token labeled [CLS] followed by patch tokens p1, p2, p3 up to pN, then a box of transformer blocks. — Typed schematic.

4. According to Darcet et al. (2023), what problem do register tokens solve?Define

A photo cut into patches feeds a row of patch tokens p1, p2, p3 up to pN followed by four learnable register tokens r1 to r4, all entering transformer blocks. — Typed schematic of an input with 4 register tokens. Placing the registers after the patches is illustrative.

5. What does a position embedding give a ViT?Define

A 3 by 3 grid of patches labeled p1 to p9 beside a table with nine rows, pos 1 to pos 9, each row a strip of shaded cells. — Typed schematic of a learnable position table.

6. For a ViT input, what does 'grid' mean?Define

A photo 192 pixels wide and 128 pixels high, cut into squares by thin lines. Beside it the values H = 128, W = 192 and patch size P = 32, and the words grid equals question mark. — Typed schematic with illustrative sizes.

7. In a vision-language model, what is the backbone (the vision tower)?Define

A photo goes into a box labeled vision tower, which produces patch tokens, which go into a box labeled LLM that gives a text answer. A note under the vision tower says it is also called the backbone. — Typed schematic of a VLM.

8. In the context of a ViT, what is pooling?Define

A row of output tokens h1, h2, h3 up to hN goes into a highlighted box labeled pooling and then into a box labeled classifier head. — Typed schematic.

9. At the same input resolution, what is the tradeoff between patch size 14 and patch size 16?Define

Typed schematic: the outlined squares are one patch of each size on the same pixels.

10. What does NaFlex refer to in SigLIP 2?Define

A spec card for SigLIP 2 SO400m/14 listing 400M parameters and patch size 14, with the NaFlex line showing a question mark. A side box says most open VLMs in 2026 ship this tower. — Typed from the lesson's 2026 production config section.

11. What does MAE train a ViT to do?Define

A photo cut into a 4 by 4 grid goes into a ViT encoder, which leads to a box that reads trained to do, question mark. The heading names MAE, He et al., 2021. — Typed schematic.

12. Which method learns purely from images through student-teacher self-distillation, with no labels and no captions?Define

Typed schematic of the setup described in the stem.

13. Why do modern backbones like SigLIP 2 favor 2D-RoPE over the original ViT's learnable 1D positional embedding?Explain

Two panels. The original ViT has a learnable 1D position table with rows pos 0, pos 1, pos 2 up to pos N. SigLIP 2 uses 2D-RoPE, shown as a 3 by 4 grid of patches, each labeled with a row and column pair. — Typed schematic.

14. In the 2026 production SigLIP 2 SO400m/14 config, what happens to the register tokens when visual tokens are handed to the LLM?Explain

Typed schematic of the tower output and the handoff to the LLM. The order of the tokens is illustrative.

15. Which pooling strategy is correctly matched with the model family that defaults to it?Explain

Two rows of output tokens [CLS], p1 to p4. In the first the [CLS] output is highlighted and labeled as the one used. In the second the four patch outputs are highlighted and labeled as averaged. — Typed schematic of two pooling choices, without model names.

16. How does SigLIP differ from CLIP?Explain

An image and the typed caption a photo of a dog go through an image encoder and a text encoder into a box labeled CLIP's loss, contrastive. The heading says CLIP trains on 400M image-text pairs. — Typed schematic. The caption text is an example, not from the lesson.

17. Which pairing of backbone and strength is correct?Explain

Two task cards. Task 1 is labeling every region of a photo. Task 2 is finding the photo that matches the caption a red bicycle. — Typed examples.

18. Why does raising resolution from 224 to 384 to 512 'almost always help' but come with a cost?Explain

Three square photos of increasing size drawn to scale and labeled 224 by 224, 384 by 384 and 512 by 512, under a heading saying the same model is used for all three. — Typed schematic; squares drawn to the same scale.

19. ViT-g/14 and SigLIP SO400m/14 are the two workhorse encoders for 2026 open VLMs. What design choice do they share?Explain

Two cards. ViT-g/14 from DINOv2 has about 1B parameters, hidden size 1536 and depth 40. SigLIP SO400m/14 has 400M parameters, hidden size 1152 and depth 27. — Parameter counts are from the lesson text; hidden size and depth are from main.py's config zoo.

20. For the canonical ViT-B/16 config at 224x224 resolution, what is the total sequence length fed to the transformer?Apply

A 224 by 224 by 3 photo goes into a patch embedding box with patch size P equals 16, then into a ViT-B/16 transformer box, and the total sequence length fed to it is marked with a question mark. — Typed schematic.

21. When a P x P x 3 patch is flattened before the linear projection into hidden dimension D, how long is the flattened vector?Apply

Typed schematic.

22. Using the Concept section's ViT-B/16 parameter breakdown at 224 resolution, roughly how many parameters does the model have in total?Apply

A table of the lesson's ViT-B/16 breakdown: patch embedding 591k, cls token plus position embeddings 152k, 12 blocks at 7.1M each 85M, final LN 1.5k, and the total shown as a question mark. — Typed from the lesson's Parameter count section.

23. A ViT-B/16 takes a 448x448 image instead of 224x224. What is its sequence length with a CLS token?Apply

Two photos drawn to scale, 224 by 224 and 448 by 448, for the same ViT-B/16 with patch size 16. A note says a [CLS] token is added and the sequence length is a question mark. — Typed schematic; squares drawn to the same scale.

24. You are building a VLM that feeds vision-encoder patch tokens straight into an LLM for visual question answering. What should you do with the patch tokens?Apply

A photo goes into a vision encoder that outputs patch tokens p1 to pN. Below them a wide box for the LLM carries the typed question What color is the car. — Typed schematic; the question text is an example.

25. A DINO-style ViT without register tokens gives poor segmentation. A few patches have very high norm and attention concentrates on them. Which fix does the lesson point to?Analyze

An 8 by 8 heat map of output norm per patch in which three cells are bright orange and the rest dark blue, beside notes: no register tokens, a few patches have very high norm, attention concentrates on them, result is poor segmentation. — Typed schematic matching the stem; not measured data.

26. A ViT trained at 224 with a learnable 1D position table loses quality when run on 384x384 inputs at patch 16. Which diagnosis fits?Analyze

A learnable 1D position table with rows pos 0, pos 1 up to pos N, a smaller photo labeled train 224 and a larger photo labeled run 384, under a heading saying the model was trained at 224 and run at 384 with patch 16. — Typed schematic; photos drawn to scale.

27. A teammate picks a vision backbone because the calculator reports a small token count and parameter count, and concludes it will perform well. Which assumption is wrong?Analyze

A terminal with real main.py output for a small ViT-S/32 config at 224, listing image, patch size, grid, sequence length, hidden and depth, parameter counts and FLOPs. Beside it a box quotes a teammate: few tokens, few parameters, so it will perform well. — Output of main.py's print_config for a config typed for this picture (ViT-S/32 is not in the lesson's zoo); the parenthetical after the sequence length is trimmed. The quote is typed.

28. You need per-patch features from a frozen backbone for segmentation and depth, and you want clean, artifact-free patch outputs. Which setup fits best?Design

Typed schematic of the setup in the stem.

29. You are adding page reading (OCR) to a VLM, pages come in many aspect ratios, and context is limited. Which encoder design fits?Design

Three sketched pages with lines of text: a portrait page, a wide landscape page and a narrow tall page. A box says the LLM context budget is limited. — Typed sketches of different aspect ratios.

30. Your vision tower has 4 registers and no CLS token. You want one image embedding for retrieval and also want to feed the LLM. Which design fits?Design

The tower output: four register tokens r1 to r4 and patch outputs p1 to pN, with a note that there is no CLS token. Beside it two needs: retrieval needs one image embedding and the LLM needs visual tokens for VQA. — Typed schematic.

Use It

code/main.py is a patch tokenizer and geometry calculator. It takes (image H, W, patch P, hidden D, depth L) and reports:

Grid shape and sequence length after patching.
Token sequence for a synthetic 8x8 pixel toy image, walking through the flatten and project path.
Parameter count broken down by patch embed, position embed, transformer blocks, and head.
FLOPs per forward pass at the target resolution.
A comparison table across ViT-B/16 @ 224, ViT-L/14 @ 336, DINOv2 ViT-g/14 @ 224, SigLIP SO400m/14 @ 384, and Qwen2.5-VL ViT @ 896.

Run it and match the parameter counts to the published numbers. Then change the patch size and resolution to feel what they do to the token count.

Check: Use It0/7

1. In main.py, the SigLIP entry is ViTConfig(image_size=378, patch_size=14, registers=4, cls_token=False). What does seq_length return for it?Apply

An excerpt of main.py: the SigLIP config with image size 378, patch size 14, 4 registers and no CLS token, the four lines of seq_length, and a call seq_length(SIGLIP) whose result is a question mark. — The ViTConfig entry and seq_length lines are copied from main.py. The SIGLIP name and the call line are typed.

2. In main.py, the Qwen2.5-VL entry is ViTConfig(image_size=896, patch_size=14, hidden=1280) with the default registers and cls_token. What does seq_length return?Apply

An excerpt of main.py: the ViTConfig dataclass fields with the default values of registers and cls_token replaced by dots, the Qwen2.5-VL config with image size 896, patch size 14, hidden 1280, depth 32 and 16 heads, and a call seq_length(QWEN) whose result is a question mark. — The dataclass fields and the config values are copied from main.py, with the two defaults hidden (shown in the explanation). The QWEN name and the call line are typed.

3. What does main.py's grid_shape(384, 14) do?Apply

A terminal showing the signature of grid_shape with its body hidden, then the call grid_shape(384, 14), and a question mark where the result goes. — Signature copied from main.py; the body is hidden and the call is typed.

4. The toy walkthrough in main.py cuts an 8x8 image with patch_size=4 and projects each patch from 16 values to a hidden size of 4. What does it produce?Apply

An 8 by 8 grayscale image that brightens from top-left to bottom-right, beside the settings patch_size equals 4, each patch flattened to 16 values, projected to hidden size 4, and the output marked with a question mark. — The gray levels follow the toy image's pixel rule from main.py; no numbers are drawn.

5. In the toy image, pixel (row r, column c) holds r*8 + c, and patches are taken left to right, top to bottom. What are the first eight values of patch 1, the top-right 4x4 patch?Apply

Gray levels follow the pixel rule from main.py; no pixel values are drawn.

6. In flops_per_forward, the sequence length n doubles while the hidden size and depth stay fixed. How does the cost change?Apply

The code of flops_per_forward from main.py: attn equals 4*n*d*d plus 2*n*n*d, mlp equals 2*n*d*4*d*2, and the return multiplies depth by attn plus mlp. A last line says n becomes 2*n with d and depth fixed. — Code copied from main.py; the last line restates the stem.

7. A teammate gets a ValueError from the calculator for SigLIP at 384 with patch 14, and edits grid_shape to floor the division. What is wrong with that?Analyze

A terminal with the SigLIP SO400m/14 config at image size 384 and patch size 14, a call to seq_length that ends in a ValueError whose message is hidden, and a box saying a teammate wants to edit grid_shape to floor the division instead of raising. — Config line typed from the zoo entry with 378 changed to 384. The error message is left out here and shown in the explanation.

Ship It

This lesson produces outputs/skill-patch-geometry-reader.md. Give it a ViT config (patch size, resolution, hidden dim, depth) and it returns a token count, a parameter count, and a VRAM estimate with justifications. Reach for it when you pick a vision backbone for a VLM, since it catches the "tokens exploded and my LLM context filled up" surprise early.

Check: Ship It0/1

1. A VLM team swaps a ViT-B/16 @ 224 encoder for a patch-14 encoder at 896 and the LLM context now fills after a couple of images. Which cause fits?Analyze

Two context-window bars. Before, with a ViT-B/16 at 224, three small image blocks fill a small part of the bar. After, with a patch-14 encoder at 896, two large image blocks fill the whole bar. — Typed schematic, not to scale; no token counts are drawn.

Author's Note

I'd run main.py before reading the formulas. It starts with an 8x8 grayscale toy image, cuts it into four 4x4 patches, and projects each 16-pixel patch down to 4 numbers with a fixed fake matrix. That is the whole primitive in a few lines of list indexing, with no framework hiding the flatten step.

The part I'd poke at is seq_length. It adds one slot for a CLS token and one per register on top of the grid, and the position table is sized from that count, so the configs in the zoo show how those extra tokens cost real parameters. The zoo also marks SigLIP as cls_token=False with 4 registers, which is how the mean-pool-plus-registers setup from the Concept section looks in code.

One thing to watch: grid_shape raises if the image size isn't divisible by the patch size. That is why the SigLIP entry runs at 378 instead of 384, since 384 doesn't divide by 14. The prose rounds to 384, so trust the code when you check the numbers.

I'd also keep the calculator in proportion. It counts parameters and FLOPs from formulas, so it tells you the cost of a config and nothing about whether the encoder is any good. Pick the pretraining objective from the Concept section first, then use this to see what the token budget does to your LLM context.

Exercises

Compute the patch-token sequence length for Qwen2.5-VL at native 1280x720 input with patch size 14. How does that compare to a CLS-only representation?
A 1080p frame (1920x1080) at patch 14 produces how many tokens? At 30 FPS over a 5-minute video, how many total visual tokens? Which cost saves you most: pooling, frame sampling, or token merging?
Implement mean pooling over patch tokens in pure Python. Verify that mean-pool over 196 tokens of a DINOv2 output matches what the model's forward returns when you ask for a pooled embedding.
Read Section 3 of "Vision Transformers Need Registers" (arXiv:2309.16588). Describe in two sentences what artifact the registers absorb and why it matters for downstream dense prediction.
Modify code/main.py to support patch-n'-pack: given a list of images of different resolutions, produce a single packed sequence and the block-diagonal attention mask. Verify against Lesson 12.06 when you reach it.

Key Terms

Term	What people say	What it actually means
Patch	"16x16 pixel square"	A fixed-size non-overlapping region of the input image; becomes one token
Patch embedding	"Linear projection"	A shared learned matrix (or Conv2d with stride=P) mapping flattened patch pixels to D-dim vectors
CLS token	"Class token"	Prepended learnable vector whose final hidden state represents the whole image; optional in 2026
Register token	"Sink token"	Extra learnable tokens that absorb the high-norm attention artifacts ViTs develop during pretraining
Position embedding	"Positional info"	Per-position vector or rotation making the sequence-order-aware; 2D-RoPE is the modern default
Grid	"Patch grid"	The (H/P) x (W/P) 2D array of patches for a given resolution and patch size
NaFlex	"Native flexible resolution"	SigLIP 2 feature: single model serves multiple aspect ratios and resolutions without retraining
Backbone	"Vision tower"	The pretrained image encoder whose patch-token outputs feed the LLM in a VLM
Pooling	"Image-level summary"	Strategy to turn patch tokens into one vector: CLS, mean, attention pool, or register-based
Patch 14 vs 16	"Finer vs coarser grid"	Patch 14 produces more tokens per image, better fidelity for OCR, slower; patch 16 is the classic default