Vision Encoder Patches

Patch embedding is the front end that turns an image into a sequence of tokens a transformer can read. It cuts the pixel grid into square regions and maps each region to one vector, so a few hundred tokens each summarize one square of the image.

The need comes from a mismatch. A transformer consumes a sequence of vectors, and an image is a 3-channel grid. Reading every pixel as a token explodes the sequence length: a 224x224 RGB image is 150,528 tokens, which a 12-layer transformer cannot afford in attention. Reading the image as one giant flat vector throws away locality, and the attention layer cannot recover it.

One linear projection handles this. A 224x224 image cut into 16x16 patches produces a 14x14 grid of 196 patches. Each patch is flattened from (3, 16, 16) = 768 pixel values into one vector, then a linear layer maps it to the model's hidden dimension. The transformer sees 196 tokens of dimension hidden (commonly 768) plus a CLS token, a sequence the rest of the network can handle.

Why patches, not pixels

Attention cost grows with the square of the sequence length. A 196-token sequence costs 196 * 196 = 38,416 attention scores per head per layer, while a 150,528-token sequence costs 150,528 * 150,528 = 22.6 billion. Patches buy a 590,000x reduction in attention compute, and a single 16x16 region carries enough signal for high-level vision tasks. You pay with fine-grained spatial detail inside one patch, which is why downstream multimodal stacks often run a second high-resolution branch when fine localization matters.

Why a linear projection is enough

A patch embedding treats each patch as an independent vector and projects it with one learned matrix. That projection learns a basis: edge detectors, color filters, simple textures. It is small (768 * 768 = 589,824 parameters for ViT-Base) and trains fast. Deeper convolutional stems exist (the "hybrid" ViT), but the flat linear projection is the standard, and most modern open-weight encoders ship with this exact shape.

The Conv2d trick

A Conv2d(in_channels=3, out_channels=hidden, kernel_size=patch_size, stride=patch_size) with no padding gives the same numerical result as unfold-then-linear, because each output position dot-products the patch pixels against one filter. The convolution is the patch projection. Most production codebases ship it this way: it is faster on GPU and uses one fewer reshape.

Position embeddings

A position embedding is a vector added to each token that says where in the grid the token came from. The projection alone leaves tokens with no order, so this signal has to be supplied. The 2D sinusoidal version gives each token a fixed signal that encodes its (row, col) position. Half the embedding dimension encodes row position with sin/cos at multiple frequencies, and the other half encodes column position. The encoding is deterministic, so the table can be rebuilt for a different grid without retraining, and it interpolates cleanly to grids the model never saw at training time. In this lesson's code, PatchEmbed raises ValueError unless the input matches cfg.image_size, so a new resolution means a new config. sinusoidal_2d also requires dim divisible by 4, because each half splits again into sin and cos.

For ViT-Base/16 at 224 resolution: 590,592 parameters in the projection, 768 in the CLS token, and zero for sinusoidal position. The next lesson (59) stacks a 12-layer transformer on top of this front end.

Equivalence as a sanity check

The patch step has two spellings: a Conv2d projection and an explicit unfold-then-linear. With the same weights they must produce the same output. If they do not, the unfold math is wrong and the rest of the encoder is built on sand. The tests in this lesson exercise that equivalence.

code/main.py implements:

• PatchEmbed, an nn.Module wrapping Conv2d for patch projection.
• sinusoidal_2d(grid_h, grid_w, dim), a stateless function that builds the 2D position table.
• VisionFrontEnd, which composes patch embedding, CLS prepend, and position addition into one forward pass.
• A synthesize_image(seed) helper that builds a deterministic 224x224x3 fixture from numpy.random.
• A demo that runs one fixture image through the front end and prints the output shape, the CLS token norm, the norms of the first eight tokens, the first eight values of one position row, a batch-consistency drift, and the max difference between the Conv2d projection and unfold-then-linear.

Run it:

python3 code/main.py

Output: the 224x224 fixture is tokenized to a sequence of shape (1, 197, 768). The first token is the CLS and the next 196 are patch tokens. Every patch row of the position table has the same norm, which is the sinusoidal signature. The demo does not print that norm, so check it yourself with model.pos_embed[0, 1:].norm(dim=-1).

The same patch front end sits at the start of the modern vision-language models: CLIP ViT-L/14, SigLIP, DINOv2, the Qwen-VL family, and the InternVL stack all begin with a Conv2d patch projection plus a position signal. They differ downstream: CLS vs no-CLS pooling, register tokens, patch sizes of 14 vs 16, dynamic resolution via interpolated positions. The front end you build here is the base each of those models stands on.

code/test_main.py covers:

• patch count matches (image_size / patch_size) ** 2
• output shape matches (batch, num_patches + 1, hidden)
• the Conv2d projection equals manual unfold-then-linear on a small fixture
• sinusoidal position table is deterministic across calls
• CLS token broadcasts across batch dim without leakage

Run them:

python3 -m unittest code/test_main.py

1. Replace the sinusoidal position with a learned nn.Parameter and compare the first-epoch loss on a tiny synthetic classification task. Learned positions win at fixed resolution, and sinusoidal wins when you change resolution after training.

2. Swap the Conv2d for an explicit nn.Unfold plus nn.Linear and assert the outputs match to within float tolerance. The math is the same, spelled two ways.

3. Add support for non-square patch sizes (e.g. 32x16 for wide-aspect inputs) and verify the position table handles non-square grids.

4. Profile the patch step at batch sizes 1, 8, 64. The patch projection is rarely the bottleneck, and the attention layers downstream dominate.

5. Train the front end as a frozen feature extractor on a 4-class synthetic shape dataset (circles, squares, triangles, stars). The CLS token output should linearly separate.

• An Image is Worth 16x16 Words (ViT, 2021) for the original patch-embed framing.
• Attention Is All You Need (2017) for the sinusoidal position formula adapted here to 2D.
• DINOv2 paper for register tokens, an extension you can add as exercise 6.