Vision Transformers (ViT)

A Vision Transformer (ViT) is a standard transformer encoder that reads an image as a sequence of fixed-size patches instead of a sequence of word tokens. Before 2020, computer vision meant convolutions. The state of the art on ImageNet, COCO, and detection benchmarks came from CNN backbones, and transformers were for language.

Dosovitskiy et al. (2020), "An Image is Worth 16x16 Words," showed you can drop the convolutions entirely. Slice an image into fixed-size patches, linearly project each patch into an embedding, and feed the sequence to a vanilla transformer encoder. At sufficient scale (ImageNet-21k pretraining or bigger), ViT matches or beats ResNet-based models.

ViT started a broader pattern: one architecture, many modalities. Whisper tokenizes audio. ViT tokenizes images. Robotics gets action tokens and video gets pixel tokens. The transformer doesn't care what the tokens are. Feed it a sequence and it learns.

By 2026, ViT and its descendants (DeiT, Swin, DINOv2, ViT-22B, SAM 3) own most of vision. CNNs still win on edge devices and latency-sensitive tasks. Outside those, most stacks have a ViT somewhere in them.

Step 1: patchify

Patchify means splitting an H × W × C image into an N × (P·P·C) sequence of flat patches. The typical setup is a 224 × 224 image with 16 × 16 patches, which gives 196 patches of 768 values each.

image (224, 224, 3) → 14 × 14 grid of 16x16x3 patches → 196 vectors of length 768

Patch size is the main lever. Smaller patches mean more tokens, finer resolution, and quadratic attention cost. Larger patches are coarser and cheaper.

Step 2: linear embedding

A single learned matrix projects each flat patch to d_model. That is equivalent to a convolution with kernel size P and stride P. In PyTorch it is nn.Conv2d(C, d_model, kernel_size=P, stride=P), a two-line implementation.

Step 3: prepend [CLS] token, add positional embeddings

• Prepend a learnable [CLS] token. Its final hidden state is the image representation used for classification.
• Add learnable positional embeddings (original ViT) or 2D sinusoidal ones (later variants).
• From 2024 on, RoPE extended to 2D handles position, sometimes without explicit embeddings.

Step 4: standard transformer encoder

Stack L blocks of LayerNorm → Self-Attention → + → LayerNorm → MLP → +. The block is BERT-style but pre-LN (LayerNorm comes before attention and the MLP, where BERT normalizes after), with no vision-specific layers. That is the pedagogical punchline of the paper.

Step 5: head

For classification, take the [CLS] hidden state, apply a linear layer, then softmax. DINOv2 and SAM discard [CLS] and use the patch embeddings directly.

Variants that mattered

Why it took a while

ViT needs a lot of data to match CNNs because it lacks the CNN inductive biases (translation invariance, locality). Without more than 100M labeled images or strong self-supervised pretraining, CNNs still win at matched compute. DeiT narrowed the gap in 2021 with distillation tricks, and DINOv2 closed it in 2023 with self-supervision.

See code/main.py. It is a pure-stdlib patchify, linear embedding, and sanity-check script. There is no training, because ViT at realistic scale needs PyTorch and hours of GPU time.

Step 1: fake image

A 24 × 24 RGB image, stored as a list of rows of (R, G, B) tuples. With 6×6 patches that gives a 4 × 4 grid of 16 patches, each a flat vector of 108 values (6·6·3). The embedding width d_model is 48, set in main().

Step 2: patchify

The core of patchify is below (the real function also asserts that H and W divide evenly by P and returns (patches, (H // P, W // P)), so the caller gets the grid shape too).

def patchify(image, P):
    H = len(image)
    W = len(image[0])
    patches = []
    for i in range(0, H, P):
        for j in range(0, W, P):
            patch = []
            for di in range(P):
                for dj in range(P):
                    patch.extend(image[i + di][j + dj])
            patches.append(patch)
    return patches

Patches come out in raster order, row-major across the grid. ViTs follow this ordering.

Step 3: linear embed

Multiply each flat patch by a random (patch_flat_size, d_model) matrix. Then main.py prepends a [CLS] vector and adds 2D sinusoidal positions to the patch tokens. Here the [CLS] vector is a fixed seeded Gaussian draw (rng.gauss(0, 0.02)), not a trained parameter, and the positions are sinusoidal, not learned. The script prints a sequence length of 17 (16 patches plus [CLS]). It does not assert d_model, so check by hand that each token has 48 values.

Step 4: count parameters for a realistic ViT

main.py prints approximate parameter counts for five configurations: Tiny (~5.7M), Small (~22.0M), Base (~86.5M), Large (~304.1M), and Huge (~631.7M). ViT-Base is 12 layers, 12 heads, d=768, patch=16. For scale, ResNet-50 is about 25M, which the script does not print.

from transformers import ViTImageProcessor, ViTModel
import torch
from PIL import Image
 
processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTModel.from_pretrained("google/vit-base-patch16-224-in21k")
 
img = Image.open("cat.jpg")
inputs = processor(img, return_tensors="pt")
out = model(**inputs).last_hidden_state   # (1, 197, 768): [CLS] + 196 patches
cls_emb = out[:, 0]                       # image representation

DINOv2 embeddings are the 2026 default for image features. Freeze the backbone and train a tiny head. That works for classification, retrieval, detection, and captioning. Meta's DINOv2 checkpoints outperform CLIP on non-text vision tasks.

Patch-size picking. Small models use 16×16 (ViT-B/16). Dense prediction (segmentation) uses 8×8 or 14×14 (SAM, DINOv2). Very large models use 14×14.

The skill in outputs/skill-vit-configurator.md picks a ViT variant and patch size for a new vision task, given dataset size, resolution, and compute budget.

1. Easy. Run code/main.py. Verify that the number of patches equals (H/P) * (W/P) and that the flat patch dimension equals P*P*C.
2. Medium. Implement 2D sinusoidal positional embeddings: two independent sinusoidal codes for the row and col of each patch, concatenated. Feed them into a tiny PyTorch ViT and compare accuracy against learnable positional embeddings on CIFAR-10.
3. Hard. Build a 3-layer ViT in PyTorch and train it on 1,000 MNIST images with 4×4 patches. Measure test accuracy. Then add DINOv2-style pretraining on the same 1,000 images (simplified: train the encoder to predict patch embeddings from masked patches). Does accuracy improve?

• Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: the ViT paper.
• Touvron et al. (2021). Training data-efficient image transformers & distillation through attention: DeiT.
• Liu et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows: Swin.
• Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision: DINOv2.
• Darcet et al. (2023). Vision Transformers Need Registers: the register-token fix for DINOv2.