CLIP and Contrastive Vision-Language Pretraining

Before CLIP, vision was supervised. You collected a labeled dataset (ImageNet: 1.2M images, 1000 classes), trained a CNN, and shipped it. Labels are expensive, they skew toward whatever labelers can agree on, and they do not transfer to a new task without finetuning.

The web holds over a billion loosely labeled image-caption pairs for free. A picture of a golden retriever with alt text "my dog Max in the park" carries a supervisory signal, because the text describes the image. The open question is how to turn that into useful training.

CLIP treats it as a matching task. Given a batch of N images and N captions, learn to match each image to its own caption against N-1 distractors. The supervision is "these two things belong together; these N-1 do not." There are no class labels and no human annotation, only a contrastive loss.

The resulting embedding space does more than CLIP was trained for. ImageNet zero-shot works because "a photo of a cat" lands near pictures of cats that were never labeled as cats. That bet is why every 2026 VLM starts from a CLIP-style encoder.

The dual encoder

CLIP has two towers:

• Image encoder f: ViT or ResNet, outputs a D-dim vector per image.
• Text encoder g: small transformer, outputs a D-dim vector per caption.

Both towers normalize their outputs to unit length, so similarity is cos(f(x), g(y)) = f(x)^T g(y).

For a batch of N (image, caption) pairs, build the similarity matrix S of shape (N, N):

S[i, j] = cos(f(x_i), g(y_j)) / tau

where tau is a learned temperature (CLIP initializes to 0.07; learned in log-space).

InfoNCE loss

CLIP uses a symmetric cross-entropy over rows and columns:

loss_i2t = CE(S, labels=identity)     # each image's positive is its own caption
loss_t2i = CE(S^T, labels=identity)   # each caption's positive is its own image
loss = (loss_i2t + loss_t2i) / 2

This is InfoNCE. The softmax inside the cross-entropy pushes each image to match its own caption better than any other caption in the batch. The negatives are the other batch items, so a bigger batch means more negatives and a stronger signal. CLIP trained at batch 32k, and that scale matters.

Temperature

tau sets how sharp the softmax is. Low tau gives a sharp distribution with a hard-negative-mining effect. High tau gives a soft one where the whole batch contributes. CLIP learns log(1/tau) and clips it to prevent collapse. SigLIP 2 fixes the initial tau and uses a learned bias instead.

Why sigmoid scales better (SigLIP)

Softmax needs the whole similarity matrix in sync. In distributed training that means all-gathering the embeddings onto each replica before the softmax, which is quadratic in world size for communication.

SigLIP replaces the softmax with an element-wise sigmoid. For each pair (i, j) the loss is a binary classification of "is this the matching pair?" The diagonal is positive and everything else is negative. The loss is:

L = -1/N sum over (i, j) [ y_ij log sigmoid(S[i,j]) + (1-y_ij) log sigmoid(-S[i,j]) ]

y_ij = 1 if i == j, else 0. Each pair's loss is independent, so there is no all-gather: each GPU computes its local block and sums. SigLIP 2 scales to batch 32k-512k cheaply where CLIP would need proportionally more communication.

Zero-shot classification

Given N class names, for each class build a text template:

"a photo of a {class}"

Embed each template with the text encoder, embed your image with the image encoder, and take the argmax cosine similarity as the predicted class. Nothing is trained on the target classes.

Prompt templates matter. The original CLIP paper used 80 templates per class (plain, artistic, photo, painting, and so on) and averaged the embeddings, which bought +3 ImageNet points. Modern usage typically picks one or two templates.

Linear probes and finetuning

Zero-shot is the baseline. A linear probe (one linear layer trained on frozen CLIP features for your target classes) beats it on in-domain tasks. Full finetuning beats the linear probe in-domain but can hurt zero-shot transfer. Three regimes, three trade-offs.

SigLIP 2: NaFlex and dense features

SigLIP 2 (2025) adds four things:

• NaFlex: a single model handles variable aspect ratios and resolutions.
• Better dense features for segmentation and depth estimation, aimed at use as a frozen backbone in VLMs.
• Multilingual training on 100+ languages where CLIP was English-only.
• 1B param scale where CLIP topped out at 400M.

In 2026 open VLMs, SigLIP 2 SO400m/14 is the default vision tower. CLIP remains the default for pure image-text retrieval when the LAION-2B training distribution matches your query pattern.

ALIGN, BASIC, OpenCLIP, EVA-CLIP

ALIGN (Google, 2021) is the same idea as CLIP at 1.8B pairs, 90% noisy, and it showed that noisy data scales. OpenCLIP (LAION) is the open reproduction of CLIP on LAION-400M / 2B at multiple scales, and the go-to open checkpoint. EVA-CLIP initializes from masked image modeling and makes a strong backbone for VLMs. BASIC is Google's CLIP+ALIGN hybrid. Same family, different data and tuning.

The zero-shot ceiling

CLIP-class models cap around 76% ImageNet zero-shot (CLIP-G, OpenCLIP-G). Getting past that takes either much more data (SigLIP 2 gets 80%+) or architecture changes (supervised heads, more parameters). The benchmark is saturating. The value that matters is the embedding space downstream VLMs consume.

code/main.py is stdlib Python with no numpy and no torch. It does not train anything. It builds the pieces on synthetic embeddings so you can read the math:

1. similarity_matrix: normalize both sides and divide the cosine similarities by tau.
2. infonce_loss: the symmetric row and column cross-entropy, using log-sum-exp for numerical stability.
3. sigmoid_loss: the per-pair binary cross-entropy for comparison, with an optional bias argument.
4. zero_shot_classify: cosine similarity against a set of class vectors, sorted so the top entry is the prediction.

Four demos print the results: InfoNCE on 4 aligned pairs at three temperatures, aligned versus misaligned pairs, a zero-shot ranking, and a prompt-template ensemble. The embeddings are seeded Gaussian vectors, so the absolute numbers are toy. What carries over is how each loss responds to alignment and to tau.

This lesson produces outputs/skill-clip-zero-shot.md. Given a set of images (via path) and a list of target classes, it builds text prompts with the CLIP template, embeds both sides with a stated checkpoint (e.g., openai/clip-vit-large-patch14), and returns top-1 / top-5 predictions with similarity scores. The skill refuses to make claims about classes that are not in the prompt list.

1. Implement InfoNCE for a batch of 4 pairs by hand. Construct the 4x4 similarity matrix, run softmax, pick out the diagonal, compute cross-entropy. Verify your Python implementation against this hand calculation.

2. SigLIP uses a bias parameter b in addition to temperature: S'[i,j] = S[i,j]/tau + b. What role does b play when the batch has a large class imbalance (many more negatives than positives per row)? Read SigLIP Section 3 (arXiv:2303.15343).

3. Build a zero-shot classifier for cats vs dogs. Try two prompt templates: a photo of a {class} and a picture of a {class}. Measure accuracy on 100 test images. Does the ensemble of templates beat single?

4. Compute the communication cost of softmax InfoNCE vs sigmoid pairwise for a 512-GPU run at batch 32k. Which scales as O(N), which as O(N^2)? Cite SigLIP Section 4.

5. Read the OpenCLIP scaling-laws paper (arXiv:2212.07143, Cherti et al.). Reproduce their conclusion for data scaling from the figures: at fixed model size, what is the log-linear relationship between ImageNet zero-shot accuracy and training data size?

• Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision: the CLIP paper.
• Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training: SigLIP.
• Tschannen et al. (2025). SigLIP 2: multilingual + NaFlex.
• Jia et al. (2021). ALIGN: scale with noisy web data.
• Cherti et al. (2022). Reproducible scaling laws for contrastive language-image learning: OpenCLIP scaling laws.