Open-Vocabulary Vision — CLIP

Traditional classifiers are closed-vocabulary. A 1000-class ImageNet model can only ever predict 1000 labels, and every new category means new labelled data and a retrained head.

CLIP (Radford et al., OpenAI 2021) broke that constraint. Train on 400M (image, caption) pairs scraped from the web and you get a model that classifies into any set of categories at inference, described purely in natural language. Give it a new class by writing a sentence, nothing else.

That's zero-shot transfer, and it's why every modern vision system starts with a CLIP-family checkpoint. Detection (Grounding DINO, OWL-ViT), segmentation (CLIPSeg, SAM), retrieval, content moderation, VLMs, and text-to-image generation all build on CLIP-style joint embeddings.

Two towers

CLIP is two encoders trained as one system: an image tower and a text tower, pushed until matching pairs land at the same point in a shared embedding space.

Both towers end in a linear projection to that shared dimension (512 for CLIP-B/32, 1024 for CLIP-L/14), then get L2-normalised so a plain cosine similarity is all that's left to compute.

The objective

Take a batch of N (image, caption) pairs and build an NxN similarity matrix. Train both encoders so the diagonal (the matching pairs) comes out high and everything off the diagonal comes out low.

sim_matrix = image_embeddings @ text_embeddings.T / tau

loss_i2t = cross_entropy(sim_matrix,       targets=arange(N))
loss_t2i = cross_entropy(sim_matrix.T,     targets=arange(N))
loss = (loss_i2t + loss_t2i) / 2

The loss is symmetric because both directions need to work at inference: image-to-text and text-to-image retrieval. tau (temperature) is typically a learned scalar, initialised to 0.07.

SigLIP: a better loss

SigLIP (Zhai et al., 2023) swaps the softmax for a per-pair sigmoid:

loss = mean over pairs of log(1 + exp(-y_ij * sim_ij))
y_ij = +1 if matching, -1 otherwise

Because the loss is per-pair, it drops the batch-level normalisation CLIP's softmax needs. That lets SigLIP train well at small batch sizes, and it matches or beats CLIP given equal data.

Zero-shot classification

Given a trained CLIP:

1. For each class, compose a prompt: "a photo of a {class}".
2. Encode all class prompts with the text encoder -> T shape (C, d).
3. Encode the test image -> I shape (1, d).
4. Similarity = I @ T.T shape (1, C).
5. Argmax -> predicted class.

Prompt engineering matters here. OpenAI published 80 prompt templates for ImageNet ("a photo of a {}", "a blurry photo of a {}", "a sketch of a {}", ...), and averaging the embeddings of all templates per class buys an extra 1-3% top-1 accuracy.

Where CLIP-style models are used in 2026

• Zero-shot classification: direct use.
• Image retrieval: encode all images once, embed the query at inference.
• Text-conditioned detection: Grounding DINO and OWL-ViT wrap a CLIP text tower around a detector.
• Text-conditioned segmentation: CLIPSeg; SAM also takes text-prompt inputs via CLIP.
• VLMs: LLaVA, Qwen-VL, InternVL, and Llama's own vision-language releases all wire a CLIP-family vision encoder into an LLM.
• Text-to-image generation: Stable Diffusion and DALL-E 3 condition on CLIP text embeddings.
• Content moderation: safety classifiers score an image against CLIP embeddings of policy categories instead of training a bespoke classifier per policy.

Once you have a shared embedding space, every vision+language task becomes a distance computation.

Step 1: A tiny two-tower model

Real CLIP pairs a ViT with a transformer. Here the towers are small MLPs over pre-extracted features, so the training signal stays visible on CPU.

import torch
import torch.nn as nn
import torch.nn.functional as F
 
 
class TwoTower(nn.Module):
    def __init__(self, img_in=128, txt_in=64, emb=64):
        super().__init__()
        self.image_proj = nn.Sequential(nn.Linear(img_in, 128), nn.ReLU(), nn.Linear(128, emb))
        self.text_proj = nn.Sequential(nn.Linear(txt_in, 128), nn.ReLU(), nn.Linear(128, emb))
        self.logit_scale = nn.Parameter(torch.ones([]) * 2.6592)  # ln(1/0.07)
 
    def forward(self, img_feats, txt_feats):
        i = F.normalize(self.image_proj(img_feats), dim=-1)
        t = F.normalize(self.text_proj(txt_feats), dim=-1)
        return i, t, self.logit_scale.exp()

Two projections, a shared output dimension, a learned temperature: same shape as the real CLIP API, just smaller.

Step 2: Contrastive loss

The two towers only learn anything once a loss rewards matching pairs and punishes mismatched ones. This is the symmetric objective from the Concept section, translated directly into PyTorch:

def clip_loss(image_emb, text_emb, logit_scale):
    N = image_emb.size(0)
    sim = logit_scale * image_emb @ text_emb.T
    targets = torch.arange(N, device=sim.device)
    l_i = F.cross_entropy(sim, targets)
    l_t = F.cross_entropy(sim.T, targets)
    return (l_i + l_t) / 2

A higher logit_scale sharpens the softmax: more confident predictions, but also more risk of instability.

Step 3: Zero-shot classifier

Once the towers are trained, classification needs no retraining at all. This is the five-step zero-shot procedure from the Concept section, translated directly into code:

@torch.no_grad()
def zero_shot_classify(model, image_feats, class_text_feats, class_names):
    """
    image_feats:      (N, img_in)
    class_text_feats: (C, txt_in)   one averaged embedding per class
    """
    i = F.normalize(model.image_proj(image_feats), dim=-1)
    t = F.normalize(model.text_proj(class_text_feats), dim=-1)
    sim = i @ t.T
    pred = sim.argmax(dim=-1)
    return [class_names[p] for p in pred.tolist()]

One line per step, and it's exactly what you'd call against a production CLIP checkpoint.

Step 4: Sanity check

Before training on anything real, confirm the loss behaves the way the math predicts on random weights:

torch.manual_seed(0)
model = TwoTower()
 
img = torch.randn(8, 128)
txt = torch.randn(8, 64)
i, t, scale = model(img, txt)
loss = clip_loss(i, t, scale)
print(f"batch size: {i.size(0)}   loss: {loss.item():.3f}")

Loss should land close to log(N) = log(8) = 2.08 for a randomly initialised model. That's the symmetric cross-entropy target when no structure has been learned yet.

OpenCLIP is the community default in 2026:

import open_clip
import torch
from PIL import Image
 
model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")
tokenizer = open_clip.get_tokenizer("ViT-B-32")
 
image = preprocess(Image.open("dog.jpg")).unsqueeze(0)
text = tokenizer(["a photo of a dog", "a photo of a cat", "a photo of a car"])
 
with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features = image_features / image_features.norm(dim=-1, keepdim=True)
    text_features = text_features / text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
 
print(probs)

SigLIP is newer, trains better at small scales, and is the better pick for new work: google/siglip-base-patch16-224. Hugging Face ships both.

This lesson produces:

• outputs/prompt-zero-shot-class-picker.md, a prompt that designs class templates for zero-shot CLIP given a list of classes and a domain.
• outputs/skill-image-text-retriever.md, a skill that builds an image embedding index with any CLIP checkpoint and supports query-by-text and query-by-image.

1. (Easy) Use a pretrained OpenCLIP ViT-B/32 and do zero-shot classification on CIFAR-10 with the 80-template prompt set. Report top-1 accuracy; it should be around 85-90%.
2. (Medium) Compare single-template ("a photo of a {}") vs 80-template averaged embeddings on the same CIFAR-10 task. Quantify the gap and explain why templates help.
3. (Hard) Build a zero-shot image retrieval index: embed 1,000 images with CLIP, build a FAISS index (Meta's library for fast nearest-neighbor vector search), query with a natural language description. Report retrieval recall@5 for 20 held-out queries you write by hand.

• CLIP: Learning Transferable Visual Models from Natural Language Supervision (Radford et al., 2021)
• SigLIP: Sigmoid Loss for Language-Image Pre-Training (Zhai et al., 2023)
• OpenCLIP, the community codebase
• DINOv2 vs CLIP vs MAE: a features comparison, an HF guide with side-by-side use cases