SAM 3 & Open-Vocabulary Segmentation

The 2023 SAM was a visual-prompt-only model: click a point or draw a box and it returns a mask. Ask for "every orange in this photo" and SAM alone can't do it. You needed a detector (Grounding DINO) to produce boxes first, then SAM to segment each one. Grounded SAM turned that into a working pipeline, but it was still a cascade of two frozen models, and cascades accumulate error at every hop.

SAM 3 (Meta, Nov 2025, ICLR 2026) collapsed the cascade. Give it a short noun phrase or an image exemplar as a prompt and it returns every matching mask and instance ID in one forward pass. That's Promptable Concept Segmentation (PCS). The March 2026 Object Multiplex update (SAM 3.1) extends this to video, tracking multiple instances of the same concept efficiently across frames.

The bigger change is structural: 2D segmentation, detection, and text-image grounding have merged into one model. The production question isn't "which pipeline do I chain together" anymore. It's "which promptable model handles my use case end to end."

The three generations

Promptable Concept Segmentation

A "concept prompt" is a short noun phrase ("yellow school bus", "striped red umbrella", "hand holding a mug") or an image exemplar. The model returns segmentation masks for every instance in the image that matches the concept, plus a unique instance ID per match.

This differs from classic visual-prompt SAM in three ways:

1. No per-instance prompting required: one text prompt returns all matches.
2. Open-vocabulary: the concept can be anything describable in natural language.
3. Returns multiple instances at once rather than one mask per prompt.

Key architectural pieces

• Shared backbone: a single ViT (Vision Transformer) processes the image. Both the detector head and the memory-based tracker read from it.
• Presence head: predicts whether the concept is present in the image at all, decoupling "is this here?" from "where is it?" and cutting false positives on absent concepts.
• Decoupled detector-tracker: image-level detection and video-level tracking have separate heads so they don't interfere with each other.
• Memory bank: stores per-instance features across frames for video tracking (the same mechanism SAM 2 used).

Those four pieces run on every forward pass. What taught them to work is a data pipeline built at a different scale entirely.

Training at scale

SAM 3 was trained on 4 million unique concepts generated by a data engine that iteratively annotates and corrects using AI plus human review. The new SA-CO benchmark contains 270K unique concepts, 50x larger than prior benchmarks. SAM 3 reaches 75-80% of human performance on SA-CO and doubles existing systems on image and video PCS.

SAM 3.1 Object Multiplex

March 2026 update: Object Multiplex introduces a shared-memory mechanism for joint tracking of many instances of the same concept at once. Previously, tracking N instances meant N separate memory banks. Multiplex collapses that into one shared memory with per-instance queries, so tracking stays fast as the object count grows, without giving up accuracy.

Where Grounded SAM still matters in 2026

• When you need a specific open-vocabulary detector swapped in (DINO-X, Florence-2).
• When the SAM 3 license (gated on HF) is a blocker.
• When you need more control over the detector threshold than SAM 3 exposes.
• For research or ablation work on the detector component.

Modular pipelines still have a place. For most production work, SAM 3 is the simpler answer.

YOLO-World vs SAM 3

• YOLO-World: open-vocabulary detector only, no masks. Real-time. Best when you need boxes at high fps.
• SAM 3: full segmentation plus tracking. Slower, but richer output.

Production split: YOLO-World for fast detection-only pipelines (robotics navigation, fast dashboards), SAM 3 for anything that needs masks or tracking.

SAM-MI efficiency

SAM-MI (2025-2026) addresses SAM's decoder bottleneck. Key ideas:

• Sparse point prompting: uses a few well-chosen points instead of dense prompts, cutting decoder calls by 96%.
• Shallow mask aggregation: merges rough mask predictions into one sharper mask.
• Decoupled mask injection: decoder receives pre-computed mask features instead of re-running.

Net effect: roughly 1.6x speedup over Grounded-SAM on open-vocabulary benchmarks.

Output format for the three models

All three return the same general structure: boxes, labels, scores, masks, IDs. That's deliberate, so your downstream pipeline doesn't have to branch on which model produced the output.

Step 1: Prompt construction

Build a helper that turns a user sentence into a list of SAM 3 concept prompts. This is the boundary where "what the user typed" meets "what the model consumes".

def split_concepts(sentence):
    """
    Heuristic splitter for multi-concept prompts.
    Returns list of short noun phrases.
    """
    for sep in [",", ";", "and", "or", "&"]:
        if sep in sentence:
            parts = [p.strip() for p in sentence.replace("and ", ",").split(",")]
            return [p for p in parts if p]
    return [sentence.strip()]
 
print(split_concepts("cats, dogs and balloons"))

SAM 3 accepts one concept per forward pass; for multi-concept queries, loop or batch them.

Step 2: Post-processing helpers

Turn SAM 3's raw outputs into a clean list of detections that match our Phase 4 Lesson 16 pipeline contract.

from dataclasses import dataclass
from typing import List
 
@dataclass
class ConceptDetection:
    concept: str
    instance_id: int
    box: tuple          # (x1, y1, x2, y2)
    score: float
    mask_rle: str       # run-length encoded
 
 
def rle_encode(binary_mask):
    flat = binary_mask.flatten().astype("uint8")
    runs = []
    prev, count = flat[0], 0
    for v in flat:
        if v == prev:
            count += 1
        else:
            runs.append((int(prev), count))
            prev, count = v, 1
    runs.append((int(prev), count))
    return ";".join(f"{v}x{c}" for v, c in runs)

RLE keeps response payloads small even for many high-resolution masks. The same format works across SAM 2, SAM 3, Grounded SAM 2.

Step 3: A unified open-vocab segmentation interface

Wrap whatever backend you have (SAM 3, Grounded SAM 2, YOLO-World + SAM 2) behind a single method. Your downstream code does not change when the backend does.

from abc import ABC, abstractmethod
import numpy as np
 
class OpenVocabSeg(ABC):
    @abstractmethod
    def detect(self, image: np.ndarray, concept: str) -> List[ConceptDetection]:
        ...
 
 
class StubOpenVocabSeg(OpenVocabSeg):
    """
    Deterministic stub used for pipeline testing when real models are not loaded.
    """
    def detect(self, image, concept):
        h, w = image.shape[:2]
        return [
            ConceptDetection(
                concept=concept,
                instance_id=0,
                box=(w * 0.2, h * 0.3, w * 0.5, h * 0.8),
                score=0.89,
                mask_rle="0x100;1x50;0x200",
            ),
            ConceptDetection(
                concept=concept,
                instance_id=1,
                box=(w * 0.55, h * 0.25, w * 0.85, h * 0.75),
                score=0.74,
                mask_rle="0x80;1x40;0x220",
            ),
        ]

The real SAM3OpenVocabSeg subclass would wrap transformers.Sam3Model and Sam3Processor.

Step 4: Hugging Face SAM 3 usage (reference)

Here's the real model, via the transformers integration:

from transformers import Sam3Processor, Sam3Model
import torch
 
processor = Sam3Processor.from_pretrained("facebook/sam3")
model = Sam3Model.from_pretrained("facebook/sam3").eval()
 
inputs = processor(images=pil_image, return_tensors="pt")
inputs = processor.set_text_prompt(inputs, "yellow school bus")
 
with torch.no_grad():
    outputs = model(**inputs)
 
masks = processor.post_process_masks(
    outputs.masks, inputs.original_sizes, inputs.reshaped_input_sizes
)
boxes = outputs.boxes
scores = outputs.scores

One prompt, all matches returned in a single call. Feed the same model a sequence of frames instead of a single image and the memory bank persists each instance's features across them automatically; that's the mechanism behind the video-tracking use cases in Use It.

Step 5: What you gain and give up moving off Grounded SAM 2

An honest benchmark: what happens when you replace Grounded SAM 2 with SAM 3 in a real pipeline?

• Latency: SAM 3 saves one forward pass (no separate detector) but the model itself is heavier; usually net-neutral or a slight speedup.
• Accuracy: SAM 3 substantially better on rare or compositional concepts ("striped red umbrella"). Similar on common single-word concepts.
• Flexibility: Grounded SAM 2 lets you swap detectors (DINO-X, Florence-2, Grounding DINO 1.5); SAM 3 is monolithic.

Conclusion: SAM 3 is the default for 2026 open-vocab seg. Grounded SAM 2 is still the right answer when you need detector flexibility or different license terms.

Production deployment patterns:

• Real-time annotation: SAM 3 plus CVAT's label-as-text-prompt feature. Annotators select a label name, SAM 3 pre-labels every matching instance, and a human reviews and corrects.
• Video analytics: SAM 3.1 Object Multiplex for multi-object tracking, feeding frames to the memory-based tracker.
• Robotics: SAM 3 for open-vocab manipulation ("pick up the red cup"), running as a planning primitive.
• Medical imaging: SAM 3 fine-tuned on medical concepts, which requires an access request on HF.

Ultralytics wraps SAM 3 in its Python package:

from ultralytics import SAM
 
model = SAM("sam3.pt")
results = model(image_path, prompts="yellow school bus")

Same interface as YOLO and SAM 2.

This lesson produces:

• outputs/prompt-open-vocab-stack-picker.md: a prompt that picks SAM 3 / Grounded SAM 2 / YOLO-World / SAM-MI based on latency, concept complexity, and licensing.
• outputs/skill-concept-prompt-designer.md: a skill that turns user utterances into well-formed SAM 3 concept prompts (splitting, disambiguation, fallbacks).

1. (Easy) Run SAM 3 on 10 images with concept prompts you choose. Compare against SAM 2 + Grounding DINO 1.5 on the same images. Report which concepts each model missed.
2. (Medium) Build a "click-to-include / click-to-exclude" UI on top of SAM 3: a text prompt returns candidate instances; user clicks keep which ones count as positive. Output the final concept set as JSON.
3. (Hard) Fine-tune SAM 3 on a custom concept set (e.g. 5 types of electronic components) with 20 labelled images each. Compare to zero-shot SAM 3 on the same test set; measure mask IoU improvement.

• SAM 3: Segment Anything with Concepts (arXiv 2511.16719)
• SAM 3.1 Object Multiplex (Meta AI, March 2026)
• SAM 3 model page on Hugging Face
• Grounded SAM 2 tutorial (PyImageSearch)
• Ultralytics SAM 3 docs
• SAM3-I: Instruction-aware SAM (arXiv 2512.04585)