Multi-Object Tracking & Video Memory

A detector finds objects in a single frame. A tracker figures out which detection in frame t is the same object as a detection in frame t-1. Skip that step and you cannot count objects crossing a line, follow a ball through an occlusion, or say "car #4 has been in the lane for 8 seconds" — all you have is a stack of unrelated snapshots.

Tracking sits underneath almost every video-facing product: sports analytics, surveillance, autonomous driving, medical video analysis, wildlife monitoring, footfall counting. The building blocks repeat across all of them: a per-frame detector, a motion model (a Kalman filter or something richer), an association step (Hungarian assignment on IoU, cosine similarity, or learned features), and a track lifecycle that governs how tracks get born, updated, and killed.

2026 added two new patterns on top of that classical stack. SAM 2 tracks by feature memory instead of a motion model. SAM 3.1 Object Multiplex shares one memory bank across many instances of the same concept. This lesson works through the classical stack first, then the memory-based approach.

Tracking-by-detection

Every tracker you'll run into in 2026 is a variation on this loop. What changes between them:

• SORT (2016): Kalman filter plus IoU Hungarian matching. Simple, fast, no appearance model.
• DeepSORT (2017): SORT plus a CNN-based appearance feature per track, a ReID embedding, which handles crossings better.
• ByteTrack (2021): matches low-confidence detections as a second stage. No appearance features needed, and still the top performer on MOT17.
• BoT-SORT (2022): ByteTrack plus camera motion compensation plus ReID.
• StrongSORT and OC-SORT: ByteTrack descendants with stronger motion and appearance modeling.

Query-based tracking: the other paradigm

Tracking-by-detection always keeps detection and association as two separate, bolted-together steps. Query-based tracking collapses them into one model. MOTR and its successors (MOTRv2, MOTRv3) represent each tracked object as a learned query token that gets refined frame to frame inside a transformer, so the same forward pass that detects new objects also carries forward the identity of ones already being tracked. No external Hungarian matching, no hand-tuned IoU threshold. The network learns the association implicitly, in the same spirit as SAM 2's memory bank, but trained end to end for detection and tracking jointly rather than added on top of a segmenter.

Kalman filter in one paragraph

A Kalman filter maintains a per-track state, (x, y, w, h, dx, dy, dw, dh), plus a covariance that tracks how much to trust it. Each frame it predicts the new state from a constant-velocity model, then updates that prediction with the matched detection, weighting the detection more heavily when the prediction's uncertainty is high. The result is smooth trajectories and tracks that can survive a short occlusion, one to five frames, without losing their identity.

Every classical tracker leans on a Kalman filter for this motion-prediction step.

The Hungarian algorithm

Given an M x N cost matrix (tracks by detections), the Hungarian algorithm finds the one-to-one assignment that minimises total cost. Cost is usually 1 - IoU(track_bbox, detection_bbox), or negative cosine similarity between appearance features. Runtime is O((M+N)^3), fast enough in Python via scipy.optimize.linear_sum_assignment for M and N up to roughly 1000.

ByteTrack's key idea

Standard trackers throw away low-confidence detections (below roughly 0.5). ByteTrack keeps them as second-stage candidates: after matching tracks against high-confidence detections, whatever's left unmatched gets a second pass against the low-confidence pool, with a slightly looser IoU threshold. That second pass is what recovers short occlusions and prevents ID switches near crowds.

SAM 2 memory-based tracking

SAM 2 handles video by keeping a memory bank of per-instance spatio-temporal features. Give it a prompt, a click, a box, or text, on one frame, and it encodes that instance into memory. On every later frame, the memory cross-attends against the new frame's features, and the decoder produces a mask for the same instance right there.

No Kalman filter. No Hungarian assignment. The association happens implicitly, inside the memory-attention operation itself.

Pros:

• Survives large occlusions, since the memory carries instance identity across many frames.
• Open-vocabulary once combined with SAM 3's text prompts.
• No separate motion model required.

Cons:

• Slower than ByteTrack once you're tracking many objects.
• The memory bank grows over time and eats into the context window.

SAM 3.1 Object Multiplex

Earlier SAM 2 and SAM 3 tracking kept one memory bank per instance: track 50 objects, maintain 50 memory banks. Object Multiplex (March 2026) collapses all of them into a single shared memory with per-instance query tokens, so cost scales sub-linearly with the number of instances instead of linearly.

That's what makes Multiplex the default for 2026 crowd tracking: concert crowds, warehouse workers, traffic intersections, anywhere the object count used to be the bottleneck.

Three metrics to know

• MOTA (Multi-Object Tracking Accuracy): 1 - (FN + FP + ID switches) / GT, weighted by error type. A single number that conflates detection and association failures, so it can hide exactly the problem you care about.
• IDF1 (ID F1): the harmonic mean of ID precision and recall. It answers "did each ground-truth track keep its ID over time," which makes it the better read than MOTA whenever ID switches are what worries you.
• HOTA (Higher Order Tracking Accuracy): splits into detection accuracy (DetA) and association accuracy (AssA) instead of collapsing everything into one score. The community standard since 2020, and the most complete of the three.

Report IDF1 for surveillance, where the question is who is who. Reach for HOTA everywhere else: sports analytics (counting passes) and general academic comparison both call for it.

Step 1: IoU-based cost matrix

import numpy as np
 
 
def bbox_iou(a, b):
    """
    a, b: (N, 4) arrays of [x1, y1, x2, y2].
    Returns (N_a, N_b) IoU matrix.
    """
    ax1, ay1, ax2, ay2 = a[:, 0], a[:, 1], a[:, 2], a[:, 3]
    bx1, by1, bx2, by2 = b[:, 0], b[:, 1], b[:, 2], b[:, 3]
    inter_x1 = np.maximum(ax1[:, None], bx1[None, :])
    inter_y1 = np.maximum(ay1[:, None], by1[None, :])
    inter_x2 = np.minimum(ax2[:, None], bx2[None, :])
    inter_y2 = np.minimum(ay2[:, None], by2[None, :])
    inter = np.clip(inter_x2 - inter_x1, 0, None) * np.clip(inter_y2 - inter_y1, 0, None)
    area_a = (ax2 - ax1) * (ay2 - ay1)
    area_b = (bx2 - bx1) * (by2 - by1)
    union = area_a[:, None] + area_b[None, :] - inter
    return inter / np.clip(union, 1e-8, None)

Step 2: Minimal SORT-style tracker

This version skips the constant-velocity Kalman step and matches on IoU alone, which keeps the code short enough to read in one sitting. Production trackers need the Kalman predict; grab the full version from the sort Python package when you do.

from scipy.optimize import linear_sum_assignment
 
 
class Track:
    def __init__(self, tid, bbox, frame):
        self.id = tid
        self.bbox = bbox
        self.last_frame = frame
        self.hits = 1
 
    def update(self, bbox, frame):
        self.bbox = bbox
        self.last_frame = frame
        self.hits += 1
 
 
class SimpleTracker:
    def __init__(self, iou_threshold=0.3, max_age=5):
        self.tracks = []
        self.next_id = 1
        self.iou_threshold = iou_threshold
        self.max_age = max_age
 
    def step(self, detections, frame):
        if not self.tracks:
            for d in detections:
                self.tracks.append(Track(self.next_id, d, frame))
                self.next_id += 1
            return [(t.id, t.bbox) for t in self.tracks]
 
        track_boxes = np.array([t.bbox for t in self.tracks])
        det_boxes = np.array(detections) if len(detections) else np.empty((0, 4))
 
        iou = bbox_iou(track_boxes, det_boxes) if len(det_boxes) else np.zeros((len(track_boxes), 0))
        cost = 1 - iou
        cost[iou < self.iou_threshold] = 1e6
 
        matched_track = set()
        matched_det = set()
        if cost.size > 0:
            row, col = linear_sum_assignment(cost)
            for r, c in zip(row, col):
                if cost[r, c] < 1.0:
                    self.tracks[r].update(det_boxes[c], frame)
                    matched_track.add(r); matched_det.add(c)
 
        for i, d in enumerate(det_boxes):
            if i not in matched_det:
                self.tracks.append(Track(self.next_id, d, frame))
                self.next_id += 1
 
        self.tracks = [t for t in self.tracks if frame - t.last_frame <= self.max_age]
        return [(t.id, t.bbox) for t in self.tracks]

Sixty lines, and that's the whole contract: feed it per-frame detections, get back per-frame track IDs. A real system layers the Kalman predict, ByteTrack's second-stage re-match, and appearance features on top of this same core.

Step 3: Synthetic trajectory test

def synthetic_frames(num_frames=20, num_objects=3, H=240, W=320, seed=0):
    rng = np.random.default_rng(seed)
    starts = rng.uniform(20, 200, size=(num_objects, 2))
    velocities = rng.uniform(-5, 5, size=(num_objects, 2))
    frames = []
    for f in range(num_frames):
        dets = []
        for i in range(num_objects):
            cx, cy = starts[i] + f * velocities[i]
            dets.append([cx - 10, cy - 10, cx + 10, cy + 10])
        frames.append(dets)
    return frames
 
 
tracker = SimpleTracker()
for f, dets in enumerate(synthetic_frames()):
    tracks = tracker.step(dets, f)

Three objects moving in straight lines should keep the same ID across all 20 frames.

Step 4: ID-switch metric

def count_id_switches(tracks_per_frame, gt_per_frame):
    """
    tracks_per_frame:  list of list of (track_id, bbox)
    gt_per_frame:      list of list of (gt_id, bbox)
    Returns number of ID switches.
    """
    prev_assignment = {}
    switches = 0
    for tracks, gts in zip(tracks_per_frame, gt_per_frame):
        if not tracks or not gts:
            continue
        t_boxes = np.array([b for _, b in tracks])
        g_boxes = np.array([b for _, b in gts])
        iou = bbox_iou(g_boxes, t_boxes)
        for g_idx, (gt_id, _) in enumerate(gts):
            j = iou[g_idx].argmax()
            if iou[g_idx, j] > 0.5:
                t_id = tracks[j][0]
                if gt_id in prev_assignment and prev_assignment[gt_id] != t_id:
                    switches += 1
                prev_assignment[gt_id] = t_id
    return switches

Think of this as an IDF1-adjacent metric, simplified: it counts how many times a ground-truth object's assigned track ID changes. For the real MOTA, IDF1, and HOTA implementations, reach for py-motmetrics and TrackEval instead.

What's actually running in production in 2026:

• ultralytics: YOLOv8 with ByteTrack or BoT-SORT built in. results = model.track(source, tracker="bytetrack.yaml"). This is the default most teams reach for first.
• supervision (Roboflow): ByteTrack wrappers plus annotation utilities.
• SAM 2 / SAM 3.1: memory-based tracking via processor.track().
• A custom stack: a detector (YOLOv8 or RT-DETR) paired with sort-tracker, OC-SORT, or StrongSORT.

How to pick:

• Pedestrians, cars, or boxes at 30+ fps: ByteTrack with ultralytics.
• Many instances of one class in a crowd: SAM 3.1 Object Multiplex.
• Heavy occlusions with identifiable appearance: DeepSORT or StrongSORT, using ReID features.
• Sports or other complex interactions: BoT-SORT, or a learned tracker like MOTRv3.

This lesson ships two reusable outputs:

• outputs/prompt-tracker-picker.md: picks between SORT, ByteTrack, BoT-SORT, SAM 2, and SAM 3.1 given scene type, occlusion patterns, and latency budget.
• outputs/skill-mot-evaluator.md: writes a full evaluation harness for MOTA, IDF1, and HOTA against ground-truth tracks.

1. (Easy) Run the synthetic tracker above with 3, 10, and 30 objects. Report the ID-switch count for each, and pin down where plain IoU-only association starts to break.
2. (Medium) Add a constant-velocity Kalman predict step before association. Show that short occlusions (2-3 frames) no longer cause ID switches.
3. (Hard) Swap in SAM 2's memory-based tracker (via transformers) as an alternative backend. Run both SimpleTracker and SAM 2 on a 30-second clip of a crowd, manually label ground-truth IDs for 5 salient people, and compare ID-switch counts between the two.

• SORT (Bewley et al., 2016): the minimal tracking-by-detection paper
• DeepSORT (Wojke et al., 2017): adds an appearance feature
• ByteTrack (Zhang et al., 2022): low-confidence second pass
• BoT-SORT (Aharon et al., 2022): camera motion compensation
• HOTA (Luiten et al., 2020): decomposed tracking metric
• SAM 2 video segmentation (Meta, 2024): memory-based tracker
• SAM 3.1 Object Multiplex (Meta, March 2026)