BuildPython

Video-Language Models: Temporal Tokens and Grounding

Video is not a stack of photos. A 5-second clip has causal ordering, action verbs, and event timing that an image model cannot represent. Video-LLaMA (Zhang et al., June 2023) shipped the first open video-LLM with audio-visual grounding, and VideoChat and Video-LLaVA scaled the pattern. By 2025 Qwen2.5-VL's TMRoPE had closed much of the gap with frontier proprietary models. Each system handles temporal tokens differently: a Q-former per clip, concat-pool per frame, TMRoPE per token. This lesson reads those patterns, builds a uniform and a dynamic frame sampler, and scores temporal grounding with IoU.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 08 (LLaVA-OneVision)

Learning Objectives

Explain why temporal positional encoding changes video VLM performance independently of the vision encoder.
Compare uniform, dynamic-FPS, and event-driven frame sampling on tokens-per-second vs grounding accuracy.
Describe Q-former-per-clip (Video-LLaMA) vs pooled-per-frame (Video-LLaVA) vs M-RoPE-per-token (Qwen2.5-VL) designs.
Name the four video benchmarks: VideoMME, TempCompass, EgoSchema, Video-MMMU.

The Problem

A 1-minute video at 30 FPS is 1800 frames. At 196 visual tokens per frame (ViT-B at 224), that is 352k tokens, more than most 2024-era LLMs could take in.

Three reduction strategies exist:

Subsample frames (1-8 FPS depending on content).
Pool each frame's patch tokens aggressively (3x3 or 4x4 bilinear pool).
Compress via a Q-former that takes a 16-frame clip and outputs 64 tokens.

Each one costs you something different. Subsampling loses temporal detail, pooling loses spatial detail, and a Q-former loses a little of both but saves tokens.

Temporal position encoding is the other axis. How does the model know frame 5 came before frame 6? Video-LLaMA uses a simple 1D temporal RoPE, Video-LLaVA uses learned temporal embeddings, and Qwen2.5-VL uses TMRoPE, a full 3D scheme.

Warm-up0/3

1. A vision-language model turns each video frame into a grid of visual tokens for the LLM. When you feed it many frames from a long video, what constraint do you hit first?Warm-up

A filmstrip of video frames, each frame split into a small grid of visual tokens, all the tokens laid out in one row that feeds an LLM box. — Typed schematic. Grid sizes are drawn small for space.

2. In a transformer, what does a position embedding add to the token inputs?Warm-up

Three tokens The, cat, sat. For each, a row with its 4-number embedding, a row labelled p0, p1 or p2 with a 4-number position vector, and a row with their element-wise sums. — Token vectors are made up. Position vectors are sin and cos of the slot number (computed), sums computed.

3. Two time intervals partly overlap. Intersection over union (IoU) divides the overlap length by what?Warm-up

Two horizontal bars on a 0 to 13 second axis: interval A of 10 s and interval B of 7 s, their 4 s overlap outlined, and the formula IoU equals overlap length divided by a question mark. — Illustrative intervals.

Check: The Problem0/2

1. Subsampling frames, pooling each frame and using a Q-former all cut visual tokens. What does each one cost?Explain

A three-row table of Subsample frames, Pool each frame and Q-former, each with a question mark under what it costs. — Typed schematic.

2. A 1-minute video at 30 FPS is encoded with ViT-B at 224px, which yields 196 patch tokens per frame. Roughly how many visual tokens does the full video produce?Apply

A filmstrip for 60 seconds at 30 frames per second, one frame split into a 14 by 14 grid of 196 patch tokens, and a dashed box asking for the visual tokens for the whole video. — Typed schematic.

The Concept

Video-LLaMA: Q-former per clip + audio branch

Video-LLaMA (2023) was the first open video-LLM. Architecture:

16-frame clips at 2 FPS (so 8 seconds).
Per-frame ViT features -> Video Q-former that cross-attends over all 16 frames -> 32 learned queries -> LLM.
Parallel audio branch: waveform -> ImageBind audio encoder -> Audio Q-former -> 32 queries -> LLM.

The strength is audio-visual joint reasoning. The weakness is a fixed clip length and no arbitrary time grounding.

VideoChat and Video-LLaVA

VideoChat kept the Video-LLaMA idea but dropped audio and simplified. Video-LLaVA (Lin et al., 2023) trained a single visual encoder on both images and video frames ("alignment before projection"), which gives a unified representation. Both are a frozen CLIP encoder, an MLP, and an LLM.

Neither handles long video. Both are 8-16 frame systems.

Qwen2.5-VL and TMRoPE

Qwen2.5-VL introduced TMRoPE (Temporal-Modality Rotary Position Embedding), where each patch token carries a (t, h, w) position and t is the actual timestamp, not the frame index.

Three things separate it from a simple temporal embedding:

Absolute time, not index. The model sees "at 4.2 seconds" instead of "at frame 15."
Per-token rotation, not per-clip. Each visual token rotates independently by its timestamp.
Compatibility with dynamic FPS. If you sample at 2 FPS here and 4 FPS there, TMRoPE handles the uneven spacing natively.

That is what makes "at what second does the cat jump?" answerable. The model can output "at 4.2 seconds," where Video-LLaMA could only say "early in the clip."

Frame sampling strategies

Uniform: sample N frames evenly over the duration. Simple, but it misses motion peaks.

Dynamic FPS: sample adaptively based on motion intensity. Optical flow or frame differencing picks the high-motion segments for denser sampling. Qwen2.5-VL trains on this.

Event-driven: run a lightweight detector and sample more where action happens. VideoAgent uses this.

Keyframe + context: sample at shot boundaries plus a few adjacent frames. Used for cinematic content.

Pooling per frame

At 1 FPS and 576 tokens per frame, a 5-minute clip is 172,800 tokens. That is more than Qwen2.5-VL-72B's 128k context, so you have to cut it down first, and it is expensive either way.

A 3x3 bilinear pool reduces each frame to 64 tokens, so 5 minutes comes to 19,200 tokens. That is the sweet spot for most tasks.

Pool harder (6x6 -> 16 tokens per frame) for agent workflows, where spatial detail matters less.

The four video benchmarks

VideoMME: comprehensive video understanding, short + medium + long.
TempCompass: fine-grained temporal reasoning, "before" / "after" questions.
EgoSchema: long-horizon first-person video.
Video-MMMU: multimodal multi-discipline video questions.

A full video-VLM evaluation hits all four because they stress different axes. TempCompass is about ordering, EgoSchema is about reasoning over 3+ minutes, and VideoMME spans durations.

Grounding output formats

Temporal grounding can come out in three formats:

Free text: "The cat jumps around the 4-second mark." Easy to read but imprecise.
Structured JSON: {"event": "jump", "start": 4.1, "end": 4.3}. Qwen2.5-VL trains this.
Token-based: special <time>4.1</time> tokens interleaved with the answer. This is Qwen2.5-VL's internal format.

Token-based output is the most accurate for downstream use, and Qwen2.5-VL's JSON output parses directly.

2026 best practice

For video VLMs in 2026:

Encoder: SigLIP 2 with M-RoPE or TMRoPE (Qwen2.5-VL).
Frame sampling: dynamic FPS (1-4 depending on motion) with a max-frame cap.
Per-frame pooling: 3x3 bilinear.
Output: structured JSON with time + event fields.
Benchmarks: VideoMME + TempCompass for general; EgoSchema for long-horizon.

Check: The Concept0/24

1. What is temporal grounding in a video VLM?Define

A question box asking when the cat jumps feeds a Video VLM box whose answer is a question mark, above a 10-frame strip of the input video with a seconds axis. — Typed schematic.

2. What is TMRoPE, as used by Qwen2.5-VL?Define

A pipeline of video frames, a vision encoder producing patch tokens, and an LLM, with a separate dashed box reading TMRoPE equals question mark below it. — Typed schematic. The pipeline is generic; TMRoPE is not placed in it.

3. What is dynamic FPS sampling?Define

A bar chart of ten motion values for seconds 0 to 9, low at seconds 0, 1, 5 and 6 and high at 2 to 4 and 8 to 9, with an empty question-mark box for the frames to sample. — Motion values are the list in the lesson's main.py.

4. What does frame pooling do before frames reach the LLM?Define

Grid size 576 tokens is the lesson's per-frame figure.

5. What is a video Q-former?Define

Typed schematic.

6. What does the VideoMME benchmark measure?Define

Four benchmark name boxes with VideoMME highlighted, above an evaluation loop of video plus question, Video VLM, answer and score, and the line VideoMME measures with a question mark. — Typed schematic.

7. What does Video-LLaVA's "alignment before projection" mean?Define

Two inputs, images and video frames, on the left; a dashed region reading encoder side with a question mark; then an MLP projector box and an LLM box. — Typed schematic.

8. Which of the four video benchmarks stresses before/after temporal ordering questions?Define

A timeline from 0 to 10 with event A and event B marked, and the question did A happen before or after B. — Typed schematic. Event positions are illustrative.

9. Which pairing of benchmark and defining trait is correct?Define

A two-row table listing EgoSchema and Video-MMMU, each with a question mark in the defining trait column. — Typed schematic.

10. What is Video-LLaMA's design for a video clip, and what is its main weakness?Explain

A video clip feeds a Video-LLaMA box that reads design with a question mark, then an LLM output box, with the line main weakness with a question mark below. — Typed schematic.

11. Why does TMRoPE set t to the actual timestamp instead of the frame index?Explain

A patch token card reading t equals question mark, above two boxes: the frame index such as 15, or the timestamp such as 4.2 seconds. — Typed schematic.

12. What does TMRoPE give a video-LLM that a simple learned temporal embedding table cannot?Explain

A table with rows position 0, 1, 2 each holding a learned vector, and a box on the right reading TMRoPE gives the model what this table cannot, with a question mark. — Typed schematic.

13. How do Video-LLaMA, Video-LLaVA and Qwen2.5-VL differ in how they handle time and frame compression?Explain

Three cards named Video-LLaMA, Video-LLaVA and Qwen2.5-VL, each with two empty fields: frame compression and handling of time, both showing question marks. — Typed schematic.

14. Why does a full video-VLM evaluation run all four benchmarks rather than only one?Explain

One Video VLM box in the middle linked to four benchmark boxes, VideoMME, TempCompass, EgoSchema and Video-MMMU, each with its score shown as a question mark. — Typed schematic.

15. How do event-driven and keyframe+context sampling differ from uniform and dynamic-FPS sampling?Explain

A four-row list: Uniform, Dynamic FPS, Event-driven and Keyframe plus context, each with a question mark under places frames using. — Typed schematic.

16. Why prefer structured JSON grounding output over free text like "around the 4-second mark"?Explain

A free-text answer, the cat jumps around the 4-second mark, above a dashed box with a question mark for the structured JSON version of the same answer. — Typed schematic. The sentence is the lesson's free-text example.

17. When does 6x6 pooling (16 tokens per frame) make sense over the 3x3 default?Explain

Three token grids for one frame: 24 by 24 with no pooling (576 tokens), 8 by 8 after 3x3 pooling (64 tokens), 4 by 4 after 6x6 pooling (16 tokens). — Token counts follow the lesson's pooling figures.

18. At 1 FPS with 576 tokens per frame, a 5-minute clip is 172,800 tokens. After 3x3 bilinear pooling to 64 tokens per frame, what is the new total?Apply

An unpooled clip at 576 tokens per frame and 172,800 tokens, a 3x3 bilinear pool, 64 tokens per frame, and a dashed box asking for the new total. — Typed schematic. All figures are from the stem.

19. At 1 FPS with 576 tokens per frame, a 5-minute clip is 172,800 tokens. With 6x6 pooling to 16 tokens per frame, what is the total?Apply

An unpooled clip at 576 tokens per frame and 172,800 tokens, a 6x6 bilinear pool, 16 tokens per frame, and a dashed box asking for the new total. — Typed schematic. All figures are from the stem.

20. Asked when the cat jumps, one video model answers "early in the clip" and another answers "4.2 seconds" on the same video. Which design difference best explains this?Analyze

One video clip box feeding two models. Model A answers early in the clip, model B answers 4.2 seconds. — Typed schematic.

21. A uniform 8-frame sampler on a 10-second clip misses a half-second burst of action. Which property of uniform sampling explains this?Analyze

A 10 second axis with eight frame markers at 0.625, 1.875, 3.125, 4.375, 5.625, 6.875, 8.125 and 9.375 seconds. — Sample times from uniform_sample in main.py.

22. You must summarize a 90-minute film with many scene cuts on a 128k-context model, and exact timestamps are not needed. Which sampling approach fits best?Design

A 0 to 90 minute axis with fourteen orange scene-cut ticks at irregular positions, and the notes: 128k context, summarize the film, exact timestamps not needed. — Typed schematic. Cut positions are illustrative.

23. You are building a monitor for 10-minute camera clips on a 128k-context model. Most seconds are static, a few contain short bursts of action, and users want the second each event happens. Which plan fits best?Design

A motion trace over ten minutes that is flat apart from three short bursts, under a title naming 10-minute clips and a 128k-context model. — Typed schematic. The trace is illustrative.

24. You are evaluating a video VLM for questions about 5-minute first-person recordings, and you also need to know whether it gets before/after ordering right. Which benchmarks matter most?Design

Two need cards, one about 5-minute first-person recordings and one about before and after ordering, above the four benchmark names in plain boxes. — Typed schematic.

Use It

code/main.py includes:

Uniform and dynamic-FPS frame samplers.
A toy temporal-grounding evaluator: each ground-truth event is matched by name to a predicted event, scored with IoU on the time interval, and counted as a hit at IoU 0.3 or above. It reports recall.
A printed table comparing Video-LLaMA (16 frames, Q-former), Video-LLaVA (8 frames, MLP), VILA-1.5, Qwen2.5-VL (dynamic FPS + TMRoPE), and LLaVA-OV-1.5 (pool, 32 frames).

Check: Use It0/5

1. In the toy grounding evaluator, the ground-truth `turn` event spans 6.0 to 6.5 s and the prediction spans 5.8 to 6.2 s. What IoU does it score?Apply

A zoomed timeline from 5.6 to 6.8 s with a ground-truth bar from 6.0 to 6.5 and a prediction bar from 5.8 to 6.2, and IoU equals a question mark. — Intervals are the turn event in the lesson's main.py.

2. The toy evaluator scores three events at IoU 0.57 (jump), 0.29 (turn) and 0.27 (sit). If the hit threshold is lowered from 0.3 to 0.25, what recall does it report?Apply

A table of three events with IoU 0.57, 0.29 and 0.27, a note that the hit threshold is lowered from 0.3 to 0.25, and recall equals a question mark. — IoU values are the lesson's toy run.

3. In `dynamic_sample`, a second has raw allocation 0.22 frames (budget 12, motion 0.1 of a 5.4 total) and the sampler computes `min(fps_cap, max(1, round(raw)))` with `fps_cap=4`. How many frames does that second get?Apply

Two lines of code from dynamic_sample, the values m 0.1, total_motion 5.4, total_budget 12 and fps_cap 4, raw 0.22, and a dashed box for the number of frames. — Code lines copied from main.py. Values are from the stem.

4. A dynamic sampler is given a budget of 12 frames for a 10-second clip but returns 15, and the extra tokens push a request over the context limit. What explains it, and what is the fix?Analyze

The call dynamic_sample with total_budget 12, and two bars: 12 frames asked for and a longer bar of 15 frames returned. — Frame counts from running main.py. Bar lengths are proportional.

5. A model's `turn` prediction is off by only 0.2 to 0.3 seconds, yet it scores IoU 0.29 and is counted as a miss at a 0.3 threshold. What is the right reading?Analyze

A number line from 0 to 0.6 with a dashed threshold at 0.3 and the turn event's IoU of 0.29 just left of it, under the note that the prediction is off by only 0.2 to 0.3 seconds. — Values from the stem.

Ship It

This lesson produces outputs/skill-video-vlm-frame-planner.md. Given a video task (monitoring, action recognition, temporal grounding, summarization), it picks the frame sampler, pooling factor, output format, and expected accuracy tier.

Author's Note

I'd run main.py before trusting any of the architecture talk above. There is no model in it, only the two samplers and the IoU scorer, and that is enough to see where the real decisions live.

Look at the dynamic sampler output first. I asked for a budget of 12 frames and it returned 15, because max(1, round(raw)) gives every low-motion second at least one sample on top of whatever the high-motion seconds earn. If you need a hard token cap, you have to enforce it after sampling.

The grounding eval is the other thing I'd stare at. With the threshold at IoU 0.3, the turn prediction scores 0.29 and the sit prediction scores 0.27, so recall comes out at 0.33 even though the turn prediction is off by 0.2 to 0.3 seconds and the sit prediction by under a second. Short events punish small timing errors hard. I'd pick the tolerance with the event length in mind before comparing two models on one number.

One caveat: the architecture table in main.py is just printed rows. Nothing there implements a Q-former or TMRoPE, so treat it as a summary of the reading and not as a benchmark.

Exercises

For a 3-minute cooking demo, pick uniform vs dynamic FPS. Justify with a token count.
TMRoPE adds what specifically that a simple temporal embedding table cannot do?
Write a JSON schema for temporal grounding that a VLM can learn to emit. Include error cases.
Read Video-LLaVA's Section 3 on "Alignment Before Projection." Why is this better than training separate image and video encoders?
Given the VideoMME leaderboard, what is the gap between the top open model and the top proprietary model as of 2026? How much of that gap is attributable to temporal encoding vs base LLM scale?

Key Terms

Term	What people say	What it actually means
Temporal grounding	"Time-localized answers"	VLM outputs a specific timestamp range for when an event happens
TMRoPE	"Time-Multimodal RoPE"	3D rotary position with absolute timestamps, used by Qwen2.5-VL
Dynamic FPS	"Motion-aware sampling"	Sample more frames in high-motion segments, fewer in static ones
Frame pooling	"Spatial compress per frame"	Reduce patches per frame with bilinear interpolation before the LLM
Video Q-former	"Clip compressor"	Cross-attention bottleneck mapping N frames to K learned queries
VideoMME	"Video bench"	Comprehensive short/medium/long video benchmark, 2500+ samples