BuildPython

Long-Video Understanding at Million-Token Context

A 1-hour 4K video at 24 FPS, patched and embedded, comes out to about 60 million tokens. A 2-hour podcast episode transcribed is 30,000. A full Blu-ray feature, even with aggressive pooling, lands in the hundreds of thousands. Gemini 1.5 (March 2024) opened this era with a 10-million-token context and reliable needle-in-a-haystack recall over hour-long videos. LWM (Liu et al., February 2024) showed the ring attention path to scale. LongVILA and Video-XL pushed ingestion further, and VideoAgent swapped raw context for agentic retrieval. Each approach trades compute, recall, and engineering complexity differently, so this lesson reads them side by side.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 17 (video temporal tokens)

Learning Objectives

Compute total visual-token counts for long-form video at varying FPS and pooling.
Explain the three scaling paths: brute context (Gemini 1.5), ring attention (LWM), token compression (LongVILA / Video-XL).
Compare raw-context video VLMs vs agentic-retrieval video VLMs (VideoAgent) on accuracy and latency.
Design a needle-in-a-haystack test for a 30-minute video and measure recall at a specific minute.

The Problem

One frame of Qwen2.5-VL-sized patches at 384 native resolution is about 729 tokens. With 3x3 pooling that drops to 81. A 30-minute clip at 1 FPS is 1800 frames, or 145,800 tokens. A 2025 open VLM can take that, but it is tight. At 2 FPS the count doubles to 291,600, and only the biggest contexts hold it.

A 2-hour movie at 1 FPS is 583k tokens. That is beyond most 2026 open models, so you either need Gemini 2.5 Pro or you pool more aggressively.

Three scaling paths emerged, plus a fourth that avoids the problem by not reading the whole video.

Warm-up0/2

1. A video encoder emits a fixed number of tokens per frame. You raise the sampling rate from 1 FPS to 2 FPS. What happens to the total visual token count?Warm-up

Two rows for a 6-second clip. The 1 FPS row has six frame boxes and the 2 FPS row has twelve frame boxes, each marked k for the fixed token count per frame, with a question mark for each row's total. — Illustrative 6-second clip. k stands for the fixed tokens per frame.

2. Pooling 3x3 neighborhoods of patch tokens in each frame cuts the per-frame token count by what factor?Warm-up

A 6 by 6 grid of patch tokens with orange outlines around each 3 by 3 block, and a dashed box asking for the tokens per frame after pooling. — Illustrative 6x6 grid.

Check: The Problem0/1

1. Why do both a lower FPS and heavier pooling let a long video fit in a context window?Explain

Two boxes, sampling rate (FPS) and pooling, with arrows into a box comparing a long video to a context window, and a dashed box asking why lowering either helps. — Typed schematic, not captured output.

The Concept

Path 1: Brute context (Gemini 1.5, Claude Opus)

Throw hardware at the problem. Scale context to millions of tokens, process everything in one forward pass.

Gemini 1.5 Pro launched with 1M tokens; Gemini 1.5 Ultra to 10M; Gemini 2.5 Pro in 2026 does hours of video reliably. The paper (arXiv:2403.05530) documents needle-in-a-haystack recall at 99.7% up to ~9.5M tokens.

Engineering: a custom attention implementation with memory hierarchy (local + global + sparse) plus MoE expert routing for long-context efficiency. Not published in full detail. Not open-source.

Path 2: Ring attention (LWM, LongVILA)

Ring attention distributes long sequences across devices in a "ring" where each device holds a chunk. Attention across the full sequence happens by each device sending its chunk to the next in a ring pattern, computing partial attention, and aggregating.

LWM (Liu et al., 2024) trained a 1M-token context model this way. Training compute scales linearly with context, not quadratically, because the quadratic hit on attention is amortized across the ring's devices.

LongVILA (arXiv:2408.10188) adapted the pattern to VLMs. 1400-frame videos at 192 tokens per frame = 268k context, trained with ring attention across 8-way parallelism.

Path 3: Token compression (Video-XL, LongVA)

Cheaper than brute context: compress aggressively before the LLM sees the sequence.

Video-XL (arXiv:2409.14485) uses a visual summary token: each clip of N frames produces a single "summary" token that attends over the N. At inference, the LLM sees one summary token per clip, drastically shrinking the context.

LongVA extends LLM context from 200k to 2M with a "long context transfer" technique. Train on long-context text, transfer to long-context video via shared representation.

Token compression trades off recall at specific timestamps for scalability. The model knows generally what happened but sometimes misses exact frames.

Path 4: Agentic retrieval (VideoAgent)

Do not feed the full video to the LLM. Instead, treat the video as a database and use an LLM to query it.

VideoAgent (arXiv:2403.10517):

LLM reads the question.
LLM asks a retrieval tool for relevant clips ("show me segments with a cat").
Tool returns matching clip timestamps.
LLM reads those clips via a VLM.
LLM composes the answer or asks follow-up queries.

This is the LLM-as-agent pattern applied to long video. Cheaper inference (only relevant clips encoded), harder engineering (retrieval quality becomes the bottleneck).

Needle-in-a-haystack benchmarks

The standard long-context test: insert a unique visual or textual marker at a random point in the video, then ask a query that requires recalling it.

Metric: Recall@k across video length and marker position.

Gemini 2.5 Pro scores >99% recall at up to 90-minute videos. Open 72B models (Qwen2.5-VL-72B, InternVL3-78B) score ~85-90% at 30 minutes and degrade past 60.

VideoAgent can match or beat raw-context models at 2+ hours because retrieval hits the needle if the tool is good.

Which path to pick

For a 15-minute clip at frontier accuracy: open 72B + native context usually works. Pick Qwen2.5-VL-72B.

For 30-minute to 1-hour content: LongVILA or Video-XL for open; Gemini 2.5 Pro for closed. The quality bar matters here, and frontier means closed.

For 2+ hour content: VideoAgent or similar retrieval patterns. Alternatively, summarize to smaller chunks and feed hierarchical summaries.

2026 production pattern

In practice, production long-video pipelines are hybrid:

Run dynamic-FPS sampling + aggressive pooling on the entire video (get a 100k-token global representation).
Pass to a 72B VLM for a global summary.
If user asks detailed questions, run agentic retrieval using the summary as an index.

Brute context handles the global understanding and retrieval handles the local detail.

Check: The Concept0/22

1. What is brute context for long video?Define

A card titled brute context with rows for lesson section, examples (Gemini 1.5, Claude Opus) and source (arXiv:2403.05530), a bar for a 2-hour movie at about 583k tokens, and a dashed box asking for its meaning for long video. — Card built from the lesson's text.

2. What is ring attention?Define

A card titled ring attention with rows for lesson section, examples (LWM, LongVILA) and sources (two arXiv numbers), and a dashed box asking what it is. — Card built from the lesson's text.

3. What is token compression in long-video models?Define

A card titled token compression with rows for lesson section, examples (Video-XL, LongVA) and source (arXiv:2409.14485), and a dashed box asking what it does to the token sequence. — Card built from the lesson's text.

4. What is a needle-in-a-haystack test for a video model?Define

A card titled needle-in-a-haystack with rows for lesson section, abbreviation and the models it is run on (Gemini 2.5 Pro, Qwen2.5-VL-72B, InternVL3-78B), and a dashed box asking what the test does. — Card built from the lesson's text.

5. What is agentic retrieval for long video?Define

A card titled agentic retrieval with one row for the lesson section (Path 4 of 4 scaling paths), and a dashed box asking what it is. — Card built from the lesson's text.

6. What is VideoAgent?Define

A card titled VideoAgent with rows for authors (Wang et al.), paper (arXiv:2403.10517) and the model named in exercise 3 (Claude 3.5), and a dashed box asking what it is. — Card built from the lesson's text.

7. Gemini 1.5 reports about 99.7% needle-in-a-haystack recall up to roughly 9.5M tokens. What does that result say about brute context?Explain

A context-length axis from 0 to 10M tokens with a span up to about 9.5M tokens, a box with the reported recall of 99.7 percent, and a dashed box asking what the result says. — Numbers are the stem's.

8. How does ring attention (LWM's approach) scale training compute with context length, and why?Explain

Empty axes for training compute against context length with a tick at 1M tokens, a note that LWM trained a 1M-token context model, and a dashed box asking about scaling with length. — Typed schematic, not captured output.

9. Why does ring attention let a model train on 1M tokens when a single device could not?Explain

A bar for a 1M-token training sequence above a small box for one device's memory capacity, with a red edge showing the sequence does not fit, and a dashed box asking how ring attention gets around this. — Typed schematic, not captured output.

10. What does Video-XL's visual summary token do?Explain

A video split into three clips of five frames each, with a dashed box asking what Video-XL's visual summary token does. — Typed schematic, not captured output. Five frames per clip is illustrative.

11. In VideoAgent's agentic-retrieval pattern, what is the trade-off against brute-context approaches?Explain

Two pipelines. The brute-context one goes from whole video to long-context LLM to answer. The VideoAgent one goes from question to LLM to retrieval tool to VLM to answer. A dashed box asks for the trade-off against brute context. — Typed schematic, not captured output.

12. Why can agentic retrieval cost less than brute context on the same 2-hour video?Explain

A timeline of a 2-hour recording with ticks every 30 minutes, a box saying brute context is about 583,200 tokens at 1 FPS and 81 tokens per frame, and a box asking how agentic retrieval's cost compares. — Token figure computed with the lesson's tokens() function.

13. How do open 72B models (Qwen2.5-VL-72B, InternVL3-78B) compare to Gemini 2.5 Pro on needle-in-a-haystack recall?Explain

Empty chart axes for needle recall against video length with ticks every 15 minutes up to 90, a list of Gemini 2.5 Pro, Qwen2.5-VL-72B and InternVL3-78B, and a dashed box asking how they compare. — No values drawn.

14. Why does a needle test use a unique marker at a random point in the video?Explain

A video timeline in percent with one marked needle labelled unique sticker at about 40 percent, and a dashed box asking why the marker is unique and placed at a random point. — Typed schematic, not captured output. Position is illustrative.

15. The 2026 production pattern is hybrid. What does it combine?Explain

A long video box on the left, an answers box on the right, and a dashed box between them labelled hybrid pipeline. — Typed schematic, not captured output.

16. LongVILA trains on 1400-frame videos at 192 tokens per frame. About how long is that context?Apply

A row of frame boxes ending in an ellipsis for LongVILA training videos, lines giving 1400 frames and 192 tokens per frame, and a dashed box asking for the context length. — Typed schematic, not captured output. Numbers are the stem's.

17. A 45-minute lecture is sampled at 1 FPS with 81 tokens per frame. Roughly how many tokens is that, and which path does Which path to pick suggest?Apply

A 45-minute timeline with ticks every 15 minutes, a box giving 1 FPS and 81 tokens per frame, and a dashed box asking for the token count and the suggested path. — Settings are the stem's.

18. A compressed-token model describes the plot of a film well but misses what color a car was at 14:32. What is the most likely cause?Analyze

A film timeline in minutes with a mark at 14:32, and two boxes: asking for the plot gets a good answer, asking for the car color at 14:32 gets a missed answer. A dashed box asks for the most likely cause. — Typed schematic, not captured output. Timeline length is illustrative.

19. A VideoAgent-style system gives wrong answers on 2-hour videos even though its VLM reads clips correctly. Which part should you inspect first?Analyze

A row of five boxes: question, LLM, retrieval tool, VLM marked reads clips correctly, and answer marked wrong, with a note about 2-hour videos and a dashed box asking which part to inspect first. — Typed schematic, not captured output.

20. You must answer one-off questions like 'when does the cat jump?' over a 2-hour recording, and the only model you can host has an 80k context. Which path fits best?Design

A 2-hour recording timeline, a box saying the questions are one-off like when does the cat jump, a box saying the model that can be hosted has an 80k context, and a dashed box asking which path fits best. — Stem's facts only.

21. You must test whether a VLM recalls a detail from a 30-minute video. Which design measures recall at a specific minute?Design

A 30-minute timeline with ticks every 5 minutes and a goal line about recall of a detail at one specific minute, with a dashed box asking for the test design. — Typed schematic, not captured output.

22. Users upload 3-hour training videos. They first want an overview, then detailed questions about moments. You host a 72B VLM. Which design fits?Design

A 3-hour training video bar with ticks every 60 minutes, two boxes for the users' needs (an overview, then detailed questions about moments), a note that the hosted model is a 72B VLM, and a dashed box asking for the design. — Stem's facts only.

Use It

code/main.py does three things:

Computes token budgets for videos from 1 minute to 2 hours at 1 to 2 FPS, with 81 tokens per frame and one 32-token pooled row.
Simulates a needle-in-a-haystack run: inject a marker at a random timestamp, ask a question, score recall.
Runs an agentic-retrieval router simulator that picks specific clips to feed to a downstream VLM.

Run the budget table first and feel the scale gap.

Check: Use It0/5

1. The budget function multiplies duration, FPS and tokens per frame. With 81 tokens per frame, what does it give for a 20-minute video at 1 FPS?Apply

The two-line tokens function from main.py, three boxes giving a 20-minute video, 1 FPS and 81 tokens per frame, and a dashed box asking what tokens returns. — Function text copied from the lesson's main.py.

2. At 81 tokens per frame, how do a 1-hour video at 1 FPS and a 30-minute video at 2 FPS compare in total tokens?Apply

Two boxes. Video A is 1 hour at 1 FPS with 81 tokens per frame. Video B is 30 minutes at 2 FPS with 81 tokens per frame. A dashed box asks for the total tokens of A versus B. — Settings are the stem's.

3. The retrieval demo encodes 3 clips of 30 seconds at 1 FPS and 81 tokens per frame, plus a flat 200. What token count does that formula give?Apply

A 2-hour timeline with three thin orange clip marks, a code card noting 1 FPS, 81 tokens per frame and a flat 200, and a dashed box asking for the token count. — Captured by running the lesson's main.py. Clip positions come from the demo run; the rest is the stem's.

4. The needle demo prints a p(recall) for each model, and a teammate quotes one as that model's measured recall on 30-minute video. What is wrong with that claim?Analyze

A terminal showing four lines of the needle simulation printout, each with a model name, a needle time and a p(recall), and a box saying a teammate quotes the Qwen2.5-VL-72B 30-minute line as measured recall. — Captured by running the lesson's main.py. The section header line is left out.

5. The retrieval demo's trace line prints ~7290 clip tokens, but its total is 7,490. What explains the gap?Analyze

A terminal with the retrieval demo's output: the LLM, tool, VLM and LLM steps, then the total of about 7,490 tokens and the comparison with about 583,000 tokens for brute context, and a dashed box asking why the two token figures differ. — Captured by running the lesson's main.py. The last output line is left out.

Ship It

This lesson produces outputs/skill-long-video-strategy-planner.md. Given a video duration and query complexity, it picks between brute context, compression, and agentic retrieval, then estimates the latency and quality you should expect.

Author's Note

I'd run main.py and read the budget table before anything else. The arithmetic is one function, tokens(): duration times FPS times tokens per frame. The rows show how quickly 81 tokens per frame adds up once you leave the five-minute range. The "fits in" column is a label I wrote by hand per row, not something the code computes.

Treat the needle-in-a-haystack section as the shape of the test and nothing more. Each model gets a single trial with a random needle position and a hardcoded recall curve, so the printed p(recall) comes from numbers typed into the script. It does not measure a real model. To get a real number, run the design from exercise 2 against an actual VLM.

The retrieval demo is the same kind of toy. The three clip hits are random timestamps, and the token count is 3 clips of 30 seconds at 81 tokens per frame plus a flat 200, which is 7,490. The trace line printed above it says ~7290 clip tokens, and the total adds a flat 200 that the script does not break down. What carries over is the cost shape: a query encodes a few clips instead of the full 2 hours.

The closing strategy picker is just printed text. If you build the real planner from the Ship It skill, that is the part to replace with logic that takes duration and query type as input.

Exercises

A 45-minute lecture at 1 FPS, 81 tokens per frame. Total tokens? Fits in which models' contexts?
Design a needle-in-a-haystack test: at what minute do you inject the marker, and what is the exact query format?
Compare brute-context Qwen2.5-VL-72B (80k context) to VideoAgent (Claude 3.5 + retrieval) on a 1-hour video. Which wins on recall? Which wins on latency?
Ring attention's memory cost scales linearly in sequence length and linearly in device count. Explain why and what fails if you drop the ring-rotation phase.
Read Gemini 1.5 Section 5 on needle-in-a-haystack. What did the paper find about recall at the 1M vs 10M token boundary?

Key Terms

Term	What people say	What it actually means
Brute context	"Just more tokens"	Scale LLM context to millions of tokens; process everything in one pass
Ring attention	"LWM-style parallel"	Distributed attention pattern where each device holds a chunk and rotates
Token compression	"Summary tokens"	Reduce per-clip tokens via a learned compressor before the LLM
Needle-in-haystack	"NIH test"	Insert a unique marker at a random point, ask model to recall it at test time
Agentic retrieval	"LLM as query planner"	LLM asks a retrieval tool for relevant clips, reads them via a VLM, composes answer
VideoAgent	"Retrieval pattern for video"	Canonical agentic-retrieval design: question -> tool -> clip -> answer