CapstonePython (pipeline), TypeScript (UI)

Capstone 12 — Video Understanding Pipeline (Scene, QA, Search)

Twelve Labs productized Marengo + Pegasus, VideoDB shipped a CRUD API for video, AI2's Molmo 2 opened up VLM checkpoints, Gemini reads hours of video natively, and TimeLens-100K defined temporal grounding at scale. The 2026 pipeline has settled: scene segmentation, per-scene caption + embedding, transcript alignment, a multi-vector index, and a query that answers with (start, end) timestamps plus frame previews. The capstone is ingesting 100 hours, hitting public benchmarks, and measuring hallucination on counting and action questions.

TIME: 30 hoursPREREQUISITES: Phase 4 (CV), Phase 6 (speech), Phase 7 (transformers), Phase 11 (LLM engineering), Phase 12 (multimodal), Phase 17 (infrastructure)

Learning Objectives

Build the ingest side of a video index: scene segmentation, Whisper word-timestamp ASR, and VLM captioning, joined on (scene_id, time range) into a Qdrant collection with three named vectors per scene.
Run the query path: three dense queries merged with reciprocal rank fusion, temporal grounding to refine the (start, end) window inside the top scene, and a VLM synthesizer that answers with cited timestamps.
Evaluate on ActivityNet-QA, NeXT-GQA, and a 100-query custom set, reporting counting and action-type accuracy separately from descriptive questions.
Score the system against the rubric: temporal grounding IoU, QA accuracy, ingest throughput in hours of video per dollar, citation UX, and hallucination rate.

Problem

A video understanding pipeline turns long-form video into a corpus you can query in plain language, and answers with the moment in the footage that supports the answer: a (start, end) timestamp plus frame previews. Gemini 2.5 Pro can read a 2-hour video natively, but that doesn't help when you need to search 100 hours at once. That takes a scene-level index.

The production shape has four parts. Scene segmentation (TransNetV2 or PySceneDetect) cuts the video up. A VLM (Gemini 2.5, Qwen3-VL-Max, or Molmo 2) captions each scene. Whisper-v3-turbo supplies a transcript with word timestamps. A multi-vector index stores caption, frame embedding, and transcript side by side for every scene.

Evaluation uses two public benchmarks (ActivityNet-QA, NeXT-GQA) plus your own 100-query custom set. Hallucination on counting and action-type questions is the known-hard failure class, so the capstone measures it explicitly instead of hoping it averages out.

Warm-up0/3

1. What does automatic speech recognition (ASR) produce from an audio track?Warm-up

2. In dense retrieval, how is a query embedding usually compared against stored embeddings?Warm-up

3. What does intersection-over-union (IoU) measure between a predicted region and a ground-truth region?Warm-up

Check: Problem0/3

1. What is counting hallucination?Define

2. What is ActivityNet-QA used for in this capstone?Define

3. Why is a scene-level index still needed when a long-context VLM can read a 2-hour video natively?Explain

Concept

A multi-vector video index stores several vectors per scene, one for each way a scene can be described, so a query can match on what is shown, what is said, or what a model wrote about it. Three pipelines run in parallel at ingest. Scene segmentation cuts the video into scenes. VLM captioning generates a caption per scene and a frame embedding from a keyframe. ASR alignment produces word-level timestamps. The three streams join on (scene_id, time range), and each scene ends up with three vectors in Qdrant: caption embedding, keyframe embedding, and transcript embedding.

At query time the natural-language question fires against all three vectors, and the results merge with RRF. A temporal-grounding adapter (TimeLens-style) then refines the (start, end) window inside the top scene. The VLM synthesizer (Gemini 2.5 Pro or Qwen3-VL-Max) takes the query, the top scenes, and cropped frames, and answers with cited timestamps and a frame preview.

The hallucination measurement is the part people skip. Counting ("how many people enter the room?") and action-type ("does the chef pour before stirring?") questions are notoriously unreliable for VLMs, so report their accuracy separately from descriptive questions. A single blended number hides exactly the failure you are trying to see.

Check: Concept0/21

1. What is scene segmentation in a video pipeline?Define

2. What is a multi-vector index, as used for video scenes?Define

3. What is a frame embedding in this pipeline?Define

4. What is RRF fusion?Define

5. What is temporal grounding?Define

6. What does the ASR alignment stream provide for indexing?Define

7. What does the VLM captioning pipeline produce for each scene?Define

8. What does the VLM synthesizer take in, and what does it return?Define

9. Why does each scene get three vectors instead of one caption embedding?Explain

10. Why do the three ingest streams join on (scene_id, time range)?Explain

11. Retrieval already returns the best scene. Why run temporal grounding after it?Explain

12. Why report counting and action-type accuracy separately from descriptive accuracy?Explain

13. In main.py each stream adds 1 / (60 + rank + 1) to a scene's fused score, with rank starting at 0. What score does a scene get if it ranks first in all three streams?Apply

14. main.py calls ground_window with the query 'what happened first pour or stir' on a scene spanning 02:12-02:50 whose transcript is 'first we pour then we stir it slowly'. Which window comes back?Apply

15. main.py calls ground_window with the query 'how many cars pass through the intersection' on a scene spanning 00:00-00:32 whose transcript is 'we start here in tokyo'. What does it return?Apply

16. main.py calls ground_window with 'plating of the dish' on a scene spanning 02:50-03:30 whose transcript is 'plated presentation of the dish'. Which window comes back?Apply

17. The right scene is always retrieved and the answer is correct, but the cited (start, end) spans the whole scene instead of the moment. Which step is not doing its job?Analyze

18. After tuning retrieval, overall QA accuracy rises but counting accuracy stays flat. What does this suggest?Analyze

19. A user asks about "pouring," the chef says "now I pour the stock" but the scene caption never mentions it. Which design choice lets retrieval still find the scene?Analyze

20. Users want to ask 'what did Alice say about the budget?' across indexed meetings. Which addition fits this pipeline?Design

21. You adapt the pipeline to cooking videos and must build the evaluation set. Which plan fits?Design

Architecture

video file / URL
      |
      v
PySceneDetect / TransNetV2  (scene segmentation)
      |
      +--- per-scene keyframe --- VLM caption + frame embedding
      |                            (Gemini 2.5 Pro / Qwen3-VL-Max / Molmo 2)
      |
      +--- audio channel --- Whisper-v3-turbo ASR + word timestamps
      |
      v
multi-vector Qdrant: {caption_emb, keyframe_emb, transcript_emb}
      |
query:
  dense queries against all three -> RRF merge -> top-k scenes
      |
      v
TimeLens / VideoITG temporal grounding (refine start/end within scene)
      |
      v
VLM synth: query + top scenes + frame previews
      |
      v
answer + (start, end) timestamps + frame thumbs + citations

Stack

Scene segmentation: TransNetV2 (state-of-the-art 2024-26) or PySceneDetect
ASR: Whisper-v3-turbo via faster-whisper with word timestamps
VLM captioner + answerer: Gemini 2.5 Pro or Qwen3-VL-Max or Molmo 2
Temporal grounding: TimeLens-100K-trained adapter or VideoITG
Index: Qdrant with multi-vector support (caption / frame / transcript)
UI: Next.js 15 with HTML5 video player and scene thumbnails
Eval: ActivityNet-QA, NeXT-GQA, custom 100-question hand-labeled set
Hallucination benchmark: counting and action-type subsets with hand labels

Build It

Ingest walker. Accept YouTube URLs or local MP4s. Downscale to 720p if needed. Persist {video_id, file_path}.
Scene segmentation. Run TransNetV2 or PySceneDetect to produce [{scene_id, start_ms, end_ms, keyframe_path}]. At 100 hours, expect roughly 6k-8k scenes.
ASR pass. Run Whisper-v3-turbo on the audio, export word-level timestamps, and split the transcript into per-scene slices.
VLM captioning. For each scene, call Gemini 2.5 Pro (or Qwen3-VL-Max) with the keyframe and a short caption template. Produce a caption and a frame embedding.
Multi-vector index. Create a Qdrant collection with three named vectors. Payload: {video_id, scene_id, start_ms, end_ms, keyframe_url}.
Query. The question fires three dense queries. Merge them with reciprocal rank fusion and keep the top-k=5 scenes.
Temporal grounding. Run a TimeLens-style adapter on the top scene to refine the (start, end) window within it.
VLM synth. Call Gemini 2.5 Pro with the query, the top-3 scene clips (as images or short clips), and their transcripts. Require (video_id, start_ms, end_ms) citations.
Eval. Run ActivityNet-QA and NeXT-GQA, and build a 100-query custom set. Report overall accuracy plus a per-class breakdown (counting, action, descriptive).

Check: Build It0/8

1. Why does each stored scene carry a payload of video_id, scene_id, start_ms, end_ms and keyframe_url?Explain

2. Why does the synthesizer have to cite (video_id, start_ms, end_ms) with each answer?Explain

3. Which ingest work drives the cost that the throughput criterion measures at 100 hours of video?Explain

4. At 100 hours of video the lesson expects roughly 6k-8k scenes. About how long is the average scene?Apply

5. A 100-hour library yields about 6k-8k scenes, each indexed with the three named vectors. About how many vectors are stored?Apply

6. A job starts with all four stages (chunk 1200 ms, embed 2400 ms, index 800 ms, qa 1600 ms) pending. advanceJob runs 2000 ms after it was created. What are the stage statuses?Apply

7. A job's stage statuses are done, done, pending, pending, with none running or errored. What does overallStatus return?Apply

8. Retrieval keeps the top 5 scenes after RRF, and the moment that answers the question sits in the 4th-ranked scene. The final answer never mentions it. Which step explains this?Analyze

Use It

$ video-qa ask --url=https://youtube.com/watch?v=X "how many cars pass the intersection in the first minute?"
[scene]    23 scenes detected
[asr]      transcript complete, 4m12s
[index]    69 vectors written (23 scenes x 3)
[query]    top scene: scene 3 [00:08-01:04], confidence 0.84
[ground]   refined window: [00:12-00:58]
[synth]    gemini 2.5 pro, 1.4s
answer:    5 cars pass the intersection between 00:12 and 00:58.
citations: [scene 3: 00:12-00:58]
          [frame preview at 00:14, 00:27, 00:44, 00:51, 00:57]

Check: Use It0/2

1. In the Use It run, the top scene is [00:08-01:04] and the refined window is [00:12-00:58]. How long is the refined window compared with the scene?Apply

2. Your pipeline answers '5 cars pass the intersection' and users must be able to catch a wrong count. Which design fits?Design

Ship It

The deliverable is outputs/skill-video-qa.md. Given a YouTube URL or an uploaded video, the pipeline indexes scenes and answers questions with timestamped citations.

Weight	Criterion	How it is measured
25	Temporal grounding IoU	Intersection-over-union on held-out grounding set
20	QA accuracy	NeXT-GQA and custom 100-query
20	Ingest throughput	Hours of video per dollar spent
20	UI and citation UX	Timestamp links, thumbnail strip, jump-to-frame
15	Hallucination rate	Counting and action-type accuracy separately
100

Check: Ship It0/4

1. The rubric needs a measure of how expensive indexing is. Which criterion fits, and in what unit?Explain

2. Using the Ship It rubric, a build earns full marks on temporal grounding IoU and on hallucination rate, and zero on everything else. What is its score out of 100?Apply

3. A predicted grounding window is 00:10-00:40 and the labeled window is 00:20-00:50. What is the temporal IoU?Apply

4. Indexing a 100-hour library costs too much. Which approach fits the rubric?Design

Author's Note

I'd label the counting and action subsets first, before tuning retrieval, so you have a baseline for the failure class the Problem section calls known-hard. Otherwise the 100-query set fills up with descriptive questions, which are the easy ones, and the overall accuracy number looks better than the pipeline deserves.

The rubric gives hallucination rate 15 points, the smallest weight, which makes it easy to treat as an afterthought. I'd build Exercise 3's "counting strict" mode into the first version instead. The Use It example answers "5 cars" with five frame previews attached, and I'd want each of those counted instances clickable, so a wrong count is something a user can catch rather than something they have to trust.

Exercises

Swap Gemini 2.5 Pro for Qwen3-VL-Max on the captioning pass. Report caption quality delta on a human-rated 50-scene sample.
Reduce per-scene frame embedding to one pooled vector instead of multi-vector. Measure the retrieval regression.
Build a "counting strict" mode: the synthesizer extracts each counted instance with a timestamp and the user clicks to verify. Measure whether user-verification reduces hallucination.
Benchmark ingest cost: hours-of-video-per-dollar across three VLM choices. Pick the sweet spot.
Add speaker-diarized transcript: run pyannote speaker diarization on the audio and embed per-speaker transcripts. Demonstrate "what did Alice say about X?" queries.

Key Terms

Term	What people say	What it actually means
Scene segmentation	"Shot detection"	Cutting video into scenes at shot boundaries
Multi-vector index	"Caption + frame + transcript"	Qdrant collection with named vectors per representation
Temporal grounding	"When exactly did it happen"	Refining the (start, end) window for a query answer
Frame embedding	"Visual representation"	A vector embedding of a keyframe; used for scene-visual similarity
RRF fusion	"Reciprocal rank fusion"	Merge strategy across multiple ranked lists; a classic hybrid-retrieval trick
Counting hallucination	"Miscount"	Known failure mode of VLMs on "how many X" questions
ActivityNet-QA	"Video-QA benchmark"	Long-form video QA accuracy benchmark