Qwen-VL Family and Dynamic-FPS Video

Qwen-VL shipped in August 2023 as a direct response to LLaVA-1.5 and BLIP-2. The gap the Qwen team targeted was threefold: resolution, video, and structured output.

Resolution: LLaVA-1.5 ran at 336x336. Fine for photos, too coarse for a Chinese-language invoice or a dense spreadsheet screenshot. Qwen-VL's first innovation was 448x448 and grounded bounding-box output, letting the model point at things.

Video: Video-LLaMA stacked per-frame encoders and fed them to the LLM. It worked for short clips, not for multi-minute videos where the temporal axis is the signal. The Qwen team wanted a single encoder that understood time.

Structured output: LLaVA emitted free-form text. An agent needs JSON. Qwen-VL trained on explicit JSON output formats including bounding-box coordinates as text.

Each Qwen-VL generation pushes on at least one of these three axes.

Qwen-VL (August 2023)

The first generation: OpenCLIP ViT-bigG/14 as encoder (2.5B params), LLama-compatible Q-Former (1-step with 256 queries), Qwen-7B base. Contributions:

• 448x448 resolution (then SOTA for an open VLM).
• Grounding: trained on image-text pairs with explicit coordinate-token output. "The cat is at (112, 204), (280, 344)".
• Chinese + English multilingual training from the start.

Benchmarks at the time: competitive with GPT-4V on English, strong on Chinese. The grounding supervision was the real headline.

Qwen2-VL (September 2024): M-RoPE and native resolution

Qwen2-VL replaced the fixed-resolution + Q-Former stack with a natively dynamic-resolution ViT encoder. Key changes:

• Native dynamic resolution. The ViT accepts any HxW divisible by 28 (patch 14 with 2x spatial merge). An image at 1120x672 (40x24 merged patches) produces 960 visual tokens. No resize, no tiling, no thumbnail.
• M-RoPE (Multimodal RoPE). Each token carries a 3D position (t, h, w) instead of 1D. For images t=0, for video t = frame_index. RoPE rotates query/key vectors by a frequency per axis. No positional embedding table.
• MLP projector. Drop the Q-Former; use a 2-layer MLP on the merged patch tokens.
• Video with dynamic FPS. Video sampled at 1-2 FPS by default, but the model accepts arbitrary frame counts.

Result: Qwen2-VL-7B matched GPT-4o on several multimodal benchmarks and beat it on DocVQA (94.5 vs 88.4). The architecture change did most of the work.

Qwen2.5-VL (February 2025): dynamic FPS and absolute time

Qwen2.5-VL's big shift was video. Dynamic FPS is more than "sample more frames when needed." The paper formalized several pieces:

• Absolute time tokens. Instead of positional indices (frame 0, 1, 2...), use actual timestamps. "At 0:04, the cat jumps." The model sees <time>0.04</time> tokens interleaved with frame tokens.
• Dynamic FPS. Sample at 1 FPS for slow footage, 4+ FPS for action. The user or trainer chooses; M-RoPE adapts.
• Window attention in ViT. Spatial attention is windowed (local within blocks) for throughput; global attention every few layers.
• Explicit JSON output format. Trained on tool-call data: "{"tool": "click", "coords": [380, 220]}". Agent-ready out of the box.
• MRoPE-v2 scaling. Positions scale with max input size so a 10-minute video does not run out of frequency range.

Benchmarks: Qwen2.5-VL-72B beats GPT-4o on most video benchmarks, matches Gemini 2.0 on documents, and sets the open-model SOTA for GUI grounding (ScreenSpot: 84% accuracy vs 38% for GPT-4o).

Qwen3-VL (November 2025)

Qwen3-VL is an incremental upgrade that consolidates rather than reinvents: larger LLM backbone (Qwen3-72B), expanded training data, improved OCR, stronger reasoning via the Qwen3 "thinking mode." The ViT and M-RoPE stay. The paper focuses on data and training improvements over architecture.

The lineage takeaway: by 2025 the Qwen-VL architecture had stabilized. Additional generations scale compute and data, not primitives.

M-RoPE mathematically

Classical RoPE rotates a query q of dimension d by position m using paired coordinates:

q_rot[2i]   = q[2i]   * cos(m * theta_i) - q[2i+1] * sin(m * theta_i)
q_rot[2i+1] = q[2i]   * sin(m * theta_i) + q[2i+1] * cos(m * theta_i)
theta_i     = 10000^(-2i/d)

M-RoPE splits the hidden dim into three bands. Say d = 96. Assign 32 dims to temporal, 32 to height, 32 to width. Each band rotates by its own axis position. A patch at (t=5, h=10, w=20) gets rotations R_t(5), R_h(10), R_w(20) applied to its three bands.

Text tokens use t = text_index, h = 0, w = 0 (or a normalized choice), keeping compatibility. Video frames use t = frame_time, h = row, w = col. Single images use t = 0.

The benefit: one position encoding handles text, image, and video without branching code or different position tables.

Dynamic-FPS sampling logic

Given a video of duration T seconds and a target-tokens budget B:

1. Compute the maximum FPS you can afford: fps_max = B / (T * tokens_per_frame).
2. Pick a candidate FPS list from the motion level. High motion searches [8, 4, 2, 1, 0.5, 0.25], medium searches [4, 2, 1, 0.5, 0.25], and low searches [1, 0.5, 0.25, 0.1]. Motion can come from an optical-flow heuristic or an explicit user request.
3. Take the first candidate with fps <= fps_max. If none fits, fall back to the lowest one.
4. Sample uniformly at the chosen FPS; in Qwen2.5-VL, <time>t</time> tokens go between frames.

Qwen2.5-VL trains this logic implicitly; at inference the user controls via fps parameter. A 60-second action sequence at 4 FPS with 81 tokens per frame = 19440 tokens, manageable in a 32k context.

Structured agent output

Qwen2.5-VL's agent training explicitly targets structured tool calls:

{
  "tool": "mouse_click",
  "coords": [1024, 512],
  "button": "left",
  "modifier": null
}

Parsing is one JSON.parse call over the model's output, provided the output is valid JSON. Compare to free-form "click at (1024, 512)" which required regex and ambiguity handling. That shift is part of why Qwen2.5-VL's ScreenSpot score jumped from Qwen2-VL's 55% to 84%.

main.py implements:

• M-RoPE position computation for a packed sequence mixing text, image patches, and video frames.
• Dynamic-FPS sampler: given (duration, budget, motion_level), pick FPS and emit frame timestamps.
• A toy Qwen2.5-VL JSON-output parser that handles tool-call responses with coordinate fields.

Run it and read the sampler table. The same 32768-token budget gives a 30-second tennis rally a much higher FPS than a 10-minute security loop.

This lesson produces outputs/skill-qwen-vl-pipeline-designer.md. Given a video task (monitoring, agent, action recognition, accessibility), it emits the Qwen2.5-VL configuration (frame budget, FPS strategy, window-attention flag, agent-output mode) and a latency estimate. Use this whenever you deploy a Qwen-VL-family model for a video product.

1. Compute M-RoPE rotations for a patch at (t=3, h=5, w=7) with hidden 48 (16 per band, base theta 10000). Show the rotation angles for the first three pairs in each band.

2. A 10-minute security-camera recording at 1 FPS produces how many frames? At 384 resolution with 3x pool, how many total tokens? Does Qwen2.5-VL's default 32k context handle it?

3. Pick FPS for a 30-second tennis rally vs a 30-second recipe demo vs a 30-second UI-agent recording. Justify each with the dynamic-FPS logic.

4. Qwen2.5-VL drops the Q-Former entirely. Why does a simple MLP work in 2025 but not in 2023? (Hint: data scale and encoder quality.)

5. Parse three Qwen2.5-VL JSON tool-call outputs into Python dicts. What fails for malformed JSON and what recovery strategy does the Qwen cookbook recommend?

• Bai et al.: Qwen-VL (arXiv:2308.12966)
• Wang et al.: Qwen2-VL (arXiv:2409.12191)
• Qwen Team: Qwen2.5-VL Technical Report (arXiv:2502.13923)
• Qwen Team: Qwen3-VL (arXiv:2511.21631)
• Zhu et al.: InternVL3 (arXiv:2504.10479)