Video Generation

A 10-second 1080p video at 24fps is 240 frames of 1920×1080×3 pixels. That's ~1.5 GB of raw data per clip. Pixel-space diffusion is infeasible. You need:

1. Spatiotemporal compression. A VAE that encodes videos, not frames, into a sequence of spatial-temporal patches.
2. Temporal coherence. Frames need to share content, lighting, and object identity over seconds. The net has to model motion.
3. Compute budget. Video training is 10-100x more expensive than image for the same model size.
4. Conditioning. Text, image (first-frame), audio, or another video. Most production models accept all four.

The architecture that solved this is the Diffusion Transformer (DiT) applied to spatiotemporal patches, trained on huge (prompt, caption, video) datasets. Same diffusion loss as Lesson 06.

Patchify

Encode the video with a 3D VAE (learned spatiotemporal compression). The latent is shape [T_latent, H_latent, W_latent, C_latent]. Split into patches of size [t_p, h_p, w_p]. For Sora-style models, t_p = 1 (per-frame patches) or t_p = 2 (every two frames). A 10-second 1080p video compresses to ~20,000-100,000 patches.

Spatiotemporal DiT

A transformer processes the flat sequence of patches. Each patch has a 3D positional embedding (time + y + x). Attention is usually factorized:

• Spatial attention within each frame's patches.
• Temporal attention across frames at the same spatial location.
• Full 3D attention is 16-100x more expensive; used only at low resolution or in research.

Text conditioning

Cross-attention with a large text encoder (T5-XXL for Sora, CogVideoX-5B uses T5-XXL). Long prompts matter. Sora's training set had GPT-generated dense re-captions averaging 200 tokens per clip.

Training

Standard diffusion loss (ε or v prediction) over spatiotemporal latents. Data: web video + ~100M curated clips + synthetic text captions. Compute: 10,000+ GPU hours for even a small research run; Sora-scale is 100,000+.

Open weights are closing the gap faster than in the image space: HunyuanVideo + WAN 2.2 LoRAs already power most open-source workflows by mid-2026.

code/main.py simulates the core spatiotemporal DiT idea: patchify a small synthetic video, add a per-patch position embedding, and denoise the whole sequence jointly with one small MLP that sees every patch at once (no attention in the toy). No numpy, pure Python. Temporal coherence shows up even in 1-D when the frames share a denoiser and carry position embeddings.

Step 1: patchify a synthetic 1-D "video"

T_FRAMES = 6
 
def make_video(rng):
    # a "video" is a sequence of 1-D values following a smooth trajectory
    base = rng.gauss(0, 1)
    slope = rng.gauss(0, 0.3)
    return [base + slope * t + rng.gauss(0, 0.05) for t in range(T_FRAMES)]

Step 2: position embedding per frame

def pos_embed(t, dim):
    return sinusoidal(t, dim)

Step 3: denoiser sees the whole sequence

Instead of denoising each frame independently, the tiny net concatenates all frame values and their position embeddings, then predicts the noise for all frames jointly.

Step 4: temporal coherence test

After training, sample a video. Measure the frame-to-frame delta. If the model has learned temporal structure, the deltas come out smaller than when each frame is sampled independently.

• Independent per-frame sampling = flicker. If you run image diffusion on each frame separately, the output flickers because each frame's noise is independent. Video diffusion fixes this by coupling the frames through attention or shared noise.
• Naive 3D attention = OOM. Full 3D attention on a 10-second 1080p latent is hundreds of billions of operations. Factorize into spatial + temporal.
• Data captioning matters more than size. Sora's main upgrade over prior work was training on ~10x more detailed captions (GPT-4 re-labelled clips). OpenAI's technical report says so directly.
• First-frame conditioning. Most production models also accept an image as the first frame. This is "image-to-video" mode, and training includes that variant.
• Physics drift. Long clips (>10s) accumulate subtle inconsistencies. Sliding-window generation + keyframe anchoring helps.

Cost per second of video at quality parity has dropped 20x between 2024 and 2026.

Save outputs/skill-video-brief.md. Skill takes a video brief (duration, aspect ratio, style, camera plan, subject consistency, audio) and outputs: model + hosting, prompt scaffolding (camera language, subject description, motion descriptors), seed + reproducibility protocol, and a frame-level QA checklist.

1. Easy. In code/main.py, compare frame-to-frame delta for (a) independent per-frame sampling, (b) joint sequence sampling. Report the mean and variance of the deltas.
2. Medium. Add a first-frame condition: pin frame 0 to a given value and sample the rest. Measure how the pinned value propagates.
3. Hard. Use HuggingFace diffusers to run CogVideoX-2B on a local GPU. Time 20 inference steps at 720p for a 6-second clip. Profile the spatiotemporal attention to identify the bottleneck.

A 10-second 1080p clip at 24 fps is 240 frames × 1920 × 1080 × 3 ≈ 1.5 GB of raw pixels. After a 4× video VAE compression (2 × spatial × 2 × temporal) the latent is ~100 MB per request. Run this through a spatiotemporal DiT for 30 steps at batch 1 and you are moving ~3 GB/step through HBM: memory bandwidth, not FLOPs, is the bottleneck.

Three production knobs, all from the inference chapter of the production-inference literature:

• TP across the DiT. Text-to-video models are routinely ≥10B params. TP=4 across 4 H100s is standard; PP=2 × TP=2 for 405B-class models. Latency per step drops roughly linearly with TP up to the all-reduce wall.
• Frame batching = continuous batching. At generation time, video is conceptually a batch of frames linked by attention. Continuous batching (in-flight scheduling) applies: start rendering frame t+1 while frame t-1 is being returned, if the model architecture allows sliding-window generation.
• Clip-level prefill cache. For image-to-video, the first-frame conditioning is analogous to an LLM's prompt prefill: compute it once, reuse across the temporal decoder passes. This is effectively a KV-cache for video.

• Brooks et al. (2024). Video generation models as world simulators: Sora technical report.
• Yang et al. (2024). CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer: CogVideoX.
• Kong et al. (2024). HunyuanVideo: A Systematic Framework for Large Video Generative Models: HunyuanVideo.
• Genmo (2024). Mochi-1 Technical Report: Mochi-1.
• Alibaba (2025). WAN 2.2: open SOTA mid-2025.
• Ho, Salimans, Gritsenko et al. (2022). Video Diffusion Models: the seminal video diffusion paper.
• Blattmann et al. (2023). Align your Latents (Video LDM): Stable Video Diffusion's ancestor.