World Models & Video Diffusion

A world model is a model that predicts what a scene looks like next, given what it looks like now and, often, what action gets taken. Video generation and world modeling turned out to be the same problem once models got good enough at the first one. Generate a coherent minute of video and you've implicitly learned how the world behaves: objects stay solid, things fall, actions have consequences, and a scene's visual style holds together from the first frame to the last. Condition that prediction on an action (walk left, open the door) and the video model becomes a simulator you can steer: a game engine, a driving simulator, a robotics test environment — all learned, none of it hand-coded.

The stakes are concrete. Genie 3 generates a playable environment from a single image. Runway GWM-1 Worlds synthesizes an explorable scene with no fixed edges. Sora 2 produces minute-long video with synchronized audio and physics that mostly holds together. NVIDIA Cosmos-Drive, Wayve Gaia-2, and Tesla DrivingWorld generate driving footage realistic enough to train on. Sim-to-real for robotics is quietly rebuilding itself on top of this same idea.

This is the big-picture lesson for Phase 4: it ties image generation, video understanding, and agentic reasoning together into the architecture pattern the field is converging on.

Three families of world modeling

• Sora 2 is pure video generation conditioned on a prompt. There's no action interface, and you can't steer it mid-rollout.
• Genie 3, GWM-1 Worlds, and Mirage / Magica are action-conditioned world models. They infer a latent action from observed video, then condition future frame predictions on that action, so pressing a key or moving a camera changes what the scene does next.
• DreamerV3 and the classic RL world-model family predict in a latent space with explicit action conditioning, trained on a reward signal. Less visual. More useful for sample-efficient RL.

Video DiT architecture

A video DiT is a diffusion transformer that treats a video clip the way an image DiT treats a picture, except the patches now span time as well as space. Cut the clip into small chunks along (T, H, W), turn every chunk into one token, and run the same denoising transformer from Lesson 23 over that longer token sequence.

Video latent:          (C, T, H, W)
Patchify (spatial):    grid of P_h x P_w patches per frame
Patchify (temporal):   group P_t frames into a temporal patch
Resulting tokens:      (T / P_t) * (H / P_h) * (W / P_w) tokens

Positional encoding is 3D: a rotary or learned embedding per (t, h, w) coordinate. Attention comes in three flavors:

• Full joint: every token attends to every other token. O(N^2) with N tokens, which gets prohibitive fast for long videos.
• Divided: alternate temporal attention (same spatial position, across time: (H*W) * T^2) and spatial attention (same timestep, across space: T * (H*W)^2). What TimeSformer and most video DiTs actually use.
• Window: local windows in (t, h, w). What Video Swin uses.

Every 2026 video diffusion model runs one of these three patterns, plus AdaLN conditioning (Lesson 23) and rectified flow.

Conditioning on actions: latent action models

Genie learns a latent action per frame by discriminatively predicting the action between a pair of consecutive frames. Its decoder then conditions on that inferred latent action, not on explicit keyboard keys. At inference, you can specify a latent action directly, or sample one from a fresh prior, and the model generates the next frame consistent with it.

Sora skips the action interface entirely. Its decoder predicts the next spacetime tokens from past spacetime tokens. The prompt sets the scene at the start; nothing steers it once generation begins.

Physical plausibility

Sora 2's 2026 release explicitly advertised physical plausibility: weight, balance, object permanence, cause-and-effect. The team measured it with hand-rated plausibility scores, and the model visibly improves on dropped objects, characters colliding, and failures on purpose (a missed jump) compared with Sora 1.

Plausibility is still the dominant failure mode. The 2024-2025 wave of videos showing people eating spaghetti or drinking from glasses exposed the same underlying problem: no persistent object representation. 2026 models (Sora 2, Runway Gen-5, HunyuanVideo) reduce this. They don't eliminate it.

Autonomous driving world models

Driving world models generate realistic road scenes conditioned on trajectories, bounding boxes, or navigation maps.

• Cosmos-Drive-Dreams (NVIDIA): generates minutes of driving video for RL training.
• Gaia-2 (Wayve): trajectory-conditioned scene synthesis for policy evaluation.
• DrivingWorld (Tesla): simulates varied weather, time of day, and traffic conditions.
• Vista (ByteDance): reactive driving scene synthesis.

They replace expensive real-world data collection for corner cases (pedestrian jaywalks at night, icy intersections, unusual vehicle types) that would otherwise require millions of miles of driving.

Robotics stack: VLM + video model + inverse dynamics

The emerging three-component robotics loop:

1. VLM parses the goal ("pick up the red cup") and plans a high-level action sequence.
2. Video generation model predicts observations N frames ahead: what executing each action would look like.
3. Inverse dynamics model extracts the concrete motor commands that would produce those observations.

This replaces reward shaping and sample-heavy RL. The world model does the imagining; the inverse dynamics model closes the loop on actuation. Genie Envisioner is one instantiation, and plenty of research groups are converging on the same structure.

Evaluation

• Visual quality: FVD (Fréchet Video Distance), user studies.
• Prompt alignment: CLIPScore per frame, VQA-style evaluation.
• Physical plausibility: hand-rated on a benchmark suite (Sora 2's internal benchmark, VBench).
• Controllability (for interactive world models): action → observation consistency; can you go back to a prior state?

Model landscape in 2026

Step 1: 3D patchify for video

import torch
import torch.nn as nn
 
 
class VideoPatch3D(nn.Module):
    def __init__(self, in_channels=4, dim=64, patch_t=2, patch_h=2, patch_w=2):
        super().__init__()
        self.proj = nn.Conv3d(
            in_channels, dim,
            kernel_size=(patch_t, patch_h, patch_w),
            stride=(patch_t, patch_h, patch_w),
        )
        self.patch_t = patch_t
        self.patch_h = patch_h
        self.patch_w = patch_w
 
    def forward(self, x):
        # x: (N, C, T, H, W)
        x = self.proj(x)
        n, c, t, h, w = x.shape
        tokens = x.reshape(n, c, t * h * w).transpose(1, 2)
        return tokens, (t, h, w)

A 3D conv with stride equal to kernel acts as the spatio-temporal patchifier: (T, H, W) -> (T/2, H/2, W/2) grid of tokens.

Step 2: 3D rotary position encoding

Rotary Position Embeddings (RoPE) separately applied along t, h, w axes:

def rope_3d(tokens, t_dim, h_dim, w_dim, grid):
    """
    tokens: (N, T*H*W, D)
    grid: (T, H, W) sizes
    t_dim + h_dim + w_dim == D
    """
    T, H, W = grid
    n, seq, d = tokens.shape
    if t_dim + h_dim + w_dim != d:
        raise ValueError(f"t_dim+h_dim+w_dim ({t_dim}+{h_dim}+{w_dim}) must equal D={d}")
    assert seq == T * H * W
    t_idx = torch.arange(T, device=tokens.device).repeat_interleave(H * W)
    h_idx = torch.arange(H, device=tokens.device).repeat_interleave(W).repeat(T)
    w_idx = torch.arange(W, device=tokens.device).repeat(T * H)
    # Simplified: just scale channels by frequencies. Real RoPE rotates pairs.
    freqs_t = torch.exp(-torch.log(torch.tensor(10000.0)) * torch.arange(t_dim // 2, device=tokens.device) / (t_dim // 2))
    freqs_h = torch.exp(-torch.log(torch.tensor(10000.0)) * torch.arange(h_dim // 2, device=tokens.device) / (h_dim // 2))
    freqs_w = torch.exp(-torch.log(torch.tensor(10000.0)) * torch.arange(w_dim // 2, device=tokens.device) / (w_dim // 2))
    emb_t = torch.cat([torch.sin(t_idx[:, None] * freqs_t), torch.cos(t_idx[:, None] * freqs_t)], dim=-1)
    emb_h = torch.cat([torch.sin(h_idx[:, None] * freqs_h), torch.cos(h_idx[:, None] * freqs_h)], dim=-1)
    emb_w = torch.cat([torch.sin(w_idx[:, None] * freqs_w), torch.cos(w_idx[:, None] * freqs_w)], dim=-1)
    return tokens + torch.cat([emb_t, emb_h, emb_w], dim=-1)

Simplified additive form. Real RoPE rotates paired channels at frequencies; the positional information is the same.

Step 3: Divided attention block

class DividedAttentionBlock(nn.Module):
    def __init__(self, dim=64, heads=2):
        super().__init__()
        self.time_attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.space_attn = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.ln1 = nn.LayerNorm(dim)
        self.ln2 = nn.LayerNorm(dim)
        self.ln3 = nn.LayerNorm(dim)
        self.mlp = nn.Sequential(nn.Linear(dim, 4 * dim), nn.GELU(), nn.Linear(4 * dim, dim))
 
    def forward(self, x, grid):
        T, H, W = grid
        n, seq, d = x.shape
        # time attention: same (h, w), across t
        xt = x.view(n, T, H * W, d).permute(0, 2, 1, 3).reshape(n * H * W, T, d)
        a, _ = self.time_attn(self.ln1(xt), self.ln1(xt), self.ln1(xt), need_weights=False)
        xt = (xt + a).reshape(n, H * W, T, d).permute(0, 2, 1, 3).reshape(n, seq, d)
        # space attention: same t, across (h, w)
        xs = xt.view(n, T, H * W, d).reshape(n * T, H * W, d)
        a, _ = self.space_attn(self.ln2(xs), self.ln2(xs), self.ln2(xs), need_weights=False)
        xs = (xs + a).reshape(n, T, H * W, d).reshape(n, seq, d)
        xs = xs + self.mlp(self.ln3(xs))
        return xs

The time attention attends within each spatial position across time. The space attention attends within each frame across positions. Two passes costing O((HW) * T^2) and O(T * (HW)^2) instead of one O((THW)^2): this is the core of TimeSformer and every modern video DiT.

Step 4: Compose a tiny video DiT

class TinyVideoDiT(nn.Module):
    def __init__(self, in_channels=4, dim=64, depth=2, heads=2):
        super().__init__()
        self.patch = VideoPatch3D(in_channels=in_channels, dim=dim, patch_t=2, patch_h=2, patch_w=2)
        self.blocks = nn.ModuleList([DividedAttentionBlock(dim, heads) for _ in range(depth)])
        self.out = nn.Linear(dim, in_channels * 2 * 2 * 2)
 
    def forward(self, x):
        tokens, grid = self.patch(x)
        for blk in self.blocks:
            tokens = blk(tokens, grid)
        return self.out(tokens), grid

Not a working video generator. Just a structural demo that proves every piece is shaped correctly.

Step 5: Check shapes

vid = torch.randn(1, 4, 8, 16, 16)  # (N, C, T, H, W)
model = TinyVideoDiT()
out, grid = model(vid)
print(f"input  {tuple(vid.shape)}")
print(f"tokens grid {grid}")
print(f"output {tuple(out.shape)}")

Expect grid = (4, 8, 8) and out = (1, 256, 32) after patching. The head then projects each token back into its own spatio-temporal patch, ready to be un-patchified into video.

Production access in 2026 breaks down like this:

• Sora 2 API (OpenAI): text-to-video with synchronized audio, priced at a premium.
• Runway Gen-5 / GWM-1 (Runway): image-to-video and interactive worlds.
• Wan-Video 2.1 / HunyuanVideo: open-source, self-hosted.
• Cosmos / Cosmos-Drive (NVIDIA): driving simulation, open weights.
• Genie 3: research preview, request access.

Building an interactive world-model demo: start with Wan-Video for quality, then layer on a latent-action adapter for interactivity. Building an autonomous-driving simulation: Cosmos-Drive is the 2026 open reference.

For robotics, the stack that's actually running in the wild:

1. Language goal -> VLM (Qwen3-VL) -> high-level plan.
2. Plan -> latent-action video model -> imagined rollout.
3. Rollout -> inverse dynamics model -> low-level actions.
4. Actions executed -> observation fed back into step 1.

This lesson produces:

• outputs/prompt-video-model-picker.md: picks between Sora 2, Runway, Wan, HunyuanVideo, and Cosmos given task, license, and latency.
• outputs/skill-physical-plausibility-checks.md: a skill defining automated checks (object permanence, gravity, continuity) to run on any generated video before shipping.

1. (Easy) Compute the token count for a 5-second 360p video at patch-t=2, patch-h=8, patch-w=8. Reason about memory for attention at this size.
2. (Medium) Swap the divided attention block above for a full joint attention block and measure the shape and parameter count. Explain why divided attention is necessary for real video models.
3. (Hard) Build a minimal latent-action video model: take a dataset of (frame_t, action_t, frame_{t+1}) triples (any simple 2D game), train a tiny video DiT conditioned on action embeddings, and show that different actions produce different next frames.

• Sora technical report (OpenAI)
• Genie: Generative Interactive Environments (Bruce et al., 2024): latent action world models
• TimeSformer (Bertasius et al., 2021): divided attention for video transformers
• DreamerV3 (Hafner et al., 2023): world models for RL
• Cosmos-Drive-Dreams (NVIDIA, 2025): driving world model
• Top 10 Video Generation Models 2026 (DataCamp)
• From Video Generation to World Model: survey repo