← Back to curriculum

Phase 12 · 25 lessons

Multimodal AI

See, hear, read, and reason across modalities — from ViT patches to computer-use agents.

01Vision Transformers and the Patch-Token PrimitiveLearn 02CLIP and Contrastive Vision-Language PretrainingBuild 03From CLIP to BLIP-2 — Q-Former as Modality BridgeBuild 04Flamingo and Gated Cross-Attention for Few-Shot VLMsLearn 05LLaVA and Visual Instruction TuningBuild 06Any-Resolution Vision: Patch-n'-Pack and NaFlexBuild 07Open-Weight VLM Recipes: What Actually MattersLearn 08LLaVA-OneVision: Single-Image, Multi-Image, Video in One ModelBuild 09Qwen-VL Family and Dynamic-FPS VideoLearn 10InternVL3: Native Multimodal PretrainingLearn 11Chameleon and Early-Fusion Token-Only Multimodal ModelsBuild 12Emu3: Next-Token Prediction for Image and Video GenerationLearn 13Transfusion: Autoregressive Text + Diffusion Image in One TransformerBuild 14Show-o and Discrete-Diffusion Unified ModelsLearn 15Janus-Pro: Decoupled Encoders for Unified Multimodal ModelsBuild 16MIO and Any-to-Any Streaming Multimodal ModelsLearn 17Video-Language Models: Temporal Tokens and GroundingBuild 18Long-Video Understanding at Million-Token ContextBuild 19Audio-Language Models: the Whisper to Audio Flamingo 3 ArcBuild 20Omni Models: Qwen2.5-Omni and the Thinker-Talker SplitBuild 21Embodied VLAs: RT-2, OpenVLA, π0, GR00TLearn 22Document and Diagram UnderstandingBuild 23ColPali and Vision-Native Document RAGBuild 24Multimodal RAG and Cross-Modal RetrievalBuild 25Multimodal Agents and Computer-Use (Capstone)Build