← Back to curriculum

Phase 10 · 24 lessons

LLMs from Scratch

Build, train, and understand large language models.

01Tokenizers: BPE, WordPiece, SentencePieceBuild 02Building a Tokenizer from ScratchBuild 03Data Pipelines for Pre-TrainingBuild 04Pre-Training a Mini GPT (124M Parameters)Build 05Scaling: Distributed Training, FSDP, DeepSpeedBuild 06Instruction Tuning (SFT)Build 07RLHF: Reward Model + PPOBuild 08DPO: Direct Preference OptimizationBuild 09Constitutional AI and Self-ImprovementBuild 10Evaluation: Benchmarks, Evals, LM HarnessBuild 11Quantization: Making Models FitBuild 12Inference OptimizationBuild 13Building a Complete LLM PipelineBuild 14Open Models: Architecture WalkthroughsLearn 15Speculative Decoding and EAGLE-3Build 16Differential Attention (V2)Build 17Native Sparse Attention (DeepSeek NSA)Build 18Multi-Token Prediction (MTP)Build 19DualPipe ParallelismLearn 20DeepSeek-V3 Architecture WalkthroughLearn 21Jamba — Hybrid SSM-TransformerLearn 22Async and Hogwild! InferenceBuild 23Speculative Decoding and EAGLEBuild 24Gradient Checkpointing and Activation RecomputationBuild