Open Models: Architecture Walkthroughs

In Lesson 04 you wrote 350 lines of numpy and had a GPT-2-shaped model. Llama 3 405B has a 200-page technical report, and it's tempting to think the two are different beasts. They aren't. The report describes the same object with five or six well-motivated modifications, plus a thousand implementation details about scaling. The skeleton (embedding, transformer blocks, attention, MLP, norm, head) is unchanged.

This lesson is a diff. For each major open model family I list what changed from GPT-2, why, and what it cost. By the end you can read a fresh model card and translate it back to the GPT-2 baseline in your head.

The payoff comes when Meta releases Llama 5 or DeepSeek releases V4. You won't need a new mental model. You look at the config, see which of the well-known knobs moved, and work out what that means downstream. The 2026 architectures are a finite toolbox, and each new model picks a different subset.

The Invariant Core

The autoregressive open models in this lesson share:

• Token embedding matrix (vocab_size x hidden_dim).
• Stack of N decoder blocks: norm, self-attention, residual, norm, MLP, residual.
• Final norm and linear head projecting to vocab_size (often weight-tied with embeddings).
• Causal mask, next-token cross-entropy loss.

That is the shape. The rest is knobs.

The Six Knobs That Actually Move

Across the 2024-2026 frontier open models, the same six design choices get picked again and again:

1. Normalization. LayerNorm -> RMSNorm.
2. Positional encoding. Learned absolute -> RoPE (plus variants: YaRN, NTK).
3. Activation. GELU -> SwiGLU (or GeGLU).
4. Attention head sharing. MHA -> GQA -> MQA -> MLA.
5. Dense vs sparse MLP. Dense -> Mixture-of-Experts.
6. Pre-norm placement. Pre-norm stays. Post-norm is gone.

Everything else (learning rate schedule, data mix, batch size, context length) lives in the training config, not the architecture. That leaves six knobs.

Knob 1: RMSNorm

LayerNorm subtracts mean, divides by std, scales, and shifts. RMSNorm keeps only the scale:

RMSNorm(x) = x / sqrt(mean(x^2) + eps) * gamma

No mean subtraction. No bias. One matmul fewer per token. Zhang and Sennrich (2019) argued it matched LayerNorm on machine translation while being 10% faster. The open models in the table below run it.

The cost is nothing I know of. You get a small throughput win and simpler code.

Knob 2: RoPE

Learned position embeddings were a 1024-slot lookup table in GPT-2. Context 1025 is off the end of the table, so a model with learned positions has no way to extrapolate past its training length.

Rotary Position Embedding (RoPE, Su et al. 2021) injects position by rotating each Q and K vector in pairs before the attention dot product. The angle of rotation is a deterministic function of position, so there is nothing learned and nothing to run out of. With scaling tricks (NTK-aware interpolation, YaRN), a model trained on 8k context can stretch to 128k at inference with modest accuracy loss.

q_rotated = rotate(q, angle(pos))
k_rotated = rotate(k, angle(pos))
score = q_rotated . k_rotated

The Llama, Mistral, Qwen, DeepSeek, and Gemma models in the table all use RoPE. Gemma 2 uses a hybrid (RoPE on most layers, local sliding-window attention on others).

Knob 3: SwiGLU

GPT-2's MLP is x -> gelu(xW1 + b1) -> (...)W2 + b2. SwiGLU (Shazeer 2020) replaces the activation with a gated product:

SwiGLU(x) = (xW1) * sigmoid(xW1) * xV

Two projections in parallel instead of one, gated by the Swish activation. In practice it gives lower perplexity per parameter. Llama 2 adopted it and the other families followed. The MLP's hidden size is usually set so that total parameter count matches the original dense MLP: if GPT-2 used ff_dim = 4 * hidden, SwiGLU uses ff_dim = (2/3) * 4 * hidden = 8/3 * hidden.

Knob 4: Attention Head Sharing

GPT-2 used Multi-Head Attention (MHA): every head has its own Q, K, V projection.

Multi-Query Attention (MQA, Shazeer 2019) shares one K and one V across all heads. That cuts the KV cache by a factor of num_heads, a 12x to 32x reduction on a typical model, at the price of a slight accuracy drop on hard benchmarks.

Grouped-Query Attention (GQA, Ainslie et al. 2023) is the middle ground: G groups of Q heads share one K and one V. Llama 3 8B uses GQA with 32 Q heads and 8 KV heads (G=8), so the KV cache shrinks 4x versus full MHA.

Multi-Head Latent Attention (MLA, DeepSeek 2024) compresses K and V into a shared low-rank latent, projecting them back up per head. This shrinks the KV cache further while keeping per-head expressiveness. DeepSeek-V2 and V3 rely on it for long-context performance.

Above roughly 13B parameters, the models in the table use GQA or MLA. Full MHA at that scale makes the KV cache a serious problem.

Knob 5: Mixture of Experts

A dense MLP activates all its parameters for every token. An MoE MLP has K experts per block and a router that picks the top-k experts per token (typically top-2). Only those experts' weights see a forward pass for that token.

router_logits = xW_r
indices, weights = top_k(router_logits, k=2)
output = sum_i weights[i] * expert[indices[i]](x)

The appeal is that you can have 64 experts of size 7B each, which makes the total parameter count huge, while running only 2 of them per token, so per-token compute matches a dense 7B model. Mixtral 8x7B has 47B total parameters but activates only 13B per token. DeepSeek-V3 has 671B total parameters but activates only 37B per token.

The upside is more parameters and capacity for the same compute. The downsides: the expert memory still has to live somewhere, so serving needs more VRAM than a dense equivalent. Load-balancing the router is hard. Fine-tuning the router during alignment is its own research area.

Knob 6: Pre-norm stays

The original transformer applied layer norm after each sublayer. The open models in the table put a norm before each sublayer instead (Gemma 2 adds one after as well). Pre-norm is easier to train at depth. Nothing to argue about.

Model-by-Model Diff

This table makes the six knobs concrete.

Scan the columns. Past GPT-2, the rows agree on RMSNorm, on SwiGLU or its GeGLU cousin, and on RoPE. Above 7B the attention column reads GQA unless MLA replaced it. MoE is what separates the top end.

Reading a config.json

Llama 3 8B config:

{
  "hidden_size": 4096,
  "intermediate_size": 14336,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "max_position_embeddings": 131072,
  "rope_theta": 500000.0,
  "rms_norm_eps": 1e-5,
  "vocab_size": 128256
}

Most of these fields map to something you have already implemented.

• hidden_size: embedding dimension.
• intermediate_size: MLP hidden size (3.5x hidden, a bit above the 8/3 rule of thumb).
• num_hidden_layers: stack depth.
• num_attention_heads: Q heads.
• num_key_value_heads: KV heads (GQA).
• max_position_embeddings: training context length.
• rope_theta: RoPE base frequency. Meta scaled it from the default 10k to 500k for long-context extrapolation.
• rms_norm_eps: numerical stability.
• vocab_size: tokens.

From these alone you compute total parameters, KV cache, and peak activation memory. See code/main.py for the exact formulas.

Activation memory budget

Above a few billion parameters, activations dominate training memory. A rule of thumb for pre-training with gradient checkpointing:

activation_mem ~ batch_size * seq_len * hidden_size * num_layers * bytes_per_element

For Llama 3 8B at batch 1, seq 8192, BF16, 32 layers, hidden 4096: the formula gives 8192 * 4096 * 32 * 2 bytes = 2.1 GB with checkpointing, which keeps about one hidden-sized tensor per layer. Without it each layer holds roughly 17 such tensors, about 36 GB. That 17 comes from Korthikanti et al. (2022), who count about 34 bytes per sequence-batch-hidden element per layer for a GPT-style block in 16-bit precision. It leaves out the attention-score term and is only a rough guide for Llama's SwiGLU and GQA layers. This is why flash-attention and ring-attention matter: they rewrite the attention computation so activations fit.

KV Cache budget

For inference at max context:

kv_cache = 2 * num_layers * num_kv_heads * head_dim * max_seq_len * bytes_per_element

Llama 3 8B at 128k context, BF16, head_dim = hidden / num_heads = 128: 2 * 32 * 8 * 128 * 131072 * 2 = 17.2 GB per sequence. That is 16 GiB, which is why main.py prints 16.0GB for this config: it divides by 1024 per step instead of 1000.

The 8B weights are 16 GB in BF16, so the KV cache for a single 128k sequence is larger than the weights. That memory pressure is what drives GQA, MLA, and KV cache quantization research.

When Each Model Wins

• Single 80GB GPU, no MoE: Llama 3 8B, Mistral 7B, Gemma 2 9B. They are easy to serve and have wide tooling.
• Single node (8x80GB), big capacity: Llama 3 70B, Qwen 2.5 72B. The strongest dense open capability in this list.
• Biggest open capability, accept MoE complexity: DeepSeek V3, Mixtral 8x22B. The best capability per active FLOP.
• Long-context needs: Llama 3 (128k with RoPE scaling), DeepSeek (MLA advantage).
• Low-latency serving: Gemma 2 9B (sliding window cuts long-context compute).

main.py is a calculator. Given a config dict in the shape of a HuggingFace config.json, it prints parameter counts by component, KV cache at max context, the MLP ratio, and a short verdict on the architecture (MHA, GQA, MLA, MoE).

config = {
    "hidden_size": 4096, "intermediate_size": 14336,
    "num_hidden_layers": 32, "num_attention_heads": 32,
    "num_key_value_heads": 8, "vocab_size": 128256,
    "max_position_embeddings": 131072,
}

The script walks the config field by field and counts parameters for the embedding, attention (with the GQA reduction), the MLP (with SwiGLU's extra projection), and the norms. It treats the output head as tied to the embedding, so the head is not counted a second time. It then computes the KV cache at the stated context length and prints a summary.

See code/main.py for the implementation.

Run the calculator on the configs bundled in the script: GPT-2 Small, Mistral 7B, Llama 3 8B and 70B, Mixtral 8x7B, Qwen 2.5 72B, and DeepSeek V3. Compare the parameter breakdowns. The MoE models have a total parameter count far above their active count. Mixtral's active share is about a quarter of its total, and DeepSeek V3's is about 5%. DeepSeek V3's KV cache is also smaller than Llama 3 70B's (15.2 GB against 40.0 GB) despite roughly ten times the total parameters, and it is smaller than Llama 3 405B's too, though that config is not bundled. That is MLA at work.

Then plug in a config for any model you have locally, read the summary, and decide whether it fits your GPU.

This lesson produces outputs/skill-open-model-picker.md. Given a deployment target (GPU type, VRAM, context length, latency budget) and a task profile (chat, code, reasoning, long-context), it recommends an open model, a quantization scheme from Lesson 11, and an inference stack from Lesson 12, with explicit reasoning about the six architectural knobs.

1. Read the Qwen 2.5 72B config from HuggingFace. Compute total parameters from scratch. Compare to the HF-reported value and identify where any delta comes from (head dim rounding, KV sharing factor, etc.).

2. DeepSeek V3 uses 256 experts with top-8 routing. Compute the ratio of activated experts to total experts and compare to Mixtral 8x7B's top-2 of 8. What does the shift from 25% of experts active to about 3% imply about capacity per FLOP?

3. Compute the KV cache for Llama 3 405B at 128k context in FP8 and BF16. At FP8 it is half the BF16 number. How many parallel sequences can you serve on a single 8xH100 node (80GB each = 640GB total, minus weight memory)?

4. Gemma 2 alternates full-attention and sliding-window-attention layers. Write the math for the KV cache when half the layers use a 4096-token sliding window instead of full context. How much memory does that save at 8k total context?

5. Find a recent frontier open model that was released after this lesson was written. Identify which of the six knobs it picked and whether it introduced a seventh knob. The curriculum will feel out of date the moment a new architecture ships. The goal is to update your table without rebuilding your mental model.

• Dubey et al. (2024). The Llama 3 Herd of Models: the architectural and training reference for the dense Llama 3 family
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report: MLA plus auxiliary-loss-free load balancing plus 671B MoE
• Jiang et al. (2024). Mixtral of Experts: the canonical MoE open model paper
• Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding: the RoPE paper
• Shazeer (2020). GLU Variants Improve Transformer: SwiGLU, GeGLU, and friends
• Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models: the GQA paper
• Gemma 2 Team (2024). Gemma 2: Improving Open Language Models at a Practical Size: hybrid full+sliding attention, pre+post-norm
• Qwen Team (2024). Qwen 2.5 Technical Report: YaRN context extension and long-context training recipes