DeepSeek-V3 Architecture Walkthrough

DeepSeek-V3 is the first open model at the frontier whose architecture looks meaningfully different from the Llama family. Llama 3 405B is "GPT-2 with six knobs turned." DeepSeek-V3 turns the same six and adds four more. Reading the Llama 3 config is a good warmup, but the shape of the attention block, the routing logic, and the training-time objective differ enough that this model needs its own walkthrough.

The open-weights release shifted what "frontier capability" means for open models, and many 2026 training runs are copying the blueprint. If your work touches frontier LLM training or inference, you need to be able to read this config cold.

The invariant core, again

DeepSeek-V3 is still autoregressive and still a stack of decoder blocks. Each block has attention, an MLP, and two RMSNorms. The MLP is SwiGLU, positions use RoPE, norms are pre-norm, and embeddings are weight-tied. That is the same baseline as the Llama and Mistral families.

The twist: MLA instead of GQA

From Phase 10 · 14 you know GQA shrinks the KV cache by sharing K and V across groups of Q heads. Multi-Head Latent Attention (MLA) goes further. K and V are compressed into one shared low-rank latent (the kv_lora_rank) and decompressed per head on the fly, so the cache stores only the latent: typically 512 floats per token per layer, against 8 x 128 = 1024 floats for the GQA shape below.

At 128k context, DeepSeek-V3 with MLA stores one shared latent c^{KV} per token per layer. K and V are both derived from it through up-projections that can be absorbed into the subsequent matmul.

kv_cache = num_layers * kv_lora_rank * max_seq_len * bytes_per_element
         = 61 * 512 * 131072 * 2
         = 7.6 GB

A hypothetical GQA baseline (Llama 3 70B shape, 8 KV heads, head dim 128) would pay:

kv_cache = 2 * 61 * 8 * 128 * 131072 * 2
         = 30.5 GB

MLA comes out 4x smaller than a Llama-3-70B-style GQA cache at 128k context.

The cost is a decompression step per head in each attention computation. That extra compute is small next to the bandwidth saved, so MLA is a net win for long-context inference.

The routing: auxiliary-loss-free load balancing

MoE routers pick the top-k experts for each token. A naive router piles work onto a few experts and leaves the rest idle. The standard fix is an auxiliary loss term that penalizes load imbalance. It works, but it slightly degrades main-task performance.

DeepSeek-V3 drops the auxiliary loss. Each expert gets a bias term added to its router logits, and training adjusts it with a simple rule: if expert e is overloaded, decrease bias_e; if underloaded, increase it. There is no extra loss term, so the objective stays clean and expert load stays balanced. The effect on the main loss is not measurable, and the MoE has one less hyperparameter to tune.

The MTP: denser training and a free draft

From Phase 10 · 18 you know DeepSeek-V3 adds a D=1 MTP module that predicts the token two positions ahead. In training, each hidden state is supervised on D+1 = 2 targets, which is a denser signal. At inference the trained module doubles as a speculative-decoding draft with 80%+ acceptance.

It costs 14B parameters on top of the 671B main model, a 2.1% overhead.

The training: DualPipe

From Phase 10 · 19 you know DualPipe is a bidirectional pipeline that overlaps forward and backward chunks with cross-node all-to-all communication. At DeepSeek-V3's 2,048-H800 scale it recovers roughly 245k GPU-hours that 1F1B would have lost to pipeline bubbles.

The config, field by field

Here is the DeepSeek-V3 config (simplified):

hidden_size: 7168
intermediate_size: 18432   (dense MLP hidden size, used on first few layers)
moe_intermediate_size: 2048 (expert MLP hidden size)
num_hidden_layers: 61
first_k_dense_layers: 3    (first 3 layers use dense MLP)
num_attention_heads: 128
num_key_value_heads: 128   (formally equal to num_heads under MLA, but
                           the real compression is in kv_lora_rank)
kv_lora_rank: 512          (MLA latent dimension)
num_experts: 256            (MoE expert count per block)
num_experts_per_tok: 8      (top-8 routing)
shared_experts: 1           (always-on shared expert per block)
max_position_embeddings: 163840
rope_theta: 10000.0
vocab_size: 129280
mtp_module: 1               (1 MTP module at depth 1)

Parse it:

• hidden_size=7168 is the embedding dimension.
• num_hidden_layers=61 is the total block depth.
• first_k_dense_layers=3: the first 3 blocks use a dense MLP of size 18432, and the remaining 58 use MoE.
• num_attention_heads=128 gives 128 query heads.
• kv_lora_rank=512: K and V are compressed to this latent dimension and decompressed per head.
• num_experts=256, num_experts_per_tok=8: each MoE block holds 256 routed experts and sends each token to the top 8.
• shared_experts=1: on top of the 256 routed experts, one always-on expert sees every token. Think of it as a "dense floor" that gives each token something reliable.
• moe_intermediate_size=2048 is each expert's MLP hidden size. It is smaller than the dense MLP because there are 256 of them.

Parameter accounting

The full calculation lives in code/main.py. The headline:

• Embedding: vocab * hidden = 129280 * 7168 = ~0.93B.
• First 3 dense blocks: attention with MLA (~84M per block) + dense MLP (~396M per block) + norms. About 481M per block, 1.4B total.
• 58 MoE blocks: attention with MLA (~84M) + 256 experts (44M apiece) + 1 shared expert (44M) + router (1.8M) + norms. That is ~11.4B per block including all experts, and ~661B for the 58 MoE blocks.
• MTP module: the calculator's dense approximation comes to 0.7B, against the 14B in the published model.

The grand total is 664.5B. Without the MTP approximation the core architecture is 663.8B, about 1% under the published 671B. The calculator uses the simple head dimension hidden / heads = 56 for MLA and leaves out bias tensors and other fine-grained pieces that the appendix of Section 2 of DeepSeek's report itemizes, so a small gap is expected.

Active parameters per forward:

• Attention: 84M per layer * 61 = 5.1B (every layer fires).
• MLP active: the first 3 layers are dense (3 * 396M = 1.2B), and each of the 58 MoE layers runs 8 routed experts, 1 shared expert, and the router, about 398M per layer (the MLP part only: the "moe layer, active" line the script prints is 482.6M because it adds the 84M of attention). Total: ~24.3B.
• Embedding + norms: 0.93B.
• Total active: ~30.4B, which is 4.6% of the calculator's total. The published figure is 37B, and the MTP module is trained but not always run at inference.

The 671B / 37B ratio

That is an 18x sparsity ratio: active params are 5.5% of total. DeepSeek-V3 is one of the sparsest frontier MoE models to ship open weights. Mixtral 8x7B at 13/47 (28%) is much denser, and Llama 4 Maverick at 17B/400B (4.25%) is in the same range. The DeepSeek bet is that at frontier scale, more experts with a lower activation ratio give better quality per active FLOP.

Where DeepSeek-V3 sits

The follow-on: R1, V4

DeepSeek-R1 (2025) is a reasoning-training run on the V3 backbone with the same architecture. What changed is the post-training recipe (large-scale RL on verifiable tasks), not the pretraining architecture.

DeepSeek-V4 (if it ships) is expected to keep MLA + MoE + MTP and add DSA (DeepSeek Sparse Attention), the successor to NSA from Phase 10 · 17. The lineage is stable: architecture-level innovations accumulate, and each version turns more knobs.

code/main.py is the parameter calculator specialized to DeepSeek-V3's shape. Run it, compare its output to the paper's numbers, then point it at variants: 256 experts vs 512, top-8 vs top-16, MLA rank 512 vs 1024.

What to look at:

• Total parameter count vs the published 671B.
• Active parameter count vs the published 37B.
• KV cache at 128k context, MLA vs GQA.
• The per-layer breakdown, to see where the parameter budget goes.

This lesson produces outputs/skill-deepseek-v3-reader.md. Give it a DeepSeek-family model (V3, R1, or a future variant) and it returns a component-by-component architecture reading. It names each field of the config, derives parameter counts by component, and says which of the four DeepSeek-specific innovations the model uses.

1. Run code/main.py. Compare the calculator's total-parameter estimate to the published 671B and identify where the delta comes from. The paper's Section 2 has the full itemization.

2. Modify the config to use MLA rank 256 instead of 512. Compute the resulting KV cache size at 128k context. What percentage reduction does it buy, and at what cost to the per-head expressiveness?

3. Compare DeepSeek-V3's (256 experts, top-8) routing to a hypothetical (512 experts, top-8) variant. Total parameters grow; active parameters stay the same. What does the extra expert capacity buy in theory, and what does it cost at inference?

4. Read Section 2.1 of the DeepSeek-V3 technical report (arXiv:2412.19437) on MLA. Explain in three sentences why the K and V decompression matrices can be "absorbed" into the subsequent matmul for inference-time efficiency.

5. DeepSeek-V3 uses FP8 training for most operations. Compute the memory savings of FP8 vs BF16 for storing the 671B weights. How does this intersect with the 14.8T-token training budget?

• DeepSeek-AI: DeepSeek-V3 Technical Report (arXiv:2412.19437): the full architecture, training, and results document
• DeepSeek-V3 model card on Hugging Face: config files and deployment notes
• DeepSeek-V2 paper (arXiv:2405.04434): the predecessor that introduced MLA
• DeepSeek-R1 paper (arXiv:2501.12948): the reasoning-training successor on V3's architecture
• Native Sparse Attention (arXiv:2502.11089): the future direction for DeepSeek-family attention
• DualPipe repository: the training-schedule reference