Differential Attention (V2)

Softmax attention has a property that is harmless at 4k tokens and a headache at 128k. For a query q, the attention weights are softmax(qK^T / sqrt(d)), and softmax does not produce exact zeros. Each non-matching token gets some positive mass. That residual mass is noise, and it scales with context length. At 128k tokens, if each non-matching token gets only 0.0001% of the probability, the 127,999 of them combined contribute about 13% of the total. The model has to learn to route around a noise floor that grows with context.

In practice this shows up as attention-head interference: hallucinated citations in long-context RAG, lost-in-the-middle failures on 100k-token retrieval tasks, and accuracy that slips on needle-in-haystack benchmarks past 32k. The Differential Transformer paper (arXiv:2410.05258, ICLR 2025) measured the gap. DIFF Transformers reached lower perplexity, higher long-context accuracy, and fewer hallucinations than same-size baselines.

DIFF V1 had three problems that kept it out of frontier pre-training pipelines. Its value cache had to be loaded twice per decode step. It needed custom CUDA kernels, which broke FlashAttention compatibility. And its per-head RMSNorm destabilized long-run training at 70B-plus scale. DIFF V2 (Microsoft unilm blog, January 20, 2026) fixes those three. This lesson covers both versions, builds the difference operator, and measures noise cancellation on a toy query.

The noise floor of softmax

For a query q and keys K = [k_1, ..., k_N], attention weights are:

w_i = exp(q . k_i / sqrt(d)) / sum_j exp(q . k_j / sqrt(d))

No w_i is zero. If k_i is unrelated to q, the score q . k_i is not 0. It fluctuates around zero with variance ||q||^2 / d. After normalization each unrelated token still contributes O(1/N) to the weighted sum, and the N-1 of them together contribute O((N-1)/N) = O(1). That is not a small quantity.

What the model wants is closer to a hard top-k: high weight on matching tokens, near-zero weight everywhere else. Softmax is too smooth to do that directly.

The differential idea

Split each head's Q and K projections into two: Q = (Q_1, Q_2) and K = (K_1, K_2). Compute two attention maps:

A_1 = softmax(Q_1 K_1^T / sqrt(d))
A_2 = softmax(Q_2 K_2^T / sqrt(d))

Output:

DiffAttn = (A_1 - lambda * A_2) V

The subtraction cancels whatever noise distribution the two maps share. If both maps put roughly uniform weight on the 127k unrelated tokens, which is what you get at random initialization, that weight cancels. The signal is a peaked weight on the few relevant tokens, and it only cancels if it shows up in both maps at the same magnitude. Once the model trains, it does not.

lambda is a learnable scalar per head, parameterized as lambda = exp(lambda_q1 dot lambda_k1) - exp(lambda_q2 dot lambda_k2) + lambda_init. It can be negative. lambda_init defaults to a small positive number like 0.8.

Why this works like noise cancellation

Think of two noisy microphones recording the same voice. Both pick up the speaker plus correlated background noise. Subtract one from the other and the shared noise drops out. The voice survives because the two signals differ in phase or amplitude enough to prevent full cancellation. The per-head lambda learns that balance.

V1 vs V2: the diff

V1 kept the parameter count equal to the baseline Transformer. To get two queries per head it halved the head dimension. That cost head expressiveness and halved the value cache per head, which hurt more: decode had to load the value cache twice per step, once per softmax branch. The result was decode slower than baseline despite matching parameter count.

V2 doubles the number of query heads and keeps the KV heads the same, borrowing parameters from the up-projection. The head dimension stays at the baseline value. After the subtraction, the extra dimension is projected back down to match the baseline Transformer's O_W projection. Three things follow:

1. Decode speed matches baseline (the KV cache is loaded once).
2. FlashAttention runs unchanged (no custom kernel).
3. Arithmetic intensity at decode goes up (more compute per byte loaded from HBM).

V2 also drops the per-head RMSNorm that V1 used to stabilize the subtraction. At 70B-class pre-training scales that RMSNorm destabilized late training. V2 replaces it with a simpler initialization scheme that keeps training stable without the extra module.

When to reach for it

The value grows with context length. At 4k tokens the noise floor is small enough that standard attention is fine. At 128k it is hurting you.

How it stacks with other 2026 knobs

code/main.py implements differential attention in pure Python, on a synthetic setup where you know where the signal is, so you can measure noise cancellation directly. One note on scope: main() runs the measurement on logits (a trained branch and an untrained branch over 1024 positions). diff_attention takes already-projected Q1, K1, Q2, K2 and implements the full operation, but main() does not call it.

Step 1: standard softmax attention

Stdlib only: lists of lists, manual matmul, and a softmax that subtracts the max for numerical stability.

def softmax(row):
    m = max(row)
    exps = [math.exp(x - m) for x in row]
    s = sum(exps)
    return [e / s for e in exps]

Step 2: split Q, K into two halves

V1 style halves the head dimension. V2 style keeps the head dimension and doubles the number of heads. diff_attention takes the four split inputs, so the math is the same either way and only the bookkeeping differs.

Step 3: two softmax branches + subtraction

A1 = [softmax([dot(q1, k) / scale for k in K1]) for q1 in Q1]
A2 = [softmax([dot(q2, k) / scale for k in K2]) for q2 in Q2]
diff_weights = [[a1 - lam * a2 for a1, a2 in zip(r1, r2)] for r1, r2 in zip(A1, A2)]
out = [[sum(w * v[j] for w, v in zip(row, V)) for j in range(d_v)] for row in diff_weights]

The output weights can be negative. That is fine, because the weighted sum over V handles signed weights.

Step 4: noise cancellation measurement

main() builds 1024 logits for the first branch (Gaussian noise with std 0.5, plus a signal logit of 4.0 at position 500) and 1024 noise logits for the second. It computes the signal-to-noise ratio, meaning the weight on the signal position divided by the mean absolute weight elsewhere, for the first branch alone and for the difference at lambda values from 0.0 to 1.0. A second sweep varies the noise std and compares standard attention against differential attention at lambda = 0.8.

The target for differential attention is a signal-to-noise gain of roughly 3x-10x, depending on how far the two branches have been trained to differ. The toy trains nothing, so its numbers are smaller. At noise std 0.5, lambda = 0.8 takes the ratio from 47.72 to 92.31, and the sweep shows the gain shrinking as the noise grows.

Step 5: V1 vs V2 parameter accounting

Given a config (hidden=4096, heads=32, d_head=128, and 8 KV heads for V2), the toy prints:

• Baseline Transformer: Q, K, V, and O each of size hidden * hidden, so 4 * hidden * hidden in total.
• DIFF V1: Q and K each of size hidden * hidden in total (two half-width halves), V and O unchanged, head dim halved internally. It adds per-head lambda parameters, 4 * n_heads * (d_head // 2), which comes to 8.2K.
• DIFF V2: Q of size 2 * hidden * hidden, K and V of size hidden * kv_heads * d_head each, and O counted at the doubled width. It adds 4 * n_heads * d_head lambda parameters.

At this config the toy prints a V2 delta of 8.4M over a 67.1M baseline attention block.

DIFF V2 is not yet shipping in every production inference server as of April 2026, but integration is underway in vLLM and SGLang. Meanwhile the pattern shows up in:

• Microsoft internal long-context production models.
• Research replications in several open model training runs targeting 256k-plus context.
• Hybrid architectures that combine DIFF attention with sliding-window attention on alternate layers.

When you would reach for it in 2026:

• Training a new model from scratch with a 64k-plus effective context target. Add differential attention from the start, because retraining later is expensive.
• Fine-tuning a long-context model where lost-in-the-middle failures dominate your eval. A LoRA on the Q projections can approximate the DIFF structure.

When you would not:

• You are serving a pre-trained dense model whose long-context performance is already stable. The retraining cost rarely pays back on existing weights.
• Your context stays under 16k. The noise floor is negligible there.

This lesson produces outputs/skill-diff-attention-integrator.md. Given a model architecture, target context length, hallucination profile, and training budget, it produces an integration plan for adding differential attention to a new pre-training run or a LoRA fine-tune.

1. Run code/main.py. Check that the signal-to-noise ratio for differential attention beats standard softmax attention on the synthetic query at low noise. Extend the noise sweep and find the crossover point where standard attention comes out ahead.

2. Compute the parameter-count delta from baseline to DIFF V1 and from baseline to DIFF V2 for a 7B-class model (hidden=4096, heads=32, d_head=128, 32 layers). Show which components gained parameters and which stayed the same.

3. Read Section 3 of the DIFF V1 paper (arXiv:2410.05258) and Section 2 of the DIFF V2 Hugging Face blog. In two sentences, explain why the V1 per-head RMSNorm was necessary and why V2 could remove it without training divergence.

4. Implement an ablation: compute differential attention with lambda = 0 (pure first softmax) and lambda = 1 (full subtraction). On the synthetic query, measure how signal-to-noise changes across the sweep. Find the lambda that maximizes signal-to-noise.

5. Extend the toy to GQA + DIFF V2. Pick 8 KV heads and 32 Q heads. Show that the KV cache size matches a baseline GQA model with the same (8, 32) configuration.

• Ye et al.: Differential Transformer (arXiv:2410.05258, ICLR 2025): the original paper with noise-cancellation theory and long-context ablations
• Microsoft unilm: Differential Transformer V2 (Hugging Face blog, January 2026): the production-stack rewrite, matching baseline decode, FlashAttention-compatible
• Understanding Differential Transformer Unchains Pretrained Self-Attentions (arXiv:2505.16333): theoretical analysis of why the subtraction recovers pretrained attention structure
• Shared DIFF Transformer (arXiv:2501.17900): parameter-sharing variant
• Vaswani et al.: Attention Is All You Need (arXiv:1706.03762): the baseline Transformer DIFF subtracts from
• Liu et al.: Lost in the Middle (arXiv:2307.03172): the long-context benchmark DIFF attention targets