Positional Encoding — Sinusoidal, RoPE, ALiBi

Positional encoding is any signal added to the embeddings or to the attention scores that tells the model where each token sits in the sequence. Attention needs one because it has no sense of order on its own.

The attention matrix softmax(Q K^T / √d) V is built from pairwise similarities. Shuffle the rows of X and the rows of the output get shuffled the same way. Nothing inside attention looks at position.

For a bag-of-words model that's fine. For language, code, audio, and video, where order carries meaning, it breaks the model.

The fix is to inject position into the embeddings somehow. Three eras of answers:

1. Absolute sinusoidal (Vaswani 2017). Add sin/cos of position to the embedding. Simple, no learned parameters, extrapolates poorly beyond trained lengths.
2. RoPE (Rotary Position Embeddings) (Su 2021). Rotate Q and K vectors by an angle proportional to position. Encodes relative position directly in the dot product. Dominant in 2026.
3. ALiBi (Attention with Linear Biases) (Press 2022). Skip embeddings and add a per-head linear penalty to attention scores based on distance. Excellent length extrapolation.

As of 2026, most frontier open models use RoPE: Llama 2/3/4, Qwen 2/3, Mistral, Mixtral, DeepSeek-V3, Kimi. A handful of long-context models use ALiBi or its modern variants. Absolute sinusoidal is mostly historical.

Absolute sinusoidal

Absolute sinusoidal encoding is a fixed matrix PE of shape (max_len, d_model), computed once, whose row pos gets added to the embedding at that position:

PE[pos, 2i]   = sin(pos / 10000^(2i / d_model))
PE[pos, 2i+1] = cos(pos / 10000^(2i / d_model))

You add it as X' = X + PE[:N] before attention. Each dimension is a sinusoid at a different frequency, and the model learns to read position from the phase pattern. It breaks past max_len: nothing told the model what happens at position 2048 when it only saw positions 0–2047.

RoPE

RoPE rotates the Q and K vectors instead of touching the embeddings. For each pair of dimensions (2i, 2i+1), the rotation angle is the position times a per-pair frequency:

[q'_2i    ]   [ cos(pos·θ_i)  -sin(pos·θ_i) ] [q_2i   ]
[q'_2i+1  ] = [ sin(pos·θ_i)   cos(pos·θ_i) ] [q_2i+1 ]

θ_i = base^(-2i / d_head),  base = 10000 by default

Apply the same rotation to keys at position pos_k. The dot product q'_m · k'_n then depends on (m - n) alone. The attention score reflects relative distance even though the rotation was keyed off absolute positions. Beautiful trick.

You can rescale base (NTK-aware, YaRN, LongRoPE) to stretch RoPE to longer contexts. Plain base rescaling often works with little or no fine-tuning, while YaRN and LongRoPE usually add a short fine-tuning stage for the best quality. Llama 3.1 went from 8K to 128K context with a larger base and continued long-context training. These scaling schemes are described in prose only: main.py has no NTK-aware, YaRN, or LongRoPE code.

ALiBi

ALiBi skips the embedding trick and biases the attention scores directly:

attn_score[i, j] = (q_i · k_j) / √d  -  m_h · |i - j|

m_h is a head-specific slope (e.g. 1 / 2^(8·h/H)). Closer tokens get boosted and far tokens get penalized. It adds no training-time cost. The paper reports that length extrapolation beats sinusoidal and matches RoPE at its original trained length.

What to pick in 2026

RoPE won because it slots into attention without changing the architecture, encodes relative position, and its base hyperparameter gives a clean knob for long-context fine-tuning.

Step 1: sinusoidal encoding

The full version is in code/main.py, where it is called sinusoidal_pe. The core is a short double loop over positions and dimension pairs:

def sinusoidal_pe(n, d, base=10000.0):
    pe = [[0.0] * d for _ in range(n)]
    for pos in range(n):
        for i in range(d // 2):
            theta = pos / (base ** (2 * i / d))
            pe[pos][2 * i]     = math.sin(theta)
            pe[pos][2 * i + 1] = math.cos(theta)
    return pe

Add the result to the embedding matrix before the first attention layer.

Step 2: RoPE applied to Q, K

RoPE rotates Q and K. apply_rope copies its input and returns a new rotated list, leaving the original vector untouched. For each pair of dims:

def apply_rope(x, pos, base=10000.0):
    d = len(x)
    out = list(x)
    for i in range(d // 2):
        theta = pos / (base ** (2 * i / d))
        c, s = math.cos(theta), math.sin(theta)
        a, b = x[2 * i], x[2 * i + 1]
        out[2 * i]     = a * c - b * s
        out[2 * i + 1] = a * s + b * c
    return out

Apply the same function to Q at position m and K at position n. Each coordinate pair of Q is rotated by m·θ_i and of K by n·θ_i, so their dot product depends only on the gap m - n, and attention gets relative position for free.

Step 3: ALiBi slopes and bias

main.py splits this into two functions: one for the per-head slopes, one for the bias matrices.

def alibi_slopes(n_heads):
    # slope_h = 2 ** (-8 * h / n_heads) for h = 1..n_heads
    return [2 ** (-8 * (h + 1) / n_heads) for h in range(n_heads)]
 
def alibi_bias(n_heads, seq_len, causal=True):
    out = []
    for m in alibi_slopes(n_heads):
        head_bias = []
        for i in range(seq_len):
            row = []
            for j in range(seq_len):
                if causal and j > i:
                    row.append(float("-inf"))
                else:
                    row.append(-m * abs(i - j))
            head_bias.append(row)
        out.append(head_bias)
    return out  # add to attention scores before softmax

Add out[h] to the (seq_len, seq_len) attention score matrix of head h, then softmax. With causal=True (the default), future positions j > i get -inf so softmax gives them zero weight. The demo in main.py passes causal=False so you can see the full symmetric penalty matrix.

Step 4: verify the relative-distance property of RoPE

Pick two random vectors a, b. Rotate them by (pos_a, pos_b), then by (pos_a + k, pos_b + k). The two dot products must match within floating-point error. That property is what RoPE is built around: the score ignores the absolute offset, and only the relative gap matters.

Core PyTorch has no RoPE function. Libraries such as torchtune and Hugging Face transformers ship their own implementations, and flash_attn includes a fused rotary kernel, so production code rarely hand-rolls the rotation. The scaling schemes below are described in prose only; main.py implements none of them.

from transformers import AutoModel
model = AutoModel.from_pretrained("meta-llama/Llama-3.2-3B")
# model.config.rope_scaling → {"rope_type": "llama3", "factor": 32.0, "low_freq_factor": 1.0, "high_freq_factor": 4.0, "original_max_position_embeddings": 8192}

Llama 3.2 uses Meta's own llama3 scaling variant, not YaRN, but rope_scaling is where you check which scheme any checkpoint uses.

Long-context tricks in 2026:

• NTK-aware interpolation. Rescale base to base * (scale_factor)^(d/(d-2)) when extending from 4K to 16K+.
• YaRN. A smarter interpolation that preserves attention entropy on long contexts. Qwen2's long-context configs use it.
• LongRoPE. Microsoft's 2024 method, which uses evolutionary search to pick per-dimension scale factors. Phi-3-Long uses it.
• Position interpolation + fine-tuning. Shrink positions by the extension factor and fine-tune for 1–5B tokens. Surprisingly effective.

The skill in outputs/skill-positional-encoding-picker.md picks an encoding strategy for a new model given target context length, extrapolation needs, and training budget.

1. Easy. Plot the sinusoidal PE matrix as a heatmap for max_len=512, d=128. Confirm the "stripes get wider as dimension index grows" pattern.
2. Medium. Implement NTK-aware RoPE scaling. Train a tiny LM on sequences of length 256, then test on length 1024 with and without scaling. Measure perplexity.
3. Hard. Implement ALiBi and RoPE in the same attention module. Train a 4-layer transformer on a copy task with sequences of length 512, extrapolate to 2048 at test time, and compare degradation.

• Vaswani et al. (2017). Attention Is All You Need §3.5: original sinusoidal.
• Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding: the RoPE paper.
• Press, Smith, Lewis (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation: ALiBi.
• Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models: state of the art RoPE scaling.
• Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation: Meta's Llama 2 long-context paper.
• Ding et al. (2024). LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens: the Microsoft method used by Phi-3-Long and cited in the Use It section.
• HuggingFace Transformers: modeling_rope_utils.py: production-grade implementations of the RoPE scaling schemes (default, linear, dynamic, YaRN, LongRoPE, Llama-3).