Özer Cem KELAHMETKELAHMET
AI Simulations · Language models

Positional encoding: sinusoids and rotary embeddings (RoPE)

Attention by itself ignores order: shuffle the words and the scores stay the same. Positional encodings fix that. Explore the original sinusoids and the rotary embeddings used by most modern language models.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The heatmap is PE(pos, i) for 128 positions; the line below is the similarity of one position's encoding with all others. On the right, one pair of dimensions of a query and a key is rotated by its position, and the score is shown for (m, n) and for both shifted by 20.

Things to try

  • Move position p: the similarity peaks at p and falls off on both sides.
  • Change m and n but keep m − n fixed: the RoPE score does not change.
  • Switch the base to 500 000 and watch the slow dimensions stretch.

How it works in real systems

Llama, Qwen, Mistral and most open models use RoPE; long-context versions raise its base or rescale the frequencies (NTK-aware scaling, YaRN). BERT and GPT-2 learned absolute position embeddings instead.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Sinusoidal positional encoding

import torch, math
def sinusoidal(T, d, base=10000.0):
    pos = torch.arange(T)[:, None]
    i = torch.arange(0, d, 2)[None]
    ang = pos / base ** (i / d)
    pe = torch.zeros(T, d); pe[:, 0::2] = torch.sin(ang); pe[:, 1::2] = torch.cos(ang)
    return pe                                            # added to the token embeddings
pe = sinusoidal(128, 64)
sim = pe @ pe[40] / 32
print(sim[35:46].round(decimals=2))                      # peaks at 40, depends on the distance

Step 2 · Rotary position embedding (RoPE)

RoPE rotates each pair of query/key dimensions by an angle proportional to the position. It is applied to q and k inside every attention layer, not added to the embeddings.

def rope(x, pos, base=10000.0):                          # x: (..., T, d), pos: (T,)
    d = x.shape[-1]
    inv = 1.0 / base ** (torch.arange(0, d, 2) / d)
    ang = pos[:, None] * inv[None]                       # (T, d/2)
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]
    out = torch.empty_like(x)
    out[..., 0::2] = x1 * cos - x2 * sin
    out[..., 1::2] = x1 * sin + x2 * cos
    return out

Step 3 · Check that RoPE scores depend only on distance

torch.manual_seed(0)
q, k = torch.randn(64), torch.randn(64)
def score(m, n): return (rope(q[None], torch.tensor([float(m)]))[0] @ rope(k[None], torch.tensor([float(n)]))[0]).item()
print(score(12, 5), score(32, 25), score(112, 105))      # equal: only m − n = 7 matters

Long-context models raise the base (for example 500 000) or rescale the angles (NTK scaling, YaRN) so that far positions stay distinguishable.

More simulations

← LR schedules · All AI Simulations · KV cache →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open