Positional encoding: sinusoids and rotary embeddings (RoPE)
Attention by itself ignores order: shuffle the words and the scores stay the same. Positional encodings fix that. Explore the original sinusoids and the rotary embeddings used by most modern language models.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The heatmap is PE(pos, i) for 128 positions; the line below is the similarity of one position's encoding with all others. On the right, one pair of dimensions of a query and a key is rotated by its position, and the score is shown for (m, n) and for both shifted by 20.
Things to try
- Move position p: the similarity peaks at p and falls off on both sides.
- Change m and n but keep m − n fixed: the RoPE score does not change.
- Switch the base to 500 000 and watch the slow dimensions stretch.
How it works in real systems
Llama, Qwen, Mistral and most open models use RoPE; long-context versions raise its base or rescale the frequencies (NTK-aware scaling, YaRN). BERT and GPT-2 learned absolute position embeddings instead.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Sinusoidal positional encoding
import torch, math
def sinusoidal(T, d, base=10000.0):
pos = torch.arange(T)[:, None]
i = torch.arange(0, d, 2)[None]
ang = pos / base ** (i / d)
pe = torch.zeros(T, d); pe[:, 0::2] = torch.sin(ang); pe[:, 1::2] = torch.cos(ang)
return pe # added to the token embeddings
pe = sinusoidal(128, 64)
sim = pe @ pe[40] / 32
print(sim[35:46].round(decimals=2)) # peaks at 40, depends on the distance
Step 2 · Rotary position embedding (RoPE)
RoPE rotates each pair of query/key dimensions by an angle proportional to the position. It is applied to q and k inside every attention layer, not added to the embeddings.
def rope(x, pos, base=10000.0): # x: (..., T, d), pos: (T,)
d = x.shape[-1]
inv = 1.0 / base ** (torch.arange(0, d, 2) / d)
ang = pos[:, None] * inv[None] # (T, d/2)
cos, sin = ang.cos(), ang.sin()
x1, x2 = x[..., 0::2], x[..., 1::2]
out = torch.empty_like(x)
out[..., 0::2] = x1 * cos - x2 * sin
out[..., 1::2] = x1 * sin + x2 * cos
return out
Step 3 · Check that RoPE scores depend only on distance
torch.manual_seed(0)
q, k = torch.randn(64), torch.randn(64)
def score(m, n): return (rope(q[None], torch.tensor([float(m)]))[0] @ rope(k[None], torch.tensor([float(n)]))[0]).item()
print(score(12, 5), score(32, 25), score(112, 105)) # equal: only m − n = 7 matters
Long-context models raise the base (for example 500 000) or rescale the angles (NTK scaling, YaRN) so that far positions stay distinguishable.