# Positional encoding: sinusoids and rotary embeddings (RoPE)

Attention by itself ignores order: shuffle the words and the scores stay the same. Positional encodings fix that. Explore the original sinusoids and the rotary embeddings used by most modern language models.

<div data-sim="posenc"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The heatmap is PE(pos, i) for 128 positions; the line below is the similarity of one position's encoding with all others. On the right, one pair of dimensions of a query and a key is rotated by its position, and the score is shown for (m, n) and for both shifted by 20.

## Things to try

- Move position p: the similarity peaks at p and falls off on both sides.
- Change m and n but keep m − n fixed: the RoPE score does not change.
- Switch the base to 500 000 and watch the slow dimensions stretch.

## How it works in real systems

Llama, Qwen, Mistral and most open models use RoPE; long-context versions raise its base or rescale the frequencies (NTK-aware scaling, YaRN). BERT and GPT-2 learned absolute position embeddings instead.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Sinusoidal positional encoding

```python
import torch, math
def sinusoidal(T, d, base=10000.0):
    pos = torch.arange(T)[:, None]
    i = torch.arange(0, d, 2)[None]
    ang = pos / base ** (i / d)
    pe = torch.zeros(T, d); pe[:, 0::2] = torch.sin(ang); pe[:, 1::2] = torch.cos(ang)
    return pe                                            # added to the token embeddings
pe = sinusoidal(128, 64)
sim = pe @ pe[40] / 32
print(sim[35:46].round(decimals=2))                      # peaks at 40, depends on the distance
```

### Step 2 · Rotary position embedding (RoPE)

RoPE rotates each pair of query/key dimensions by an angle proportional to the position. It is applied to q and k inside every attention layer, not added to the embeddings.

```python
def rope(x, pos, base=10000.0):                          # x: (..., T, d), pos: (T,)
    d = x.shape[-1]
    inv = 1.0 / base ** (torch.arange(0, d, 2) / d)
    ang = pos[:, None] * inv[None]                       # (T, d/2)
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]
    out = torch.empty_like(x)
    out[..., 0::2] = x1 * cos - x2 * sin
    out[..., 1::2] = x1 * sin + x2 * cos
    return out
```

### Step 3 · Check that RoPE scores depend only on distance

```python
torch.manual_seed(0)
q, k = torch.randn(64), torch.randn(64)
def score(m, n): return (rope(q[None], torch.tensor([float(m)]))[0] @ rope(k[None], torch.tensor([float(n)]))[0]).item()
print(score(12, 5), score(32, 25), score(112, 105))      # equal: only m − n = 7 matters
```

Long-context models raise the base (for example 500 000) or rescale the angles (NTK scaling, YaRN) so that far positions stay distinguishable.

<div class="cl-end"></div>

## More simulations

[← LR schedules](/ai-simulations/learning-rate-schedules) · [All AI Simulations](/ai-simulations) · [KV cache →](/ai-simulations/kv-cache)
