# Attention: how a transformer decides what to look at

Attention lets every word gather information from the words it needs. Pick a word and see where it looks, compare heads with different jobs, and sharpen or soften the softmax.

<div data-sim="attention"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Each row of the grid is one word asking a question (its query); each column is a word offering an answer (its key). The colour is the attention weight after softmax, so every row adds up to 100%. With the causal mask on, a word can only look at earlier words, as in a language model that writes left to right.

## Things to try

- Select 'it' in head 1: it attends mostly to 'cat', the word it refers to.
- Switch to head 3: each word looks at the one before it, a pattern real models learn early.
- Lower the temperature: attention becomes nearly one-hot. Raise it: it spreads evenly.
- Turn off the mask: words can now look ahead, as in an encoder such as BERT.

## How it works in real systems

Real transformers learn their query, key and value projections from data, use dozens of heads in parallel in every layer, and stack dozens of layers. The toy vectors here are hand-made so that the heads have readable jobs; the arithmetic is the same.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Scaled dot-product attention in ten lines

```python
import torch, math
def attention(Q, K, V, causal=True):
    d = Q.shape[-1]
    scores = Q @ K.transpose(-2, -1) / math.sqrt(d)            # (T, T): how well each query matches each key
    if causal:
        T = scores.shape[-1]
        scores = scores.masked_fill(torch.triu(torch.ones(T, T, dtype=torch.bool), 1), float("-inf"))
    weights = scores.softmax(-1)                                # every row sums to 1
    return weights @ V, weights
torch.manual_seed(0)
T, d = 6, 8
x = torch.randn(T, d)
Wq, Wk, Wv = (torch.randn(d, d) / d ** 0.5 for _ in range(3))
out, w = attention(x @ Wq, x @ Wk, x @ Wv)
print(w.round(decimals=2))
```

### Step 2 · Multi-head attention

Split the model dimension into heads, attend in each head separately, concatenate, project.

```python
import torch.nn as nn
class MHA(nn.Module):
    def __init__(self, d, h):
        super().__init__(); self.h = h; self.qkv = nn.Linear(d, 3 * d); self.o = nn.Linear(d, d)
    def forward(self, x):                                       # x: (B, T, d)
        B, T, d = x.shape
        q, k, v = self.qkv(x).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        y = torch.nn.functional.scaled_dot_product_attention(q, k, v, is_causal=True)   # fused, memory-efficient
        return self.o(y.transpose(1, 2).reshape(B, T, d))
print(MHA(64, 8)(torch.randn(2, 10, 64)).shape)
```

### Step 3 · Look at a real model's attention

```python
from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
m = AutoModel.from_pretrained("bert-base-uncased", output_attentions=True)
enc = tok("The cat sat on the mat because it was tired", return_tensors="pt")
att = m(**enc).attentions                                   # one tensor per layer: (batch, heads, T, T)
tokens = tok.convert_ids_to_tokens(enc["input_ids"][0])
i = tokens.index("it")
layer, head = 5, 0
print(sorted(zip(att[layer][0, head, i].tolist(), tokens), reverse=True)[:4])
```

Heads differ a lot: some track the previous token, some link pronouns to nouns, some attend to punctuation.

<div class="cl-end"></div>

## More simulations

[← Tokenisation](/ai-simulations/tokenisation-bpe) · [All AI Simulations](/ai-simulations) · [Sampling →](/ai-simulations/sampling-temperature-top-p)
