Sampling: temperature, top-k and top-p
A language model does not output one answer; it outputs a probability for every possible next token. How you sample from that distribution decides whether the text is safe and repetitive or creative and risky.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Grey bars are the model's raw probabilities for the next word; blue bars are what remains after temperature, top-k and top-p, renormalised. Sample once to fill in the sentence, or 200 times to see the frequencies as orange lines.
Things to try
- Set temperature to 0.05: the top word wins every time (greedy decoding).
- Set temperature to 2.5: 'bananas' and 'purple' start to appear.
- Keep temperature high but set top-p to 0.75: the nonsense words are cut off.
- Compare top-k 3 with top-p 0.75: one keeps a fixed number of words, the other adapts to the shape of the distribution.
How it works in real systems
Chat products typically use a temperature between 0.2 and 1 with top-p around 0.9; code and extraction tasks use temperatures near 0. The model is the same; only the sampling changes.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Temperature, top-k and top-p on a logit vector
import torch
def sample(logits, temperature=1.0, top_k=0, top_p=1.0, generator=None):
logits = logits / max(temperature, 1e-6)
if top_k:
kth = torch.topk(logits, top_k).values[-1]
logits = logits.masked_fill(logits < kth, float("-inf"))
probs = logits.softmax(-1)
if top_p < 1.0:
sp, idx = probs.sort(descending=True)
keep = sp.cumsum(-1) - sp < top_p # keep the smallest set whose mass reaches top_p
probs = torch.zeros_like(probs).scatter(0, idx[keep], sp[keep])
probs /= probs.sum()
return torch.multinomial(probs, 1, generator=generator).item(), probs
logits = torch.tensor([3.1, 2.7, 2.4, 2.0, 1.7, 1.5, 1.0, -0.3, -1.6, -2.2])
print(sample(logits, temperature=0.7, top_p=0.9)[1].round(decimals=3))
Step 2 · With Hugging Face generate()
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name); model = AutoModelForCausalLM.from_pretrained(name)
inputs = tok("The model was trained on tables and", return_tensors="pt")
for t in (0.2, 0.8, 1.5):
out = model.generate(**inputs, do_sample=True, temperature=t, top_p=0.9, top_k=50, max_new_tokens=15)
print(t, tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Step 3 · Measure diversity
Sample the same prompt many times and count distinct continuations; low temperature gives repetition, high temperature gives noise.
outs = model.generate(**inputs, do_sample=True, temperature=0.8, top_p=0.9, max_new_tokens=5, num_return_sequences=20)
texts = [tok.decode(o[inputs["input_ids"].shape[1]:]) for o in outs]
print(len(set(texts)), "distinct of", len(texts))