Inside a large language model: tokens, attention, sampling and the KV cache
A language model is a stack of transformer layers that predicts the next token. This page follows one request through the model, from tokens to the generated answer.
Overview
When you send a prompt, the text is cut into tokens, each token becomes a vector, dozens of transformer layers mix information between the vectors, and the last layer produces a probability for every token in the vocabulary. One token is chosen, appended, and the process repeats until the answer is complete.
One request, step by step
| Step | What happens | Read more | Try it |
|---|---|---|---|
| Tokenisation | Text is split into sub-word tokens | BPE, SentencePiece | Simulation |
| Embeddings | Each token becomes a vector | Embeddings | Simulation |
| Position | The model is told the order of tokens | Transformers | Simulation |
| Attention | Each token gathers context from earlier tokens | Attention | Simulation |
| Feed-forward or experts | Each token's vector is transformed | MoE | Simulation |
| Output | A probability for every vocabulary token | GPT architecture | |
| Sampling | One token is chosen | Simulation |
Sampling settings
Temperature sharpens or flattens the probabilities: low values make answers predictable, high values more varied. Top-p keeps only the smallest set of tokens whose probabilities add up to p. Beam search explores several continuations and suits translation more than chat. Structured outputs restrict sampling to tokens that keep the output valid JSON.
Why generation is fast: the KV cache
Each new token needs the keys and values of all earlier tokens. Storing them in the KV cache means each step computes only the new token. The cache grows with context length and number of users, which is why serving long contexts is limited by GPU memory. See the KV cache simulation.
Reasoning models
Reasoning models are trained to produce a long chain of intermediate thinking before the final answer, spending more test-time compute on hard questions. The mechanics are the same next-token loop; the training teaches the model to use it to think.
Frequently asked questions
Why do LLMs make things up?
The model predicts plausible text, not verified facts. When knowledge is missing or ambiguous, a fluent but wrong continuation can be the most likely one. Retrieval and citations reduce this; see RAG and grounding.
What limits the context window?
Attention cost and KV-cache memory both grow with context length, so long contexts are expensive to train and serve.
Summary
Tokens become vectors, attention mixes context, the output layer scores every token, sampling picks one, and the KV cache makes the loop fast.
Back to the topic
This page is part of Generative AI and LLMs, which maps every concept, tool, simulation and article on the subject.