Özer Cem KELAHMETKELAHMET
Topics · Generative AI and LLMs

Inside a large language model: tokens, attention, sampling and the KV cache

A language model is a stack of transformer layers that predicts the next token. This page follows one request through the model, from tokens to the generated answer.

Overview

When you send a prompt, the text is cut into tokens, each token becomes a vector, dozens of transformer layers mix information between the vectors, and the last layer produces a probability for every token in the vocabulary. One token is chosen, appended, and the process repeats until the answer is complete.

One request, step by step

Step What happens Read more Try it
Tokenisation Text is split into sub-word tokens BPE, SentencePiece Simulation
Embeddings Each token becomes a vector Embeddings Simulation
Position The model is told the order of tokens Transformers Simulation
Attention Each token gathers context from earlier tokens Attention Simulation
Feed-forward or experts Each token's vector is transformed MoE Simulation
Output A probability for every vocabulary token GPT architecture
Sampling One token is chosen Simulation

Sampling settings

Temperature sharpens or flattens the probabilities: low values make answers predictable, high values more varied. Top-p keeps only the smallest set of tokens whose probabilities add up to p. Beam search explores several continuations and suits translation more than chat. Structured outputs restrict sampling to tokens that keep the output valid JSON.

Why generation is fast: the KV cache

Each new token needs the keys and values of all earlier tokens. Storing them in the KV cache means each step computes only the new token. The cache grows with context length and number of users, which is why serving long contexts is limited by GPU memory. See the KV cache simulation.

Reasoning models

Reasoning models are trained to produce a long chain of intermediate thinking before the final answer, spending more test-time compute on hard questions. The mechanics are the same next-token loop; the training teaches the model to use it to think.

Frequently asked questions

Why do LLMs make things up?

The model predicts plausible text, not verified facts. When knowledge is missing or ambiguous, a fluent but wrong continuation can be the most likely one. Retrieval and citations reduce this; see RAG and grounding.

What limits the context window?

Attention cost and KV-cache memory both grow with context length, so long contexts are expensive to train and serve.

Summary

Tokens become vectors, attention mixes context, the output layer scores every token, sampling picks one, and the KV cache makes the loop fast.

Back to the topic

This page is part of Generative AI and LLMs, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open