# Inside a large language model: tokens, attention, sampling and the KV cache

A language model is a stack of transformer layers that predicts the next token. This page follows one request through the model, from tokens to the generated answer.

## Overview

When you send a prompt, the text is cut into tokens, each token becomes a vector, dozens of transformer layers mix information between the vectors, and the last layer produces a probability for every token in the vocabulary. One token is chosen, appended, and the process repeats until the answer is complete.

## One request, step by step

| Step | What happens | Read more | Try it |
|---|---|---|---|
| Tokenisation | Text is split into sub-word tokens | [BPE](/tools/bpe), [SentencePiece](/tools/sentencepiece) | [Simulation](/ai-simulations/tokenisation-bpe) |
| Embeddings | Each token becomes a vector | [Embeddings](/tools/embeddings) | [Simulation](/ai-simulations/embedding-space) |
| Position | The model is told the order of tokens | [Transformers](/tools/transformers) | [Simulation](/ai-simulations/positional-encoding) |
| Attention | Each token gathers context from earlier tokens | [Attention](/tools/attention-mechanism) | [Simulation](/ai-simulations/attention) |
| Feed-forward or experts | Each token's vector is transformed | [MoE](/tools/mixture-of-experts) | [Simulation](/ai-simulations/mixture-of-experts) |
| Output | A probability for every vocabulary token | [GPT architecture](/tools/gpt-architecture) | |
| Sampling | One token is chosen | | [Simulation](/ai-simulations/sampling-temperature-top-p) |

## Sampling settings

Temperature sharpens or flattens the probabilities: low values make answers predictable, high values more varied. Top-p keeps only the smallest set of tokens whose probabilities add up to p. [Beam search](/ai-simulations/beam-search) explores several continuations and suits translation more than chat. [Structured outputs](/tools/structured-outputs) restrict sampling to tokens that keep the output valid JSON.

## Why generation is fast: the KV cache

Each new token needs the keys and values of all earlier tokens. Storing them in the [KV cache](/tools/kv-cache) means each step computes only the new token. The cache grows with context length and number of users, which is why serving long contexts is limited by GPU memory. See the [KV cache simulation](/ai-simulations/kv-cache).

## Reasoning models

[Reasoning models](/tools/reasoning-models) are trained to produce a long chain of intermediate thinking before the final answer, spending more [test-time compute](/tools/test-time-compute) on hard questions. The mechanics are the same next-token loop; the training teaches the model to use it to think.

## Frequently asked questions

### Why do LLMs make things up?

The model predicts plausible text, not verified facts. When knowledge is missing or ambiguous, a fluent but wrong continuation can be the most likely one. Retrieval and citations reduce this; see [RAG and grounding](/topics/generative-ai-llms/rag-and-grounding).

### What limits the context window?

Attention cost and KV-cache memory both grow with context length, so long contexts are expensive to train and serve.

## Summary

Tokens become vectors, attention mixes context, the output layer scores every token, sampling picks one, and the KV cache makes the loop fast.

## Back to the topic

This page is part of **[Generative AI and LLMs](/topics/generative-ai-llms)**, which maps every concept, tool, simulation and article on the subject.
