# Size the model: parameters, FLOPs, KV cache and training memory

Before renting a single GPU, a team turns the architecture into numbers: how many parameters, how many FLOPs per token, how much memory for training. This simulation does that arithmetic for any transformer you describe.

<div data-sim="tfssize"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The first bar shows where the parameters sit: embeddings, output layer, attention and MLP. The second shows training memory for one replica: bf16 weights, the fp32 master copy, gradients, Adam's two moments and activations, against the line of one 80 GB GPU. The table gives every formula used.

## Things to try

- Pick the Llama-3-8B-like preset: the count lands at about 8.0 billion, the published size.
- Set KV heads equal to heads: attention grows and the KV cache quadruples.
- Raise the context to 32,768: activations dominate memory unless you turn on checkpointing.
- Untie the embeddings on the 1B model: the output layer adds hundreds of millions of parameters.

## How it works in real systems

Teams size runs with exactly these estimates: about 6 FLOPs per parameter per token for training, about 16 bytes per parameter of state in mixed precision with Adam, and the Megatron activation formula. The numbers decide the parallelism plan and the hardware budget before any code runs.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Count parameters from the architecture

```python
def params(L, d, heads, kv_heads, mlp, vocab, swiglu=True, tied=False):
    d_kv = d // heads * kv_heads
    attn = 2 * d * d + 2 * d * d_kv
    ffn = (3 if swiglu else 2) * d * mlp
    per_layer = attn + ffn + 2 * d
    return vocab * d * (1 if tied else 2) + L * per_layer + d

print(params(32, 4096, 32, 8, 14336, 128256) / 1e9)   # ≈ 8.03 (Llama-3-8B)
```

### Step 2 · Check it against a real model

```python
# pip install transformers accelerate
from transformers import AutoConfig, AutoModelForCausalLM
from accelerate import init_empty_weights
cfg = AutoConfig.from_pretrained("meta-llama/Meta-Llama-3-8B")   # needs access approval; any Llama-style config works
with init_empty_weights():
    m = AutoModelForCausalLM.from_config(cfg)
print(sum(p.numel() for p in m.parameters()) / 1e9)
```

### Step 3 · Training memory and FLOPs

```python
def train_memory_gb(N, seq, micro, d, L, heads, recompute=False):
    state = 16 * N                                 # bf16 weights + fp32 master + grads + Adam m, v
    act = (2 * seq * micro * d if recompute else seq * micro * d * (34 + 5 * heads * seq / d)) * L
    return (state + act) / 1e9

N = 8.03e9
print(train_memory_gb(N, 8192, 1, 4096, 32, 32), "GB per replica before sharding")
print(6 * N * 15e12 / (989e12 * 0.4) / 86400, "H100-days for 15 T tokens at 40% MFU")
```

<div class="cl-end"></div>

## More simulations

[← Tokenizer and vocabulary](/ai-simulations/train-tokenizer-vocabulary) · [All AI Simulations](/ai-simulations) · [Scaling laws →](/ai-simulations/train-scaling-laws)
