# GRPO: reinforcement learning from groups of answers

Reasoning models such as DeepSeek-R1 were trained with GRPO. For each prompt the model writes several answers, a reward function scores them, and each answer is reinforced in proportion to how much better it is than the group's average. There is no separate value network: the group itself is the baseline.

<div data-sim="grpo"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The policy is a model that can give six answers to “What is 17 + 25?”, three right and three wrong, with the probabilities shown as bars; the grey tick is where the reference model started. Each training step samples a group of G answers, scores them, turns the scores into advantages (reward minus the group mean, divided by the group's spread) and takes four clipped policy-gradient steps. The chart tracks the probability of a correct answer, the mean reward and the KL divergence from the reference.

## Things to try

- Press Train with the defaults: the probability of a correct answer climbs past 90% within a few dozen steps.
- Set the group size to 2 and train: many groups have identical rewards, every advantage is 0 and learning stalls.
- Switch to the reward model that likes long answers and set the KL penalty to 0: the long wrong answer takes over. That is reward hacking.
- Raise the KL penalty: the policy stays near the reference and learns more slowly.
- Try clip ε = 0.5 with a high learning rate: updates get larger and the curves become noisier.

## How it works in real systems

In real training each 'answer' is a full model completion of hundreds or thousands of tokens, the policy is the language model itself, and the advantage of an answer is applied to every token in it. DeepSeek used rule-based rewards that check the final answer of maths and code problems and the output format. Libraries such as Hugging Face TRL, verl and OpenRLHF implement GRPO; the group size is typically 8 to 64 completions per prompt.














<!-- simx:start -->
## Key formulas

$$\hat A_i=\frac{r_i-\operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}$$

The group-relative advantage: no value network, the group is the baseline.

$$\mathcal J=\frac1G\sum_{i}\min\!\Big(\rho_i\hat A_i,\ \mathrm{clip}(\rho_i,1-\epsilon,1+\epsilon)\hat A_i\Big)-\beta\,\mathrm{KL}(\pi_\theta\Vert\pi_{\text{ref}}),\quad \rho_i=\frac{\pi_\theta(o_i)}{\pi_{\text{old}}(o_i)}$$

The GRPO objective, maximised.

## Common mistakes

- Rewards a policy can game: a learned reward model is a proxy and the policy will find its gaps.
- Groups that are too small: identical rewards give zero advantage and waste the samples.
- Removing the KL term without watching for drift: the policy can lose abilities the reward does not measure.

## Real-world numbers

- GRPO was introduced in DeepSeekMath (2024) and used to train DeepSeek-R1.
- DeepSeek-R1-Zero was trained with GRPO on rule-based rewards for correctness and format, without supervised fine-tuning first.
- GRPO drops the value network, one of the four models PPO-based RLHF keeps in memory (policy, value, reward and reference).

## Further reading

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning (GRPO)](https://arxiv.org/abs/2402.03300)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL](https://arxiv.org/abs/2501.12948)
- [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347)

<!-- simx:end -->

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · GRPO on a toy policy

Six answers, rewards from a checker, group-relative advantages and the clipped objective, in NumPy.

```bash
pip install numpy trl datasets transformers
```

```python
import numpy as np
rng = np.random.default_rng(0)
correct = np.array([1, 1, 1, 0, 0, 0])
theta = np.array([0.8, 0.1, -0.6, 0.9, 0.4, 0.2])
softmax = lambda z: np.exp(z - z.max()) / np.exp(z - z.max()).sum()
ref = softmax(theta)
G, lr, eps, beta = 8, 0.3, 0.2, 0.04

for step in range(200):
    old = softmax(theta)
    group = rng.choice(6, size=G, p=old)
    r = correct[group].astype(float)
    adv = (r - r.mean()) / (r.std() + 1e-8)
    for _ in range(4):                               # a few clipped updates per batch
        p = softmax(theta)
        grad = np.zeros(6)
        for a, A in zip(group, adv):
            rho = p[a] / old[a]
            if (A > 0 and rho > 1 + eps) or (A < 0 and rho < 1 - eps):
                continue                              # clipped: no gradient
            grad += A * rho * (np.eye(6)[a] - p) / G
        kl = (p * np.log(p / ref)).sum()
        grad -= beta * p * (np.log(p / ref) - kl)    # gradient of the KL penalty
        theta += lr * grad
print('P(correct) =', softmax(theta)[correct == 1].sum().round(3))
```

### Step 2 · GRPO with TRL

The same algorithm on a real language model. `reward_funcs` scores each completion; `num_generations` is the group size.

```python
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

dataset = load_dataset('trl-lib/tldr', split='train')

def reward_len(completions, **kwargs):
    # a simple rule-based reward: prefer completions close to 20 characters
    return [-abs(20 - len(c)) for c in completions]

args = GRPOConfig(output_dir='Qwen2-0.5B-GRPO', num_generations=8, logging_steps=10)
trainer = GRPOTrainer(model='Qwen/Qwen2-0.5B-Instruct', reward_funcs=reward_len, args=args, train_dataset=dataset)
trainer.train()
```

Watch the logged reward and KL. A length reward like this one is easy to game, exactly as in the simulation's reward-hacking setup.

<div class="cl-end"></div>

## More simulations

[← Speculative decoding](/ai-simulations/speculative-decoding) · [All AI Simulations](/ai-simulations) · [CLIP →](/ai-simulations/clip-contrastive-learning)
