# Evaluation: confidence intervals, leaked test data and pass@k

A benchmark score is a measurement with an error bar. This simulation re-runs a benchmark 2,000 times to show the spread, tests whether one model really beats another, adds test data leakage and computes pass@k for code.

<div data-sim="tfseval"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The histograms show the scores two models get across re-runs, with the 95% interval shaded. The side panel gives the p-value for the difference and the number of questions needed to detect it. The bottom chart shows pass@k for problems of three difficulties.

## Things to try

- With 100 questions, put the models three points apart: the intervals overlap heavily.
- Move to 5,000 questions: the same gap becomes clear.
- Leak 20% of the test set into model B: its score jumps with no change in skill.
- Read pass@k at k = 10 for problems solved by one sample in twenty.

## How it works in real systems

Evaluation teams report intervals, test for contamination with n-gram overlap and canary strings, rotate held-out sets and prefer fresh benchmarks. pass@k uses the unbiased estimator from the HumanEval paper (Chen et al., 2021).

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Intervals and a significance test

```python
from statsmodels.stats.proportion import proportion_confint, proportions_ztest
n, a, b = 500, 360, 375                    # correct answers of model A and B
print(proportion_confint(a, n, method="wilson"), proportion_confint(b, n, method="wilson"))
z, p = proportions_ztest([b, a], [n, n]); print("p-value", round(p, 3))
```

### Step 2 · pass@k, the unbiased way

```python
import numpy as np
def pass_at_k(n, c, k):
    if n - c < k: return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
print([round(pass_at_k(20, 1, k), 3) for k in (1, 5, 10)])
```

### Step 3 · Check for contamination

```python
def ngrams(text, n=13):
    w = text.lower().split(); return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
train_index = set()
for doc in training_sample: train_index |= ngrams(doc)
leaked = [q for q in test_set if ngrams(q["question"]) & train_index]
print(f"{len(leaked)}/{len(test_set)} test items share a 13-gram with training data")
```

<div class="cl-end"></div>

## More simulations

[← SFT and DPO](/ai-simulations/train-post-training-sft-dpo) · [All AI Simulations](/ai-simulations) · [Training run →](/ai-simulations/training-run)
