Evaluation: confidence intervals, leaked test data and pass@k
A benchmark score is a measurement with an error bar. This simulation re-runs a benchmark 2,000 times to show the spread, tests whether one model really beats another, adds test data leakage and computes pass@k for code.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The histograms show the scores two models get across re-runs, with the 95% interval shaded. The side panel gives the p-value for the difference and the number of questions needed to detect it. The bottom chart shows pass@k for problems of three difficulties.
Things to try
- With 100 questions, put the models three points apart: the intervals overlap heavily.
- Move to 5,000 questions: the same gap becomes clear.
- Leak 20% of the test set into model B: its score jumps with no change in skill.
- Read pass@k at k = 10 for problems solved by one sample in twenty.
How it works in real systems
Evaluation teams report intervals, test for contamination with n-gram overlap and canary strings, rotate held-out sets and prefer fresh benchmarks. pass@k uses the unbiased estimator from the HumanEval paper (Chen et al., 2021).
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Intervals and a significance test
from statsmodels.stats.proportion import proportion_confint, proportions_ztest
n, a, b = 500, 360, 375 # correct answers of model A and B
print(proportion_confint(a, n, method="wilson"), proportion_confint(b, n, method="wilson"))
z, p = proportions_ztest([b, a], [n, n]); print("p-value", round(p, 3))
Step 2 · pass@k, the unbiased way
import numpy as np
def pass_at_k(n, c, k):
if n - c < k: return 1.0
return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
print([round(pass_at_k(20, 1, k), 3) for k in (1, 5, 10)])
Step 3 · Check for contamination
def ngrams(text, n=13):
w = text.lower().split(); return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
train_index = set()
for doc in training_sample: train_index |= ngrams(doc)
leaked = [q for q in test_set if ngrams(q["question"]) & train_index]
print(f"{len(leaked)}/{len(test_set)} test items share a 13-gram with training data")