Özer Cem KELAHMETKELAHMET
AI Simulations · Training from scratch

Evaluation: confidence intervals, leaked test data and pass@k

A benchmark score is a measurement with an error bar. This simulation re-runs a benchmark 2,000 times to show the spread, tests whether one model really beats another, adds test data leakage and computes pass@k for code.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The histograms show the scores two models get across re-runs, with the 95% interval shaded. The side panel gives the p-value for the difference and the number of questions needed to detect it. The bottom chart shows pass@k for problems of three difficulties.

Things to try

  • With 100 questions, put the models three points apart: the intervals overlap heavily.
  • Move to 5,000 questions: the same gap becomes clear.
  • Leak 20% of the test set into model B: its score jumps with no change in skill.
  • Read pass@k at k = 10 for problems solved by one sample in twenty.

How it works in real systems

Evaluation teams report intervals, test for contamination with n-gram overlap and canary strings, rotate held-out sets and prefer fresh benchmarks. pass@k uses the unbiased estimator from the HumanEval paper (Chen et al., 2021).

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Intervals and a significance test

from statsmodels.stats.proportion import proportion_confint, proportions_ztest
n, a, b = 500, 360, 375                    # correct answers of model A and B
print(proportion_confint(a, n, method="wilson"), proportion_confint(b, n, method="wilson"))
z, p = proportions_ztest([b, a], [n, n]); print("p-value", round(p, 3))

Step 2 · pass@k, the unbiased way

import numpy as np
def pass_at_k(n, c, k):
    if n - c < k: return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
print([round(pass_at_k(20, 1, k), 3) for k in (1, 5, 10)])

Step 3 · Check for contamination

def ngrams(text, n=13):
    w = text.lower().split(); return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
train_index = set()
for doc in training_sample: train_index |= ngrams(doc)
leaked = [q for q in test_set if ngrams(q["question"]) & train_index]
print(f"{len(leaked)}/{len(test_set)} test items share a 13-gram with training data")

More simulations

← SFT and DPO · All AI Simulations · Training run →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open