Özer Cem KELAHMETKELAHMET
Topics · Machine learning

Evaluating machine learning models: splits, metrics, leakage and calibration

A model is only as good as the way it was measured. This page covers splitting data, choosing metrics, spotting leakage and checking whether predicted probabilities can be trusted.

Overview

Most failed models were not bad models; they were badly measured. A score is meaningful only on data the model never influenced, split the way the model will be used, with a metric that reflects the real decision.

Splitting the data

Situation Split Why
Independent rows Random train, validation, test Standard
Small data Cross-validation Uses every row for testing once
Forecasting By time, test on the future The future must not leak into training
Repeated entities By customer, patient or device Same entity in both sets inflates scores

Metrics

Task Metric Use when
Classification Accuracy Classes balanced, errors equally costly
Precision and recall False alarms and misses cost differently
ROC AUC Ranking quality, threshold not yet chosen
Log loss Probabilities matter
Regression MAE, RMSE Typical and large errors
CRPS Probabilistic forecasts

See evaluation metrics. The model lab shows AUC, ROC, precision and recall live as you change data.

Leakage

Leakage is information from the answer hiding in the inputs: a column filled in after the outcome, statistics computed on the whole dataset before splitting, duplicates across splits. Leaked models look excellent and fail in production. Fit all preprocessing on the training split only.

Calibration

A calibrated model's 70% predictions come true about 70% of the time. Many models are over-confident. Reliability diagrams show the gap; temperature or isotonic scaling fixes it. Try the calibration simulation.

Is the difference real?

On small test sets, a few points of accuracy can be noise. Report confidence intervals and test differences; the evaluation simulation shows how interval width shrinks with test size.

Frequently asked questions

How big should the test set be?

Big enough that the confidence interval is narrower than the differences you care about; for a few points of accuracy that is usually thousands of rows.

Can I tune on the test set?

No. Tune on validation data or with cross-validation, and use the test set once at the end.

Summary

Split like production, choose metrics that match the decision, hunt for leakage, check calibration, and report uncertainty.

Back to the topic

This page is part of Machine learning, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open