Evaluating machine learning models: splits, metrics, leakage and calibration
A model is only as good as the way it was measured. This page covers splitting data, choosing metrics, spotting leakage and checking whether predicted probabilities can be trusted.
Overview
Most failed models were not bad models; they were badly measured. A score is meaningful only on data the model never influenced, split the way the model will be used, with a metric that reflects the real decision.
Splitting the data
| Situation | Split | Why |
|---|---|---|
| Independent rows | Random train, validation, test | Standard |
| Small data | Cross-validation | Uses every row for testing once |
| Forecasting | By time, test on the future | The future must not leak into training |
| Repeated entities | By customer, patient or device | Same entity in both sets inflates scores |
Metrics
| Task | Metric | Use when |
|---|---|---|
| Classification | Accuracy | Classes balanced, errors equally costly |
| Precision and recall | False alarms and misses cost differently | |
| ROC AUC | Ranking quality, threshold not yet chosen | |
| Log loss | Probabilities matter | |
| Regression | MAE, RMSE | Typical and large errors |
| CRPS | Probabilistic forecasts |
See evaluation metrics. The model lab shows AUC, ROC, precision and recall live as you change data.
Leakage
Leakage is information from the answer hiding in the inputs: a column filled in after the outcome, statistics computed on the whole dataset before splitting, duplicates across splits. Leaked models look excellent and fail in production. Fit all preprocessing on the training split only.
Calibration
A calibrated model's 70% predictions come true about 70% of the time. Many models are over-confident. Reliability diagrams show the gap; temperature or isotonic scaling fixes it. Try the calibration simulation.
Is the difference real?
On small test sets, a few points of accuracy can be noise. Report confidence intervals and test differences; the evaluation simulation shows how interval width shrinks with test size.
Frequently asked questions
How big should the test set be?
Big enough that the confidence interval is narrower than the differences you care about; for a few points of accuracy that is usually thousands of rows.
Can I tune on the test set?
No. Tune on validation data or with cross-validation, and use the test set once at the end.
Summary
Split like production, choose metrics that match the decision, hunt for leakage, check calibration, and report uncertainty.
Back to the topic
This page is part of Machine learning, which maps every concept, tool, simulation and article on the subject.