# Evaluating machine learning models: splits, metrics, leakage and calibration

A model is only as good as the way it was measured. This page covers splitting data, choosing metrics, spotting leakage and checking whether predicted probabilities can be trusted.

## Overview

Most failed models were not bad models; they were badly measured. A score is meaningful only on data the model never influenced, split the way the model will be used, with a metric that reflects the real decision.

## Splitting the data

| Situation | Split | Why |
|---|---|---|
| Independent rows | Random train, validation, test | Standard |
| Small data | [Cross-validation](/tools/cross-validation) | Uses every row for testing once |
| Forecasting | By time, test on the future | The future must not leak into training |
| Repeated entities | By customer, patient or device | Same entity in both sets inflates scores |

## Metrics

| Task | Metric | Use when |
|---|---|---|
| Classification | Accuracy | Classes balanced, errors equally costly |
| | Precision and recall | False alarms and misses cost differently |
| | ROC AUC | Ranking quality, threshold not yet chosen |
| | Log loss | Probabilities matter |
| Regression | MAE, RMSE | Typical and large errors |
| | CRPS | Probabilistic forecasts |

See [evaluation metrics](/tools/evaluation-metrics). The [model lab](/model/lab) shows AUC, ROC, precision and recall live as you change data.

## Leakage

Leakage is information from the answer hiding in the inputs: a column filled in after the outcome, statistics computed on the whole dataset before splitting, duplicates across splits. Leaked models look excellent and fail in production. Fit all preprocessing on the training split only.

## Calibration

A calibrated model's 70% predictions come true about 70% of the time. Many models are over-confident. Reliability diagrams show the gap; temperature or isotonic scaling fixes it. Try the [calibration simulation](/ai-simulations/tfm-calibration).

## Is the difference real?

On small test sets, a few points of accuracy can be noise. Report confidence intervals and test differences; the [evaluation simulation](/ai-simulations/train-evaluation) shows how interval width shrinks with test size.

## Frequently asked questions

### How big should the test set be?

Big enough that the confidence interval is narrower than the differences you care about; for a few points of accuracy that is usually thousands of rows.

### Can I tune on the test set?

No. Tune on validation data or with cross-validation, and use the test set once at the end.

## Summary

Split like production, choose metrics that match the decision, hunt for leakage, check calibration, and report uncertainty.

## Back to the topic

This page is part of **[Machine learning](/topics/machine-learning)**, which maps every concept, tool, simulation and article on the subject.
