Cross-validation
Cross-validation splits data into folds to train and test a model several times, giving a reliable estimate of performance. Learn k-fold, stratified and time-series methods.
Overview
Cross-validation estimates how well a model will perform on data it has never seen. Instead of making one split into training and test data, you divide the data into several parts, called folds. The model is trained several times, each time on all folds but one, and tested on the fold left out. Averaging the scores gives a more reliable estimate than any single split, and the spread of scores tells you how much the result depends on luck.
It answers the central question of supervised learning: not "how well does the model fit the data it saw?" but "how well will it do on the next data?" A model that only memorises its training data looks perfect on that data and fails in use. Cross-validation exposes this.
An everyday picture: a teacher who gives one exam on a single chapter learns little about what students really know. A teacher who tests on a different chapter each week, and averages the marks, gets a fairer picture. Cross-validation tests the model on each part of the data in turn.
What it is
| Method | How it splits | Use |
|---|---|---|
| Hold-out | One train and one test split | Large datasets, quick checks |
| k-fold | k equal folds, each used once as test | The general default |
| Stratified k-fold | k-fold keeping class proportions in each fold | Classification, especially imbalanced |
| Group k-fold | Keeps all rows of one group in the same fold | Repeated measures, customers, patients |
| Time-series split | Trains on the past, tests on the following period | Forecasting and any time-ordered data |
| Leave-one-out | Each single example is a fold | Very small datasets |
| Repeated k-fold | k-fold run several times with different shuffles | Reducing variance of the estimate |
| Nested cross-validation | An inner loop tunes, an outer loop evaluates | Honest estimates when tuning |
Cross-validation has two main jobs. The first is model selection: choosing between models or hyperparameter values. The second is performance estimation: reporting how good the final approach is likely to be. Using the same folds for both jobs makes the estimate optimistic, which is why nested cross-validation and a final untouched test set exist.
Where it came from
The worry behind cross-validation is old. In the early 1930s, work by Larson in psychology pointed out that evaluating a prediction method on the same data used to build it gives over-optimistic results, and suggested checking on separate data. Through the 1960s, researchers in statistics and pattern recognition used early leave-one-out procedures to estimate error rates, and Mosteller and Tukey described a primitive form in 1968.
The method was formalised in the mid-1970s. Mervyn Stone published "Cross-validatory choice and assessment of statistical predictions" in 1974, and Allen independently described leave-one-out the same year. Seymour Geisser, in 1975, described the general idea of averaging several hold-out estimates and the version with V folds, now usually called k-fold. Breiman and colleagues used cross-validation to choose tree size in CART in 1984, and it has since become the standard way to tune and compare models, built into libraries such as scikit-learn.
How it works
Step 1: set aside a final test set
If you can, hold out a test set at the start and do not touch it until the very end. Cross-validation runs on the rest.
Step 2: choose the splitting scheme
Pick the scheme that matches how the model will be used. Use stratified folds for classification, group folds when rows from the same customer or patient must not appear on both sides, and time-based splits when you will predict the future from the past.
Step 3: train and score on each fold
For each fold, train on the other folds and score on this one. Every preprocessing step that learns from data, such as scaling, imputation, encoding or feature selection, must be fitted inside the training folds only. Putting preprocessing and model together in a pipeline makes this automatic.
Step 4: summarise
Report the mean score and its spread, for example the standard deviation across folds. A model with mean AUC 0.82 and spread 0.01 is more trustworthy than one with mean 0.83 and spread 0.06.
Step 5: tune, but keep evaluation honest
When searching hyperparameters, each candidate is scored by cross-validation and the best is chosen. That best score is slightly optimistic. Nested cross-validation adds an outer loop so each outer fold is scored by a model tuned without seeing it. Tools such as Optuna and Ray Tune run these searches at scale.
Step 6: refit and test once
Refit the chosen approach on all non-test data, then score once on the final test set.
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipe, X, y, cv=cv, scoring="roc_auc")
print(scores.mean(), scores.std())
Key concepts and terms
| Term | Meaning |
|---|---|
| Fold | One of the parts the data are divided into |
| Training fold | Data used to fit the model in one round |
| Validation fold | Data used to score the model in that round |
| Generalisation | How well a model performs on new data |
| Overfitting | Fitting noise in the training data that does not repeat |
| Data leakage | Information from the validation data reaching training |
| Stratification | Keeping class proportions equal across folds |
| Grouped split | Keeping related rows together in one fold |
| Nested cross-validation | Separate loops for tuning and for evaluation |
| Bias-variance trade-off | More folds give less biased but more variable estimates |
| Test set | Data kept aside for one final evaluation |
A worked example
This scenario is illustrative. A clinic wants to predict which patients will miss appointments. It has 40,000 appointments from 9,000 patients over two years, with a 12 per cent no-show rate.
A first attempt uses ordinary shuffled five-fold cross-validation and reports ROC AUC 0.86. A reviewer points out two problems. Many patients have several appointments, so the same patient appears in training and validation folds and the model partly recognises individuals. And the clinic will predict future appointments, not random past ones.
The team switches to a scheme that groups by patient and splits by time: train on months 1 to 12 and validate on month 13, then train on 1 to 13 and validate on 14, and so on for six rounds. The AUC falls to 0.77, with a spread of 0.02 across rounds. This lower number is the honest one. A final test on the last two months, never used in tuning, gives 0.76, close to the cross-validated estimate, which confirms the scheme was realistic.
Strengths and limitations
Strengths:
- Uses all the data for both training and evaluation.
- Gives a measure of uncertainty through the spread of scores.
- Reduces the risk of a lucky or unlucky single split.
- Works with any model and any metric.
- Standard, well supported and easy to automate.
Limitations:
- Costs k times as much training as a single split.
- Gives misleading results if the split does not match real use.
- Tuning on cross-validation scores makes them optimistic.
- Assumes future data resemble past data.
- Leakage through preprocessing is easy to introduce by accident.
How it compares with alternatives
| Approach | Cost | Reliability | Best for |
|---|---|---|---|
| k-fold cross-validation | k model fits | High if the split matches use | Most datasets |
| Single hold-out split | One fit | Variable on small data | Very large data, deep learning |
| Bootstrap estimates | Many fits | Good, can be biased | Small samples, uncertainty |
| Time-based backtesting | Several fits | High for forecasting | Time series, see ARIMA |
| Online evaluation and model monitoring | Ongoing | Measures real behaviour | Deployed systems |
Practical guidance and best practices
- Choose a split that mirrors production: time order, groups and class balance.
- Put all preprocessing inside a pipeline so it is fitted per fold.
- Use five or ten folds as a default; use repeated folds when data are small.
- Report the mean and spread of the score, not just the mean.
- Use nested cross-validation or a separate test set when you tune heavily.
- Fix random seeds so experiments can be repeated.
- Use the same folds when comparing models, so differences are fair.
- Check that the final test result is close to the cross-validated estimate.
Common mistakes
- Scaling, imputing or selecting features on the full dataset before splitting.
- Shuffling time-ordered data, so the model learns from the future.
- Letting rows from the same person or device appear on both sides of a split.
- Reporting the best tuned cross-validation score as the expected performance.
- Running many experiments against the test set until one looks good.
- Ignoring large differences between folds.
Frequently asked questions
How many folds should I use?
Five or ten folds are common defaults and work well for most datasets. More folds use more data for training each time, so estimates are less pessimistic, but they cost more and can vary more. For very large datasets, a single hold-out split is often enough.
Do I still need a test set if I use cross-validation?
It is strongly recommended when you tune or compare many models. Choosing the best of many options by cross-validation makes its score optimistic. A test set touched only once gives an unbiased final check.
What is data leakage in cross-validation?
It is any route by which information from the validation fold reaches training. Common examples are fitting a scaler on all data, selecting features using all labels, target-encoding on full data, or having the same customer in both folds. Leakage makes scores look better than they will be in practice.
How do I cross-validate time series?
Use forward-chaining splits: train on an initial period, validate on the next period, then extend the training window and repeat. Never train on data that come after the validation period. scikit-learn's TimeSeriesSplit implements this pattern.
Is leave-one-out better than k-fold?
Not usually. Leave-one-out uses almost all data for training, but it is expensive and its estimate can vary a lot. Ten-fold cross-validation is generally a better balance except on very small datasets.
Glossary
- Cross-validation: rotating which part of the data is held out and averaging the results.
- k-fold: splitting data into k equal parts, each used once for validation.
- Hold-out set: data kept aside from training for evaluation.
- Leakage: validation information leaking into training.
- Stratified split: a split that keeps class proportions constant.
- Group split: a split that keeps related rows together.
- Nested cross-validation: an inner tuning loop inside an outer evaluation loop.
- Backtesting: evaluating a forecasting model on successive future periods.
Summary
Cross-validation estimates how well a model will generalise by training and testing it on several different splits and averaging the results. Formalised by Stone and Geisser in the mid-1970s, it is now the standard way to compare models and tune hyperparameters. Its value depends on matching the split to real use, keeping all preprocessing inside the folds and separating tuning from final evaluation. Done well, it gives an honest, repeatable number with a measure of its own uncertainty.
Related
It belongs to ML foundations in my stack, under Machine learning. Others in the same group: