# Hyperparameter tuning

Hyperparameter tuning finds the settings a model cannot learn by itself. Learn grid, random and Bayesian search, Hyperband, nested cross-validation and budgets.

## Overview

A machine learning model has two kinds of numbers. Parameters, such as the weights of a network or the split points of a tree, are learnt from data during training. Hyperparameters, such as the learning rate, the number of trees or the strength of a penalty, are set before training starts and control how the learning happens. Hyperparameter tuning is the search for the hyperparameter values that make the trained model perform best on data it has not seen.

Tuning matters because the same algorithm can be excellent or useless depending on its settings. A [gradient-boosting](/tools/gradient-boosting) model with a learning rate that is too high overfits in a few rounds; one with a learning rate that is too low never finishes learning. A network with the wrong learning rate diverges or crawls. Yet each trial of a hyperparameter setting means training a model, which can take seconds or days. Good tuning is therefore a problem of search under a budget: find good settings with as few expensive trials as possible, and measure them honestly.

An everyday analogy is tuning a recipe. The ingredients are fixed, but oven temperature, baking time and the amount of sugar are your choices. You cannot calculate the perfect values; you bake, taste and adjust. Trying every combination would take months. A sensible cook tries a few widely different settings, notices which direction tastes better, and narrows in, stopping obviously burnt batches early instead of waiting for the timer.

## What it is

| Method | Idea | Best for |
|---|---|---|
| Manual tuning | A person adjusts settings from experience | Early exploration, small budgets |
| Grid search | Try every combination on a fixed grid | One or two hyperparameters, cheap trials |
| Random search | Sample settings at random from ranges | A sensible default for many dimensions |
| Bayesian optimisation | Fit a model of score versus settings and pick promising points | Expensive trials, up to a few dozen dimensions |
| Successive halving and Hyperband | Start many trials cheaply, keep only the best, give them more budget | Iterative training where early results are informative |
| Population-based training | Evolve a population of runs, copying and perturbing the best during training | Large deep learning and reinforcement learning jobs |
| Evolutionary and other search | Mutate and select good configurations | Complex, conditional or architectural spaces |

Every tuning setup has four parts. A **search space** lists each hyperparameter, its type (continuous, integer, categorical) and its range, often on a log scale. An **objective** is the validation metric to optimise, usually from [cross-validation](/tools/cross-validation). A **search strategy** decides which settings to try next. A **budget** limits trials, time or compute. Tools such as [Optuna](/tools/optuna) and [Ray Tune](/tools/ray-tune) let you plug these parts together.

## Where it came from

For decades, hyperparameters were set by hand or by grid search, which was easy to run in parallel and easy to explain. In 2012 James Bergstra and Yoshua Bengio published "Random search for hyper-parameter optimization" in the Journal of Machine Learning Research. They showed that random search usually finds good settings faster than grid search, because in most problems only a few hyperparameters really matter, and a grid wastes most trials repeating the same values of the important ones.

Model-based methods followed. Bergstra and colleagues introduced the tree-structured Parzen estimator (TPE) in "Algorithms for hyper-parameter optimization" at NeurIPS 2011. Jasper Snoek, Hugo Larochelle and Ryan Adams's "Practical Bayesian optimization of machine learning algorithms" at NeurIPS 2012 used Gaussian processes and showed that [Bayesian optimisation](/tools/bayesian-optimization) could beat expert manual tuning.

Then came methods that save budget by stopping poor trials early. Kevin Jamieson and Ameet Talwalkar applied successive halving to hyperparameter optimisation at AISTATS 2016. Lisha Li and colleagues introduced Hyperband (arXiv 2016, Journal of Machine Learning Research 2018), which runs successive halving with several trade-offs between the number of trials and the budget per trial. Stefan Falkner, Aaron Klein and Frank Hutter combined Hyperband with Bayesian optimisation in BOHB at ICML 2018. Liam Li and colleagues described ASHA, an asynchronous version for large clusters, at MLSys 2020. DeepMind's Max Jaderberg and colleagues introduced population-based training in 2017. On the evaluation side, Gavin Cawley and Nicola Talbot's 2010 JMLR paper "On over-fitting in model selection and subsequent selection bias in performance evaluation" explained why tuning and evaluation must be separated, the motivation for nested cross-validation.

On the tooling side, Optuna was described by Takuya Akiba and colleagues at KDD 2019, and Ray Tune, part of the Ray project, was described by Richard Liaw and colleagues at UC Berkeley in a 2018 paper.

## How it works

### Step 1: decide what to tune

Not every hyperparameter matters equally. For gradient-boosted trees, the learning rate, number of trees, tree size (depth or leaves), minimum samples per leaf and subsampling rates usually matter most. For neural networks, the learning rate dominates, followed by batch size, weight decay, the [learning-rate schedule](/tools/learning-rate-schedules) and dropout. For an [SVM](/tools/svm), the penalty C and the kernel width matter most. Start with the two to five that matter most and fix the rest at sensible defaults.

### Step 2: define the search space

Use log scales for anything multiplicative: learning rates, penalties and kernel widths often range over several orders of magnitude, such as 10⁻⁵ to 10⁻¹. Use integer ranges for counts and categorical choices for options such as optimiser type. Some spaces are conditional: the momentum value only matters if the optimiser is SGD. Bayesian and TPE methods handle these better than grids.

### Step 3: choose a validation scheme

The objective must be measured on data not used for training. For small and medium datasets, k-fold cross-validation gives a more stable estimate; for large data or deep learning, a single held-out validation set is usual. Keep a separate test set that the search never sees.

### Step 4: pick a search strategy

**Grid search** evaluates every combination. With 4 values for each of 5 hyperparameters that is 4⁵ = 1,024 trials, so it only suits very small spaces. **Random search** samples independently from each range. With the same number of trials it explores many more distinct values of each important hyperparameter. A rough rule from probability: if the top 5 per cent of the space is "good", 60 random trials find at least one good setting with about 95 per cent probability, because 1 − 0.95⁶⁰ ≈ 0.95.

**Bayesian optimisation** fits a cheap surrogate model to the results so far and uses an acquisition function to choose the next setting, balancing exploration of uncertain regions against exploitation of known good ones.

```text
Gaussian-process surrogate:  score(θ) ~ GP(μ(θ), k(θ, θ′))
Expected improvement:        EI(θ) = E[ max(0, score(θ) − score_best) ]
TPE:                         choose θ that maximises  ℓ(θ) / g(θ)
Successive halving rounds:   n_i = n_0 / ηⁱ  trials,  r_i = r_0 · ηⁱ  budget each
Nested CV estimate:          score = mean over outer folds of score(test_k | θ*_k)
```

Here θ is a hyperparameter setting, GP is a Gaussian process with mean function μ and kernel k that measures how similar two settings are, score_best is the best score observed so far, ℓ(θ) and g(θ) are TPE's density estimates for settings in the good and the bad groups of past trials, η is the halving factor (for example 3), n_0 and r_0 are the initial number of trials and budget per trial, i is the round number, and θ\*_k is the setting chosen by an inner search on outer training fold k.

**Successive halving** starts many trials with a small budget (a few epochs, a few trees or a subset of rows), keeps the best 1/η, and multiplies their budget by η, repeating until one or a few remain. **Hyperband** runs several successive-halving brackets with different starting sizes, which hedges against the risk that early results are misleading. **ASHA** promotes trials asynchronously, so workers never wait for a whole round to finish. **BOHB** replaces Hyperband's random sampling with a model-based proposal.

**Population-based training (PBT)** trains a population of models in parallel. Periodically, poorly performing members copy the weights and hyperparameters of better ones (exploit) and then perturb the hyperparameters (explore). The result is not one fixed setting but a schedule of hyperparameters that changed during training, which is especially useful for learning rates and for reinforcement learning.

### Step 5: evaluate honestly with nested cross-validation

If you tune on the same folds that you report, the reported score is optimistic, because the search picked the setting that happened to do best on those folds. Nested cross-validation fixes this. An outer loop splits data into k folds. For each outer fold, an inner cross-validation on the remaining data runs the whole tuning procedure, the chosen setting is retrained on all inner data and scored on the outer fold. The average outer score estimates how well "the model plus its tuning procedure" generalises. It is expensive, so it is used mainly for small datasets and for publication-quality estimates.

### Step 6: manage the budget

Decide the budget in advance: number of trials, wall-clock time or GPU hours. Use early stopping inside each trial and a pruner or scheduler across trials, so bad settings die early. Run trials in parallel where possible; random search and ASHA parallelise very well, while sequential Bayesian methods gain less from many workers. Log every trial with a tool such as [MLflow](/tools/mlflow) or [Weights & Biases](/tools/weights-and-biases), so results survive and can be compared.

### A code example with Optuna

```python
import optuna
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

def objective(trial):
    model = HistGradientBoostingClassifier(
        learning_rate=trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),
        max_leaf_nodes=trial.suggest_int("max_leaf_nodes", 8, 128, log=True),
        min_samples_leaf=trial.suggest_int("min_samples_leaf", 5, 200, log=True),
        l2_regularization=trial.suggest_float("l2_regularization", 1e-8, 10.0, log=True),
        max_iter=500, early_stopping=True, random_state=0,
    )
    return cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc").mean()

study = optuna.create_study(direction="maximize")   # TPE sampler by default
study.optimize(objective, n_trials=60)
print(study.best_params, study.best_value)
```

When no sampler or pruner is given, Optuna uses `TPESampler` and `MedianPruner`. In scikit-learn, `GridSearchCV` and `RandomizedSearchCV` cover the simpler strategies, and `HalvingGridSearchCV` and `HalvingRandomSearchCV` implement successive halving; the halving classes are still marked experimental and need `from sklearn.experimental import enable_halving_search_cv`. Ray Tune's default scheduler runs trials first-in, first-out, and its documentation recommends its ASHA scheduler over plain Hyperband.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Hyperparameter | A setting chosen before training that controls how learning happens |
| Parameter | A value learnt from data during training, such as a weight |
| Search space | The set of hyperparameters, their types and ranges |
| Trial | One training and evaluation run with one setting |
| Objective | The validation metric the search optimises |
| Surrogate model | A cheap model that predicts the score of untried settings |
| Acquisition function | The rule that picks the next setting from the surrogate |
| Multi-fidelity | Using cheap, partial evaluations to judge settings early |
| Pruning | Stopping an unpromising trial before it finishes |
| Successive halving | Repeatedly keeping the best fraction of trials and giving them more budget |
| Population-based training | Evolving a group of runs and their hyperparameters during training |
| Nested cross-validation | Tuning inside an inner loop and scoring on an outer loop |

## A worked example

This is an illustrative scenario. A team predicts customer churn from 80,000 rows and 60 features with a histogram gradient-boosting classifier. Each five-fold cross-validated trial takes about 40 seconds on their machine, and they allow two hours, or about 180 trials.

**Defaults.** Cross-validated ROC-AUC is 0.842.

**Grid search.** Three values for each of four hyperparameters is 81 trials, about 54 minutes. Best ROC-AUC 0.851. The best learning rate sits at the edge of the grid, which suggests the grid was in the wrong place.

**Random search.** 90 trials over log-scaled ranges, about an hour. Best ROC-AUC 0.857. A plot of score against each hyperparameter shows that learning rate and minimum leaf size matter; L2 regularisation barely matters.

**TPE with pruning.** 180 trials in Optuna, with each trial reporting its score after each fold and the median pruner stopping trials that fall behind. About 40 per cent of trials are pruned after one or two folds, so the search finishes in roughly 75 minutes. Best ROC-AUC 0.861.

**Honest estimate.** The team runs a smaller nested cross-validation, five outer folds with a 60-trial inner search, and gets 0.855 ± 0.004. That is slightly below the 0.861 from the search itself, which is the expected optimism of picking the best of many trials. They report 0.855 and confirm it on an untouched test set.

## Strengths and limitations

**Strengths:**

- Often gives large gains over defaults for little human effort.
- Systematic, repeatable and easy to log.
- Modern methods (TPE, Hyperband, ASHA) find good settings with far fewer full trainings than grids.
- Parallelises well on clusters with tools such as Ray Tune.
- Reveals which hyperparameters actually matter for a problem.

**Limitations:**

- Every trial costs a full or partial training run, so compute bills grow fast.
- Easy to overfit the validation set with too many trials and too little data.
- Early-stopping methods assume early results predict final ones, which is not always true.
- Results depend on the search space; a badly placed range cannot be fixed by a clever strategy.
- Gains from tuning are often smaller than gains from better data or [feature engineering](/tools/feature-engineering).

## How it compares with alternatives

| Approach | Trade-off | When to choose it |
|---|---|---|
| Good defaults | No search cost, sometimes far from optimal | Quick baselines; libraries like [CatBoost](/tools/catboost) have strong defaults |
| Manual tuning by an expert | Uses intuition, hard to reproduce | Early exploration, very expensive models |
| Random search | Simple, parallel, robust | Default for most projects |
| Bayesian optimisation / TPE | Sample-efficient, more sequential | Expensive trials, modest dimensions |
| Hyperband / ASHA | Cheap screening of many settings | Iterative training with informative early results |
| Tuning-free models | No per-dataset search | Small tabular data with models such as [TabPFN](/tools/tabpfn) |

## Practical guidance and best practices

1. Start from library defaults and get a baseline score before tuning anything.
2. Tune the few hyperparameters that matter most first; fix the rest.
3. Use log-scaled ranges for learning rates, penalties and kernel widths.
4. Prefer random search over grid search when there are more than two hyperparameters.
5. Use early stopping within trials and a pruner or scheduler across trials to save budget.
6. Keep a final test set completely outside the search, and use nested cross-validation for small datasets.
7. Set a seed and log every trial's settings, scores and run time.
8. Look at results, not just the best value: plots of score against each hyperparameter show whether ranges should widen or shift.
9. When the best value sits at the edge of a range, extend the range and search again.
10. Retrain the final model with the chosen settings on all training data before testing.

## Common mistakes

- Reporting the best cross-validation score from the search as the final performance estimate.
- Tuning on the test set, even "just once".
- Using linear ranges for learning rates, so almost all trials land in the top decade.
- Grid-searching six hyperparameters and running out of budget before covering the important one.
- Fitting preprocessing, such as scaling or target encoding, outside the cross-validation loop, which leaks information into every trial.
- Comparing tuned and untuned models and calling the tuned one "a better algorithm".

## Frequently asked questions

### What is the difference between a parameter and a hyperparameter?

Parameters are learnt from data during training, such as the coefficients of a regression or the weights of a network. Hyperparameters are chosen before training and govern the learning process, such as the learning rate, the number of layers or the penalty strength. Tuning changes hyperparameters; training changes parameters.

### Is random search really better than grid search?

Usually, when there are more than two hyperparameters. Bergstra and Bengio showed that because only a few hyperparameters tend to matter, random search tries many more distinct values of the important ones for the same budget. Grid search remains fine for one or two hyperparameters with a known good range.

### When should I use Bayesian optimisation?

When each trial is expensive, the number of hyperparameters is moderate and you can afford to run trials mostly in sequence. It squeezes the most out of a small number of trials. When you have many parallel workers and cheap trials, random search or ASHA may finish sooner in wall-clock time.

### How many trials do I need?

It depends on the number of important hyperparameters and the cost of each trial. A few dozen random or TPE trials often get most of the benefit for a handful of hyperparameters. Watch the curve of best score against trial number: when it has been flat for a long stretch, more trials are unlikely to help much.

### Do I always need nested cross-validation?

No. It is the honest way to estimate the performance of a model plus its tuning procedure on small data, but it multiplies the cost. With plenty of data, a separate untouched test set does the same job more cheaply. Use nested cross-validation when data is scarce or when you need an unbiased comparison of tuned algorithms.

### Which tool should I use?

For scikit-learn models and simple searches, `GridSearchCV` and `RandomizedSearchCV` are enough. For more efficient search with pruning, Optuna is a popular choice and works with any Python training code. For distributed search across many machines or GPUs, Ray Tune adds schedulers such as ASHA and population-based training and can use Optuna as a search algorithm.

### Should I tune before or after feature engineering?

Do the main feature work first with reasonable defaults, because changing features shifts the best hyperparameters. Then tune. If a later feature change is large, re-tune at least the most important hyperparameters, since the earlier optimum may no longer hold.

## Glossary

- Grid search: evaluating every combination from fixed lists of values.
- Random search: sampling settings independently from given distributions.
- Bayesian optimisation: model-based search that uses a surrogate and an acquisition function.
- Tree-structured Parzen estimator (TPE): a Bayesian method that models good and bad settings separately.
- Expected improvement: an acquisition function measuring how much a setting is expected to beat the current best.
- Hyperband: a method that runs successive halving with several different starting budgets.
- ASHA: an asynchronous version of successive halving for parallel clusters.
- BOHB: a combination of Bayesian optimisation and Hyperband.
- Pruner: a rule that stops unpromising trials early.
- Budget: the trials, time or compute allowed for a search.

## Summary

Hyperparameter tuning searches for the training settings that make a model perform best on unseen data. It needs a search space, an honest validation objective, a strategy and a budget. Grid search suits tiny spaces; random search is the robust default; Bayesian optimisation and TPE make each expensive trial count; successive halving, Hyperband and ASHA screen many settings cheaply by stopping poor ones early; and population-based training adapts hyperparameters during training. Keep tuning separate from final evaluation, with a held-out test set or nested cross-validation, use log scales and early stopping, log every trial, and remember that better data and features often beat better hyperparameters.