Özer Cem KELAHMETKELAHMET
AI Simulations · Training

Bayesian optimisation: find the best setting in as few tries as possible

When each try is expensive, such as one training run for one set of hyperparameters, you cannot afford a grid search. Bayesian optimisation fits a probabilistic model to the results so far, asks where an improvement is most likely, and evaluates there next. It usually finds a good setting in a few dozen tries.

Level AdvancedAbout 12 minRead first: Linear regression1 challenge in the Lab

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Somewhere on the x-axis is a hidden function you want to maximise; think of x as a learning rate and the function as validation accuracy. The blue line is the Gaussian process's best guess and the band its uncertainty, both computed exactly from the evaluations so far. The orange curve below is the acquisition function, and the dashed line marks its peak, the next evaluation. You can also click anywhere on the chart to evaluate there yourself. The chart on the right compares the best value found with the average of random search.

Things to try

  • Press Next evaluation a few times: the band shrinks around each new point and the search moves to promising regions.
  • Press Run 10 and compare with random search on the right.
  • Switch to UCB and set exploration to 0: the search keeps refining the first good peak and can miss a better one.
  • Set a long length scale: the model becomes too smooth to notice narrow peaks.
  • Add a lot of noise: single evaluations become unreliable and the model spreads its trust over neighbours.

How it works in real systems

Hyperparameter tuners such as Google Vizier, Ax/BoTorch and Optuna use the same loop with more dimensions, fitted kernel parameters and parallel evaluations; Optuna's default sampler, TPE, is a related model-based method. Bayesian optimisation is used beyond machine learning too: tuning chemical experiments, recipes and A/B tests where each trial is slow or costly.

Key formulas

$$k(x,x')=\exp\!\Big(-\frac{(x-x')^2}{2\ell^2}\Big)$$

The squared-exponential kernel with length scale ℓ.

$$\mathrm{EI}(x)=(\mu-f^{\ast}-\xi)\,\Phi(z)+\sigma\,\varphi(z),\qquad z=\frac{\mu-f^{\ast}-\xi}{\sigma}$$

Expected improvement over the best value so far.

$$\mathrm{UCB}(x)=\mu(x)+\kappa\,\sigma(x)$$

Common mistakes

  • Using Bayesian optimisation for cheap functions: random or grid search is simpler.
  • Fixing the kernel length scale by hand instead of fitting it.
  • Ignoring noise: a noisy evaluation looks like a real peak.

Real-world numbers

  • Snoek et al. (2012) showed GP-based Bayesian optimisation beating expert tuning of neural networks.
  • Google Vizier runs black-box optimisation as a service inside Google.
  • Random search often matches grid search with far fewer trials because few hyperparameters matter (Bergstra and Bengio, 2012).

Further reading

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · A Gaussian process and expected improvement

scikit-learn fits the GP; expected improvement is three lines of SciPy.

pip install scikit-learn scipy numpy
import numpy as np
from scipy.stats import norm
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, WhiteKernel

f = lambda x: np.exp(-(x - 0.7) ** 2 / 0.01) + 0.6 * np.exp(-(x - 0.2) ** 2 / 0.02) + 0.25 * np.sin(9 * x)
grid = np.linspace(0, 1, 500)[:, None]
X = np.array([[0.15], [0.8]]); y = f(X[:, 0])

def expected_improvement(mu, sigma, best, xi=0.01):
    z = (mu - best - xi) / np.maximum(sigma, 1e-9)
    return (mu - best - xi) * norm.cdf(z) + sigma * norm.pdf(z)

Step 2 · The optimisation loop

Fit, pick the point with the highest expected improvement, evaluate, repeat.

for i in range(12):
    gp = GaussianProcessRegressor(RBF(0.1) + WhiteKernel(1e-5), normalize_y=True).fit(X, y)
    mu, sigma = gp.predict(grid, return_std=True)
    x_next = grid[np.argmax(expected_improvement(mu, sigma, y.max()))]
    X = np.vstack([X, x_next]); y = np.append(y, f(x_next[0]))
    print(f'eval {len(y):2d}: x = {x_next[0]:.3f}, best so far = {y.max():.3f}')
print('true maximum on the grid:', f(grid[:, 0]).max().round(3))

The kernel's length scale is refitted on every iteration here, unlike the simulation where you set it by hand.

More simulations

← Gradient boosting · All AI Simulations · Recommender (matrix factorisation) →

Last updated Nov 19, 2026
Type to search.
    ↑↓ navigate↵ open