Bayesian optimization

Overview

Bayesian optimization is a method for finding the best inputs of a function that is expensive to evaluate and has no usable formula or gradient. It is the standard answer to the question "how do I tune this when each trial takes minutes, hours or real money?" Instead of trying settings blindly, it learns from every trial. It builds a probabilistic model of the objective, called a surrogate, and uses it to decide which setting to try next.

The decision rule balances two urges. One is to exploit: try settings that the model believes are good. The other is to explore: try settings where the model is unsure, because a surprise may be hiding there. This balance lets Bayesian optimization reach good results in far fewer trials than grid search or random search on many problems. It is widely used for hyperparameter tuning, experiment design, and engineering design, and it connects to Optimization, Probability and Bayesian deep learning.

What it is

Bayesian optimization, often shortened to BO, solves problems of the form: find x that minimizes or maximizes f(x), where f is a black box. A black box means we can only query it. We give an input and get an output, possibly noisy. We cannot see inside, and we cannot take derivatives.

The two building blocks

1. A surrogate model. A cheap statistical model of f that gives both a predicted value and an uncertainty for any input. The classic choice is a Gaussian process.
2. An acquisition function. A rule that scores each candidate input by how useful it would be to evaluate next, using the surrogate's prediction and uncertainty.

The loop

1. Evaluate f at a few initial points.
2. Fit the surrogate to all observations so far.
3. Maximize the acquisition function to choose the next point.
4. Evaluate f at that point.
5. Add the result to the data and repeat until the budget is used up.

Typical problems

• Choosing hyperparameters of a machine learning model, such as learning rate, depth or regularization.
• Tuning a data processing pipeline where each run is long.
• Designing physical experiments or materials where each sample is costly.
• Configuring systems such as compilers, databases or simulators.
• Setting the parameters of a recommendation or ranking policy tested online.

What it is not

• It is not gradient descent. It does not need gradients.
• It is not a learning algorithm for model weights. It tunes the knobs around training.
• It is not magic for high dimensions. It shines in low to moderate dimension, often under about twenty.

Where it came from

The core ideas are decades old. In the 1960s and 1970s, researchers studied how to search for the maximum of an unknown curve by modeling it statistically and choosing points where the chance of improvement looked high. The term Bayesian optimization is tied to this early work on sequential design.

Gaussian processes and kriging

Gaussian processes were developed in spatial statistics, where they are known as kriging. They were used to interpolate measurements such as mineral content across a landscape. Their ability to give a mean and a variance at every point is exactly what optimization needs.

Efficient global optimization

In 1998 a widely cited paper, "Efficient Global Optimization of Expensive Black-Box Functions", popularized expected improvement as an acquisition function and showed how to use a Gaussian process surrogate for engineering design with costly simulations.

Machine learning adopts it

In the early 2010s, the machine learning community noticed that tuning hyperparameters is an expensive black-box problem. Key milestones include:

• The 2011 work on sequential model-based optimization with tree-structured Parzen estimators, which handled conditional and categorical settings.
• Sequential model-based algorithm configuration, known as SMAC, which uses random forests as the surrogate and handles mixed spaces well.
• The 2012 paper "Practical Bayesian Optimization of Machine Learning Algorithms", which showed that Gaussian process BO could beat expert tuning on neural networks and other models.
• The 2016 survey "Taking the Human Out of the Loop: A Review of Bayesian Optimization", which collected the field's methods and applications.
• Later work on multi-fidelity methods such as Hyperband and BOHB, which combine BO with early stopping of poor trials.

Why it stayed relevant

Models got bigger and each training run got costlier. The cost per trial rose faster than compute got cheaper. Methods that waste fewer trials became more valuable, not less.

How it works

The treasure hunt analogy

You are searching for the highest point of a hilly island at night. You can only measure the height at spots you walk to, and each walk takes a day. You have thirty days.

A foolish plan is to walk to a grid of spots. A better plan is to keep a mental map. After each measurement you update the map: "the land near here looks high, the land over there is a mystery". Each morning you choose between two kinds of spots. One kind is where your map says the ground is probably high. The other kind is a mystery area where the ground could be anything, so there might be a hidden peak.

The mental map is the surrogate. The rule you use to pick between the sure thing and the mystery is the acquisition function.

The surrogate: Gaussian process

A Gaussian process is a way to describe a distribution over smooth functions. Given observed points, it gives for any new input a bell-shaped belief: a mean (best guess) and a standard deviation (how unsure).

• Near an observed point, the standard deviation is small, because we have seen the value.
• Far from any observation, the standard deviation is large.
• The shape of the functions allowed is controlled by a kernel. The kernel says how quickly the function can change when the input moves. A smooth kernel assumes gentle hills. A rough kernel allows jagged ground.
• The kernel has length scales, one per input dimension in the common automatic relevance version. Long length scales mean the function barely depends on that input.

Fitting the process means choosing kernel settings, usually by maximizing the marginal likelihood or by sampling them.

Other surrogates

The acquisition function

The acquisition function turns the surrogate into a score for each candidate. We maximize it to pick the next trial. Because it is cheap to evaluate, we can search it thoroughly with standard optimizers or many random starts.

Common choices when minimizing f:

• Probability of improvement. The chance that the new value beats the best so far. Simple, but it ignores how big the gain might be.
• Expected improvement. The average size of the gain over the best so far, taking both the chance and the amount into account. The most used rule.
• Lower confidence bound. The predicted mean minus a multiple of the standard deviation. A larger multiple means more exploration. When maximizing it is called the upper confidence bound.
• Thompson sampling. Draw one random function from the surrogate and pick its optimum. Natural for batches.
• Knowledge gradient and entropy-based rules. Choose the point that best improves what we know about the optimum, not only the value at the point. More costly to compute.

Exploration versus exploitation

This is the center of the method.

• A pure exploiter always tries where the mean is best. It can get stuck on a mediocre hill.
• A pure explorer always tries where the uncertainty is highest. It wastes trials in boring regions.
• Acquisition functions blend the two. Their parameters, such as the confidence multiplier, set the blend.

Handling noise

Most real objectives are noisy. Retraining a model with a different seed gives a different score. A Gaussian process can include a noise term so that it does not chase random wiggles. With noise, "the best observed value" is itself uncertain, and noisy versions of expected improvement address this.

Constraints

Often there are conditions, such as memory under a limit or latency under a target. Two approaches:

• Model each constraint with its own surrogate and multiply the acquisition by the probability that all constraints hold.
• Fold constraints into the objective as a penalty. Simple, but less principled.

Batches and parallel trials

If you can run several trials at once, you need several suggestions at a time. Methods include:

• Thompson sampling, which draws several functions.
• Fantasizing, which pretends the pending trials returned their predicted values and picks the next point accordingly.
• Penalizing points near pending ones so the batch is spread out.

Multi-fidelity and early stopping

Evaluating a model on a small data subset or after a few epochs is cheaper but less accurate. Multi-fidelity methods use cheap estimates to rule out bad settings early. Hyperband allocates budgets by successive halving. Combined methods such as BOHB pair that scheduling with a model that learns where good settings live.

Stopping

Common stopping rules:

• A fixed budget of trials or time.
• Expected improvement falling below a small threshold.
• No improvement in the best value for a set number of trials.

Key concepts and terms

A worked example

A hand calculation of expected improvement

Suppose we are minimizing validation loss. The best value seen so far is 1.0. For a candidate setting, the surrogate predicts a mean of 0.8 and a standard deviation of 0.2.

Define Z as the best minus the mean, divided by the standard deviation: (1.0 minus 0.8) divided by 0.2, which is 1.

Expected improvement is: (best minus mean) times Phi(Z) plus sigma times phi(Z), where Phi is the cumulative normal distribution and phi is the normal density.

• Phi(1) is about 0.8413.
• phi(1) is about 0.2420.
• So the value is 0.2 times 0.8413 plus 0.2 times 0.2420.
• That is 0.1683 plus 0.0484, which is about 0.217.

Now compare a second candidate with a mean of 1.1 and a standard deviation of 0.5:

• Z is (1.0 minus 1.1) divided by 0.5, which is minus 0.2.
• Phi(minus 0.2) is about 0.4207 and phi(minus 0.2) is about 0.3910.
• The value is minus 0.1 times 0.4207 plus 0.5 times 0.3910.
• That is minus 0.0421 plus 0.1955, which is about 0.153.

The first candidate wins because its mean is better. But notice that the second candidate, whose mean is worse than the best, still has a substantial score because it is so uncertain. That is exploration at work. If its standard deviation were 1.0 the score would rise further, which shows why unvisited regions get attention.

A code example for tuning a model

The sketch below uses a Gaussian process based optimizer from a common open source library to tune a gradient boosted model. Any BO library with a similar interface works.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
from skopt import gp_minimize
from skopt.space import Integer, Real

X, y = load_breast_cancer(return_X_y=True)

space = [
    Real(1e-3, 0.5, prior="log-uniform", name="learning_rate"),
    Integer(1, 6, name="max_depth"),
    Integer(50, 400, name="n_estimators"),
    Real(0.5, 1.0, name="subsample"),
]

def objective(params):
    lr, depth, n_est, subsample = params
    model = GradientBoostingClassifier(
        learning_rate=lr,
        max_depth=depth,
        n_estimators=n_est,
        subsample=subsample,
        random_state=0,
    )
    score = cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean()
    return -score  # minimize negative AUC

result = gp_minimize(
    objective,
    space,
    n_calls=40,
    n_initial_points=10,
    acq_func="EI",
    noise=1e-3,
    random_state=0,
)

print("best params:", result.x)
print("best AUC:", -result.fun)

A from-scratch loop in plain Python

The following shows the structure without a library. It uses a Gaussian process from a general toolkit and picks the next point by scoring random candidates.

import numpy as np
from scipy.stats import norm
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern, WhiteKernel

def f(x):  # the expensive black box (here a cheap stand-in)
    return np.sin(3 * x) + 0.5 * x**2

rng = np.random.default_rng(0)
X = rng.uniform(-2, 2, size=(4, 1))
y = f(X).ravel()

kernel = Matern(nu=2.5) + WhiteKernel(1e-4)
for step in range(20):
    gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True)
    gp.fit(X, y)

    cand = rng.uniform(-2, 2, size=(2000, 1))
    mu, sd = gp.predict(cand, return_std=True)
    best = y.min()
    z = (best - mu) / np.maximum(sd, 1e-9)
    ei = (best - mu) * norm.cdf(z) + sd * norm.pdf(z)

    x_next = cand[np.argmax(ei)].reshape(1, -1)
    X = np.vstack([X, x_next])
    y = np.append(y, f(x_next).ravel())

print("best x:", X[np.argmin(y)], "best f:", y.min())

Reading the run

• Plot the best value so far against the trial number. It should drop quickly and then flatten.
• Plot the chosen points. Early ones are spread out, later ones cluster near good regions, with occasional jumps for exploration.
• Plot the surrogate mean and band against the true curve in one dimension. The band should shrink where you sampled.

Strengths and limitations

Strengths

• Sample efficiency. Few trials are needed compared with grid or random search on smooth problems.
• No gradients. Works with any black box, including non-differentiable ones.
• Handles noise. The surrogate can model noisy outcomes.
• Principled uncertainty. The trade-off between exploring and exploiting is built in.
• Flexible. Supports constraints, batches, multiple objectives, and multiple fidelities.
• Learns structure. Length scales reveal which inputs matter.

Limitations

• Dimension. Gaussian process BO struggles beyond tens of dimensions, because distance becomes less informative and fitting is costly.
• Cost of the surrogate. Exact Gaussian processes scale cubically with the number of observations.
• Mixed and conditional spaces. Standard kernels expect continuous inputs. Categorical and conditional parameters need special handling or other surrogates.
• Non-smooth objectives. If the objective jumps, a smooth kernel fits poorly.
• Heavy noise. Very noisy objectives need many repeats.
• Sensitivity to setup. Search space bounds, scales, and priors affect results strongly.
• Sequential nature. Gains shrink when you can run many trials in parallel and the budget is large.
• Local optima of the acquisition. Maximizing the acquisition function is itself a non-convex problem.

How it compares with alternatives

Rules of thumb

• With fewer than about twenty parameters, a modest budget, and costly trials, use Bayesian optimization.
• With a huge budget of cheap trials, random search is a strong baseline.
• With a training curve that tells you early whether a trial is bad, add early stopping.
• With many categorical and conditional parameters, use a tree-based surrogate.
• Always compare with random search at equal budget. If BO does not win, something is wrong in the setup.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns of mixed types, missing cells, and often unreliable column names. Models such as TabPFN and TabICL are pre-trained on many synthetic or real tables and then predict through in-context learning: the labeled rows are passed in as context and a single forward pass yields predictions, with no weight updates. See /model/research and /model/architecture.

Bayesian optimization appears in this story in several concrete places, some at pre-training time and some at use time.

Why tuning matters less, and where it still matters

A headline benefit of in-context tabular models is that they need little or no hyperparameter search for a new table. The model adapts through the context, not through fitted weights. This removes one of the biggest uses of BO in classical tabular work, where gradient boosted trees need careful tuning.

Yet BO keeps a role:

• Baselines. To compare fairly against tree ensembles on /model/benchmarks, the baselines must be tuned. BO is the standard way to give them a strong, budgeted tuning.
• Inference settings. A few options remain, such as the number of ensemble members, the amount of context, feature subsampling, preprocessing choices and temperature of the output. These can be tuned with a small BO run per task.
• Post-processing. Thresholds for classification, calibration methods, and blending weights are low-dimensional and well suited to BO.

Designing the synthetic prior

Pre-training on synthetic tables depends on a generator that draws random tables with varied structure. The generator has settings: how many rows and columns, how many are categorical, how deep the causal graphs are, how much noise and missingness. Choosing these settings is a black-box problem, because each candidate needs a pre-training run or a proxy run before its quality shows up on real benchmarks. Bayesian optimization, often with cheap proxy runs and multi-fidelity tricks, is a natural way to search over the generator's knobs.

Pre-training recipes

Learning rates, warm-up lengths, batch composition, the mix of table sizes and masking rates all matter. Each full run is very costly, so teams tune on small models and short runs, then transfer the settings. Multi-fidelity BO fits exactly: cheap small-scale runs inform where to spend the expensive ones. This links to meta-learning and transfer learning ideas, where knowledge from one setting warms up another.

Row and column structure

The architecture of such models treats rows and columns symmetrically, often with attention across both, as in axial attention. Architectural choices, for example how many layers attend over rows versus columns, are discrete and conditional. A tree-based surrogate or a hybrid space handles them better than a plain Gaussian process.

Mixed types and missing values

Preprocessing decisions for mixed-type tables form a small search space: how to encode categories, how to scale numbers, whether to add missing indicators, how to cap outliers. For a specific table, BO can pick the combination that maximizes validation score of the downstream model. Because validation is on a small set of rows, noisy-objective BO with repeated cross-validation is appropriate.

No column names

When column names are missing or untrusted, the model relies on values and cross-column patterns. Any step that depends on guesses about meaning, such as a threshold for deciding that a numeric column is really a category, can be exposed as a parameter and tuned with BO on held-out performance.

Continual learning

For models that keep learning, there is a stability and plasticity trade-off. Learning rate, replay ratio and regularization strength control how much new data changes the model. These are tuned by running short update cycles and measuring both new-task gain and old-task loss, a two-objective problem. Multi-objective BO returns a set of trade-off points rather than one answer.

Imputation

Models that fill missing cells have settings such as masking rate during training, loss weights for numeric versus categorical cells, and the number of refinement passes. See imputation networks and masked cell modeling. Each can be tuned against a held-out set of artificially masked cells, where the true values are known and the error is easy to measure.

Anomaly detection

Detectors have parameters that are hard to set without labels: number of trees, subsample size, contamination, and score thresholds for isolation forest, or the bottleneck size of an autoencoder. With a small labeled or synthetic-injected set, BO can tune them against area under the curve or precision at k.

Sector and semantic classification

Heads that map rows or columns to sectors or semantic classes (semantic sector heads) have thresholds for open-set rejection, class weights, and calibration settings. Those are a few continuous values, a good fit for BO driven by validation metrics that reflect business cost.

Mixture-of-experts

In mixture-of-experts layers, the number of experts, how many are active per input, the load-balancing weight and the capacity factor interact. They are expensive to explore by hand. Short proxy runs plus BO give a way to find workable settings before committing to a large run.

Explaining the tuned result

After tuning, use SHAP or the surrogate's own length scales to see which parameters mattered. This tells you which knobs deserve more budget next time and which can be fixed.

Practical guidance and best practices

Define the problem well

• Write down the objective in one number, or a small set for multi-objective tuning.
• Decide the budget in trials or hours before starting.
• Make the evaluation as deterministic and as cheap as is honest.
• Use a validation scheme that matches deployment, such as time-based splits for time-ordered data.

Set up the search space

• Use log scales for parameters that span orders of magnitude, such as learning rate and regularization.
• Use integers for counts and categories for discrete choices.
• Keep bounds wide enough to contain the optimum but tight enough to avoid nonsense.
• Remove parameters that you know do not matter.
• Mark conditional parameters properly if the library supports it.

Seed the search

• Start with 5 to 10 random or space-filling points, such as a Sobol sequence.
• Add known good settings from experience or from related tasks, which is a warm start.
• Re-use results from earlier runs when the task is similar.

Choose the right tool

• Gaussian process BO for low-dimensional continuous problems.
• Random forest or tree-structured Parzen estimator surrogates for mixed or conditional spaces.
• Multi-fidelity methods when you can read partial results.
• Batch methods when trials run in parallel.

Handle noise

• Fix random seeds where possible, but do not trust a single seed.
• Average over a few repeats for the final candidates.
• Include a noise term in the surrogate.
• Re-evaluate the top few settings with more folds or seeds before declaring a winner.

Measure honestly

• Hold out a final test set that is never used inside the optimization loop. Tuning against a validation set overfits it.
• Report the budget used.
• Compare with random search at equal budget.
• Keep logs of every trial for reproducibility. Experiment trackers such as MLflow help.

Use parallelism wisely

• With a small budget, run trials sequentially or in small batches.
• With a large cluster, combine batch BO with early stopping.
• Avoid launching many near-duplicate points in one batch.

Know when to stop

• Stop when the best value has flattened and expected improvement is tiny.
• Spend leftover budget on repeats of the top settings instead of new exploration.

Common mistakes

• Tuning on the test set. Use a separate final test set.
• Linear scales for log-like parameters. The search wastes most trials in the wrong region.
• Too few initial points. The surrogate starts with a poor picture and may lock onto a false trend.
• Too wide a space. Huge bounds dilute the budget.
• Ignoring noise. The optimizer chases lucky seeds.
• Using Gaussian process BO in very high dimension. Switch to other surrogates, reduce dimension, or use random embeddings.
• No baseline. Without random search for comparison, you cannot tell if BO helped.
• Treating the best trial as certain. Re-evaluate it.
• Forgetting constraints. The best loss may violate a memory or latency limit.
• Overlooking categorical handling. Encoding categories as integers in a smooth kernel invents false order.
• Stopping too early or too late. Plot progress and use a rule.
• Not logging. Lost trial history means lost knowledge for the next project.

Frequently asked questions

What is Bayesian optimization in simple terms?

It is a smart way to search for the best settings of something that is slow or costly to test. It keeps a statistical model of how good each setting is likely to be, including how unsure it is, and uses that model to choose the next setting to try. Each new result sharpens the model.

Why is it called Bayesian?

Because it starts with a prior belief about the objective function and updates that belief with each observation using Bayes rule. The Gaussian process is a prior over functions, and the posterior after seeing data gives the predicted mean and uncertainty that drive the next choice.

When should I prefer Bayesian optimization to random search?

Prefer it when each trial is costly, the number of parameters is modest, and the objective is reasonably smooth. If trials are cheap and you can afford thousands, random search is a strong and simple alternative. In all cases, random search at equal budget is the right baseline to compare against.

What is an acquisition function?

It is the rule that decides which candidate to evaluate next. It uses the surrogate's predicted mean and uncertainty to score every candidate. Expected improvement is the most common choice, and it rewards both a good predicted value and a high chance of a surprise.

How many trials does Bayesian optimization need?

It depends on the problem, but a typical run uses a few dozen to a few hundred trials. Start with about five to ten initial points per few parameters and then let the optimizer run until progress flattens. More dimensions and more noise need more trials.

Can it handle categorical and conditional parameters?

Yes, but not with a plain Gaussian process on integer codes. Use a surrogate that supports such spaces, such as random forests or tree-structured Parzen estimators, or a Gaussian process with kernels designed for categories. Conditional parameters need a search space definition that records which parameters exist only under which choices.

Does it work with noisy objectives?

Yes. The surrogate can include a noise term, and acquisition functions have noisy versions. You should still repeat the evaluation of the best candidates, since a single lucky run may mislead both the optimizer and you.

Do tabular foundation models make hyperparameter tuning obsolete?

They reduce it a lot, because in-context learning adapts to each table without fitting weights. Some tuning remains for inference options, post-processing, and the tuning of baselines for fair comparison. Pre-training itself involves heavy tuning of recipes and synthetic data generators.

How is Bayesian optimization different from gradient descent?

Gradient descent follows the slope of a differentiable function and needs many cheap evaluations. Bayesian optimization needs no slope and aims to use as few evaluations as possible, so it suits expensive black boxes. They are often combined: gradient descent trains the model, and Bayesian optimization tunes the settings around it.

Can I run Bayesian optimization in parallel?

Yes. Batch methods propose several points at once, using ideas such as Thompson sampling or fantasized outcomes for pending trials. Parallel runs finish sooner but can be slightly less efficient per trial, because later points in a batch cannot learn from earlier ones.

Glossary

• Acquisition function: a score that tells the optimizer which point to try next.
• Black-box function: a function that can be evaluated but not inspected or differentiated.
• Expected improvement: the average gain over the best known value if a point is evaluated.
• Exploration: sampling uncertain regions to learn about them.
• Exploitation: sampling regions that already look good.
• Fidelity: the level of cost and accuracy of an evaluation, such as number of epochs.
• Gaussian process: a distribution over functions that gives a mean and a variance at any input.
• Kernel: a function that defines similarity between inputs and the smoothness of the surrogate.
• Length scale: a kernel parameter that sets how far apart inputs must be before outputs differ.
• Regret: the difference between the best possible value and the value achieved.
• Surrogate model: a cheap model of the objective used to guide the search.
• Warm start: initializing a search with results from earlier related runs.

Summary

Bayesian optimization tunes expensive black-box functions by learning a probabilistic surrogate, usually a Gaussian process, and picking each next trial with an acquisition rule such as expected improvement that balances exploration and exploitation. It is sample efficient, handles noise, constraints and batches, and works best in low to moderate dimension. Set up log scales and sensible bounds, seed with a few space-filling points, keep a clean final test set, and always compare with random search at equal budget. For tabular foundation models it tunes baselines and inference options, and it guides expensive choices in pre-training, synthetic data generators, imputation, anomaly detection and expert layers. Related reading includes SHAP for understanding results, Optimization for background, and /model/benchmarks for evaluation.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: