Optuna

Overview

Optuna is an open-source Python framework for hyperparameter optimization. You write a normal function that trains a model and returns a score. Optuna calls that function many times, each time with different settings, and learns which settings tend to give better scores. It also stops unpromising runs early, a feature called pruning, so limited compute goes further.

Optuna is popular because it is simple to start and flexible to extend. The search space is defined inside ordinary Python code, so loops, conditions and nested choices are easy. It works with almost any library, including scikit-learn, XGBoost, LightGBM, CatBoost and PyTorch. This article explains the main ideas in plain language, shows a full example, compares Optuna with Ray Tune and other methods, and covers how tuning relates to modern table models described on the model research page.

What it is

Optuna is a library for finding good hyperparameters. A hyperparameter is a setting you choose before training, such as the learning rate, the depth of a tree or the number of layers. Unlike the weights a model learns, hyperparameters are not learned by gradient descent. They must be searched.

Think of baking bread. The recipe has fixed ingredients, but you choose oven temperature, baking time and how long the dough rests. Different choices give different loaves. Tuning is the process of trying combinations and keeping the best loaf. Optuna is a smart assistant that remembers which combinations worked and suggests better ones, instead of trying every combination blindly.

The core objects

• Study: the whole optimization job. It holds the history of all attempts.
• Trial: one attempt with one set of hyperparameters.
• Objective: your function that takes a trial, trains a model and returns a number.
• Sampler: the strategy that proposes the next set of values.
• Pruner: the strategy that stops a bad trial before it finishes.
• Storage: where the study is saved, in memory or in a database.

Define-by-run

Optuna uses a define-by-run API. The search space is created while the objective function runs. If the first choice is "model type", the next choices can depend on it. For example, a neural network branch can ask for layer counts, while a tree branch asks for depth. This is natural in Python and hard to express in a fixed configuration file.

What Optuna is not

• It is not a model library. It does not train models by itself.
• It is not a guarantee of a global optimum. It finds good settings within a budget.
• It is not a replacement for good features and clean data. Tuning rarely rescues a poor setup.

Where it came from

Optuna was introduced by researchers and engineers in Japan and presented in a 2019 paper titled "Optuna: A Next-generation Hyperparameter Optimization Framework". It was released as open-source software and grew quickly through community adoption.

The field behind it is older. Early tuning used grid search, which tries every combination on a grid, and random search, which picks random combinations. A well-known result in the early 2010s showed that random search often beats grid search because only a few hyperparameters matter much. Then came sequential model-based methods, also called Bayesian optimization, which build a model of the score surface and use it to choose the next trial. The Tree-structured Parzen Estimator, or TPE, is one such method and became Optuna's default sampler.

Another strand is early stopping of bad runs. Methods such as successive halving and Hyperband give small budgets to many configurations and promote only the best to larger budgets. Optuna includes pruners that follow these ideas. The framework's design made all of this available behind one small API.

How it works

A tuning job in Optuna runs a loop. Each pass of the loop is a trial.

Step 1: create a study

You create a study and say whether to minimise or maximise the score. A study with a storage backend can be paused, resumed and shared among workers.

Step 2: the sampler proposes values

When the objective function asks for a value, for example trial.suggest_float("lr", 1e-4, 1e-1, log=True), the sampler decides what to return. At first, with no history, it picks values at random. As results arrive, it becomes smarter.

Step 3: the objective trains and scores

Your function builds a model with the suggested values, trains it, evaluates it on validation data and returns the score. This part is entirely yours.

Step 4: the result is recorded

Optuna stores the values and the score. Now the history has one more data point.

Step 5: the sampler learns

The next time the sampler is asked, it uses all past data. With TPE, it splits past trials into a good group and a bad group by score. It builds a density for each group and prefers values that are more likely in the good group than the bad group.

An everyday analogy: you are looking for a good restaurant in a new city. After a few meals, you notice the good ones are near the river and the bad ones are near the station. You stop sampling near the station and spend more meals near the river, while still trying a few odd places in case you are wrong.

Step 6: pruning

If the objective reports intermediate scores, such as validation loss after each epoch, a pruner compares the current trial with earlier ones. If it is clearly worse, Optuna stops it. This saves time and lets you try more configurations in the same budget.

Think of an audition. If a singer is clearly off key in the first bars, the judges thank them and move on, instead of listening to the whole song.

Step 7: repeat

The loop continues until you reach a number of trials, a time limit or a stop condition. Then you read the best trial and use its values for a final training run.

Samplers

Exact sampler names and availability depend on the installed version, so check the documentation.

Pruners

Key concepts and terms

A worked example

The example tunes a gradient boosted model on a table. File names are placeholders, and the code is illustrative.

import optuna
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import HistGradientBoostingClassifier

df = pd.read_csv("customers.csv")
y = df["churned"]
X = df.drop(columns=["churned"])

def objective(trial):
    params = {
        "learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),
        "max_depth": trial.suggest_int("max_depth", 2, 10),
        "max_leaf_nodes": trial.suggest_int("max_leaf_nodes", 8, 128),
        "l2_regularization": trial.suggest_float("l2_regularization", 1e-6, 10.0, log=True),
        "min_samples_leaf": trial.suggest_int("min_samples_leaf", 5, 100),
    }
    model = HistGradientBoostingClassifier(**params, random_state=0)
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
    return scores.mean()

study = optuna.create_study(
    direction="maximize",
    sampler=optuna.samplers.TPESampler(seed=0),
    study_name="churn-tuning",
    storage="sqlite:///tuning.db",
    load_if_exists=True,
)
study.optimize(objective, n_trials=60, timeout=1800)

print("best score:", study.best_value)
print("best params:", study.best_params)

What each line does

• suggest_float with log=True samples a rate across orders of magnitude, so 0.001 and 0.01 are as likely as 0.01 and 0.1.
• suggest_int samples whole numbers within a range.
• cross_val_score returns five scores. Their mean is the number Optuna maximises.
• storage saves progress to a small database file, so you can stop and resume.
• n_trials and timeout give two stop rules. The first one reached ends the run.

A pruning example

Pruning needs intermediate reports. Here is a sketch for an iterative model.

import optuna
import numpy as np

def objective(trial):
    lr = trial.suggest_float("lr", 1e-4, 1e-1, log=True)
    width = trial.suggest_categorical("width", [32, 64, 128, 256])
    model = build_model(width=width)          # placeholder function
    for epoch in range(30):
        train_one_epoch(model, lr)            # placeholder function
        val_loss = evaluate(model)            # placeholder function
        trial.report(val_loss, step=epoch)
        if trial.should_prune():
            raise optuna.TrialPruned()
    return val_loss

study = optuna.create_study(
    direction="minimize",
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5, n_warmup_steps=5),
)
study.optimize(objective, n_trials=100)

The pruner waits for five startup trials and five warm-up epochs, then compares the current loss with the median of earlier trials at the same epoch. A trial that is worse is stopped.

A numeric walk-through of how pruning saves time

Suppose each full trial takes 30 minutes and you have 10 hours. Without pruning, you can run 20 trials. Suppose pruning stops about 70 percent of trials after just 5 minutes and lets 30 percent run in full.

• Average time per trial = 0.7 x 5 + 0.3 x 30 = 3.5 + 9 = 12.5 minutes.
• Trials in 10 hours = 600 / 12.5 = 48.

You explore more than twice as many configurations in the same time. The saving depends on how early bad trials reveal themselves, so results vary by problem.

Reading the study afterwards

import optuna.visualization as vis

vis.plot_optimization_history(study).show()
vis.plot_param_importances(study).show()
vis.plot_parallel_coordinate(study).show()

df = study.trials_dataframe()
print(df.sort_values("value", ascending=False).head(5))

The history plot shows whether scores still improve or have flattened. The importance plot estimates which hyperparameters matter. If one parameter dominates, you can narrow its range in a second round and fix the rest.

Strengths and limitations

Strengths

• Simple API: a function and a few lines to start.
• Define-by-run search spaces allow conditional and nested choices.
• Smart default sampler that usually beats random search on the same budget.
• Built-in pruning to save compute.
• Works with any framework since it only needs a returned number.
• Persistent storage supports resume and distributed workers.
• Multi-objective support for trade-offs such as accuracy versus latency.
• Visualisation helpers for history, importance and relationships.
• Lightweight: runs on a laptop, scales to many workers with a shared database.

Limitations

• Sampling remains stochastic. Repeat with other seeds to check that conclusions hold.
• Noisy objectives can mislead the sampler. If the score changes a lot between identical runs, tuning chases noise.
• Very high-dimensional or purely categorical spaces are harder for model-based samplers.
• Overfitting to the validation set is possible after many trials. The best score is optimistic.
• Distributed tuning needs a shared storage backend and some care.
• It is not designed to manage a cluster, schedule GPUs or handle fault tolerance on its own. For that, a distributed tuning system may fit better.

The winner's curse

After 500 trials, the best validation score is partly luck. It is the maximum of many noisy numbers, so it is biased upwards. The honest estimate comes from a separate test set that was never used in the search. A simple rule: tune with cross-validation or a validation set, then report a single score on untouched data.

How it compares with alternatives

Optuna and Ray Tune together

The two are not rivals in every case. Ray Tune can use Optuna as a search algorithm, while Ray handles scheduling, resources and fault tolerance. A common pattern is to start with Optuna alone on a laptop or a single server, and move to Ray Tune when trials need many GPUs or machines.

Optuna versus random search

Random search is the honest baseline. For a fair comparison, give both the same number of trials and the same seeds. In many problems model-based search finds a good region sooner, but with a tiny budget or a very noisy objective the difference can be small.

Role in tabular foundation models

Tabular foundation models are trained on many tables so they can predict on a new table directly. They work from the raw table, with rows, mixed column types, missing cells and sometimes no useful column names. Hyperparameter tuning touches this area in several ways, though not always where people expect.

Tuning the classical competitors

Benchmarks of table models, such as those summarised on the benchmarks page, compare pre-trained models against tuned gradient boosting. Tuning is a big part of that fairness. If the boosted tree baseline gets a poor default setting, a pre-trained model looks better than it is. If it gets a strong search, such as a few hundred Optuna trials, the comparison is stricter. The honest report states the search budget, search space and validation protocol for every method.

Less tuning for in-context models

Models like TabPFN and TabICL are pre-trained and predict in context. They need little or no per-dataset hyperparameter search. This is one of their selling points: the expensive search that Optuna automates is replaced by a single forward pass. Still, there are light knobs, such as the number of ensemble members, the amount of context or preprocessing options. Optuna can tune these with a small budget when the extra accuracy is worth it.

Tuning during pre-training

Training a tabular foundation model itself involves many hyperparameters: learning rate schedule, model width, number of attention layers, mix of synthetic data priors, masking rate for masked-cell objectives and more. Tuning these at full scale is too expensive, so teams tune on small versions of the model and transfer the findings. Pruning helps here, because bad configurations show themselves early in the loss curve. Details of such designs are discussed on the architecture page.

Synthetic data priors as hyperparameters

Some table models are pre-trained on generated tables. The settings of the generator, such as how many columns, how many categories, how noisy, how much missingness, behave like hyperparameters of the data. Searching over them is a form of optimization where each trial means building a data mix and measuring downstream quality. Multi-objective search can balance accuracy against robustness or speed.

Imputation and missing values

When a model is used for imputation, the masking rate and the pattern of missingness in training are important settings. Optuna can tune them against a validation metric such as mean absolute error on hidden cells. The key is to keep the validation masks fixed across trials so that scores are comparable.

Anomaly detection

Detectors such as Isolation Forest have settings like the number of trees and the expected contamination. Because labels for anomalies are rare, tuning is delicate. A small labelled set or synthetic injected anomalies can serve as the objective, but one should keep in mind that tuning to injected anomalies may not match real ones. See also the anomaly detection page.

Continual learning

When a model keeps learning from new data, hyperparameters such as the learning rate and the size of any replay memory control the trade-off between learning new things and forgetting old ones. A multi-objective Optuna study can optimise both new-task accuracy and old-task retention, then show the Pareto front so a person can choose a balance.

Sector and semantic classification

Heads that classify what a table or column represents are small models on top of a larger encoder. Their hyperparameters, such as regularisation and class weights, can be tuned cheaply. Class imbalance is common, because some sectors are rare, so the objective should use a balanced metric rather than plain accuracy.

Feeding the tuner with the right validation

For tabular work, the validation design matters as much as the sampler. Use time-based splits for temporal data, group-based splits when rows share an entity, and stratified splits for rare classes. A tuner will exploit any leak in the validation scheme, and will do so very efficiently.

Practical guidance and best practices

Start small and cheap

Run ten to twenty trials with a coarse search space first. Look at the history and the importance plot. Then narrow the space and run a longer study.

Use sensible ranges

Rates and regularisation strengths should be sampled on a log scale. Integers such as depth should have realistic bounds. Overly wide ranges waste trials.

Fix the seed for the sampler and the model

This makes studies repeatable. Do not confuse a lucky seed with a good setting. Re-evaluate the best settings with several model seeds.

Choose the right objective

Pick the metric that reflects your real goal. If you care about rare positives, use area under the precision-recall curve or a cost-weighted metric, not accuracy.

Use pruning when training is iterative

For boosted trees with early stopping or neural networks with epochs, report intermediate values and enable a pruner. For single-shot models, pruning does not apply.

Cross-validate when data is small

On small tables a single validation split is noisy. Use cross-validation inside the objective, but keep an eye on cost.

Keep a final test set

Never use it for tuning. Evaluate once at the end.

Store studies in a database

Use a database URL such as sqlite:///tuning.db for a single machine, or a server database for several workers. Use fake hosts in shared examples, such as postgresql://user:password@example.internal/optuna, and keep real credentials in a secrets store, never in code.

Run in parallel carefully

Several workers can share one study. Make sure each worker uses different CPU or GPU resources and that the model training code is thread-safe or process-safe.

Warm start with known good values

You can enqueue a trial with known settings, for example library defaults, so the study starts with a decent reference.

study.enqueue_trial({"learning_rate": 0.1, "max_depth": 6,
                     "max_leaf_nodes": 31, "l2_regularization": 1.0,
                     "min_samples_leaf": 20})

Log and track

Combine Optuna with a tracking tool such as MLflow so each trial's settings and metrics are saved with the model artifact.

A tuning checklist

1. Define the objective metric and validation scheme.
2. Define a small, sensible search space.
3. Run a short exploratory study.
4. Check history and importance.
5. Narrow the space and run a longer study.
6. Re-evaluate the top few configurations with several seeds.
7. Train the final model on all training data.
8. Report one score on the untouched test set.
9. Save the study and settings.

Common mistakes

• Tuning on the test set, directly or by repeated peeking.
• Reporting the best validation score as the expected performance.
• Searching ranges that are too wide or on the wrong scale.
• Using accuracy on imbalanced data.
• Ignoring randomness in the objective and chasing noise.
• Forgetting to set a timeout and letting a study run for days.
• Running parallel workers that fight over the same GPU.
• Using random splits on time-ordered data.
• Not pruning iterative models and wasting budget.
• Tuning many parameters at once on a tiny budget.
• Saving results only in memory and losing them when the process stops.
• Comparing tuned and untuned models and calling the difference a model effect.
• Putting credentials in connection strings inside code.

The noisy objective trap

Imagine an objective whose score changes by plus or minus 0.01 between identical runs. A sampler may see a configuration score 0.01 higher by luck and then concentrate trials near it. After many trials, the search reports a "best" that is no better than average. Remedies include averaging over folds or seeds, using larger validation sets and checking the top configurations again at the end.

Multi-objective tuning

Sometimes one score is not enough. You may want high accuracy and low prediction time at once.

def objective(trial):
    depth = trial.suggest_int("depth", 2, 12)
    trees = trial.suggest_int("trees", 50, 800)
    model = build(depth, trees)             # placeholder
    acc = evaluate_accuracy(model)          # placeholder
    latency = measure_latency_ms(model)     # placeholder
    return acc, latency

study = optuna.create_study(directions=["maximize", "minimize"])
study.optimize(objective, n_trials=80)

for t in study.best_trials:
    print(t.values, t.params)

The result is not one best trial but a set of best trade-offs, the Pareto front. A deep model may be slightly more accurate but much slower. A person then picks the point that suits the use case, such as a model that must answer within a fixed time.

Search space design in more detail

Categorical choices

Use suggest_categorical for choices such as the booster type, the activation function or the encoding method. Keep the number of options small.

Conditional parameters

Only ask for a parameter when it applies. For example, ask for max_depth only when the model is a tree, and n_layers only when it is a network. This keeps the space compact and the sampler efficient.

Log versus linear scale

Use a log scale when the useful values span several orders of magnitude, such as learning rates and penalties. Use a linear scale for things like the fraction of columns sampled per tree.

Step sizes

Integers can use a step, for example batch sizes in multiples of 32. This avoids wasting trials on near-identical values.

Constraints

Some combinations are invalid or too costly. Return a bad score or raise a prune signal early for these. Do not let the trial run for an hour before failing.

Tuning for different tabular model families

Reproducibility and operations

• Set the sampler seed and the model seed separately.
• Save the study in a database and keep a copy of the code that defined the objective.
• Record library versions.
• Keep the data snapshot id with the study name.
• Name studies with a clear pattern such as project, model and date.
• Archive finished studies so that later teams can reuse the history.

Resuming and extending

Because studies are stored, you can run the optimizer for another batch of trials later. The sampler picks up from the history. If you change the search space between sessions, be careful: old trials may contain values outside the new ranges. Optuna handles this reasonably, but results are easier to interpret if the space is stable.

Frequently asked questions

What is Optuna used for?

Optuna is used to find good hyperparameters for machine learning models. You provide a function that trains a model and returns a score, and Optuna searches for settings that improve it. It is common for gradient boosted trees, neural networks and any other model with tunable settings.

What is a study and what is a trial in Optuna?

A study is the whole optimization session, including its history and settings, while a trial is one attempt with one set of hyperparameter values. A study contains many trials, and the best trial is the one with the best objective value.

What is the default sampler in Optuna?

The default sampler for single-objective problems is the Tree-structured Parzen Estimator. It separates past trials into good and bad groups and proposes new values that look more like the good group. This usually beats random search on the same number of trials.

How does pruning work?

Pruning stops trials that look unpromising before they finish. Your code reports intermediate scores, such as validation loss per epoch, and a pruner compares them with earlier trials. If the current trial is clearly behind, it is stopped, which saves time for more promising ones.

Is Optuna better than grid search?

In most practical cases yes, because grid search wastes trials on unimportant dimensions and scales badly as parameters grow. Optuna uses past results to focus on good regions and can stop weak trials early. Grid search can still make sense for a tiny space with only a few options.

Can I run Optuna on many machines?

Yes. Several workers can share one study if they point to the same storage database. Each worker pulls new trials and writes results back. For large clusters with scheduling and fault tolerance, many teams pair Optuna with a distributed system such as Ray Tune.

Does Optuna work with any machine learning library?

Yes, because it only needs your objective function to return a number. It has integrations for several libraries that report intermediate values or handle early stopping, but you can use it with plain code from any framework.

How many trials do I need?

There is no fixed number. A common starting point is thirty to one hundred trials for a handful of parameters, then more if the history is still improving. Watch the optimization history and stop when gains flatten, and always confirm on data not used in the search.

Do tabular foundation models need hyperparameter tuning?

Pre-trained table models that predict in context need far less tuning than classical models, since they do not train per dataset. They still have a few options, such as ensemble size and preprocessing, which a small Optuna study can tune. Tuning also matters on the baseline side, so that comparisons are fair.

How do I avoid overfitting the validation set during tuning?

Use cross-validation or a large enough validation set, limit the number of trials relative to data size, and keep a separate test set that is evaluated only once. Re-checking the top few configurations with new seeds also helps reveal luck.

Glossary

• Bayesian optimization: tuning that builds a model of the score surface to choose the next trial.
• Budget: the amount of compute or the number of trials allowed.
• Define-by-run: building the search space while the code executes.
• Early stopping: ending training when validation scores stop improving.
• Hyperband: a pruning method that allocates budgets across many configurations.
• Hyperparameter: a setting chosen before training.
• Objective function: the function that returns the score to optimise.
• Pareto front: the set of trade-off optimal solutions in multi-objective search.
• Pruner: a rule that stops weak trials early.
• Sampler: a rule that proposes hyperparameter values.
• Search space: the allowed values for each hyperparameter.
• Study: a full optimization session.
• TPE: a sampler that models good and bad regions with densities.
• Trial: one evaluation with one set of values.
• Winner's curse: the upward bias of the best score among many noisy ones.

Summary

Optuna turns hyperparameter tuning into a simple loop: write an objective, define the search space inside it, and let a model-based sampler and a pruner spend your budget wisely. Its define-by-run style, persistent storage and wide compatibility make it a standard choice for tuning boosted trees, neural networks and many other models.

For tabular work, tuning is both a tool and a fairness requirement. It sharpens classical baselines, tunes light options of pre-trained table models and helps design the foundation models themselves. Guard against noise, leakage and the winner's curse, keep a clean test set, and report the budget you used. For related reading, see Ray Tune, Bayesian optimization, PyCaret, XGBoost and the model research page.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: