Ray Tune

Overview

Ray Tune is a Python library for distributed hyperparameter tuning. It runs many training trials at the same time across the cores, GPUs and machines of a cluster, and it stops weak trials early so that compute is not wasted. It belongs to the Ray ecosystem, a general framework for scaling Python code, and it works with popular training libraries such as PyTorch, TensorFlow, XGBoost and LightGBM.

Ray Tune is a good fit when a single trial is expensive, when you have many trials to run, or when you have more than one machine. For small jobs on a laptop, a lighter tool such as Optuna may be simpler. The two are often used together: Optuna supplies the search strategy while Ray Tune handles scheduling, resources and fault tolerance. This article explains how Ray Tune works in everyday terms, shows a full example, compares it with alternatives and describes how large-scale tuning connects to the table models described on the model research page.

What it is

Ray Tune is a tuning layer built on Ray. Ray provides tasks and actors, which are ways to run Python functions and objects on many processes or machines. Tune uses these to run trials in parallel. Each trial is one training run with one set of hyperparameters.

A good picture is a kitchen with many cooks. Optuna is like a head chef who decides which recipes to test next. Ray Tune is like the kitchen manager who assigns stoves and cooks, watches the clock, sends away a cook whose dish is clearly burning and replaces a cook who goes home sick. The manager does not need to know the recipes. The manager needs to know the resources and the rules.

What Tune provides

• A way to describe the search space.
• A choice of search algorithms for proposing configurations.
• Schedulers that stop or modify trials early.
• Resource management, such as how many CPUs or GPUs each trial gets.
• Fault tolerance, such as restarting a trial after a machine fails.
• Checkpointing, so trials can pause and resume.
• Result reporting, logging and experiment analysis.

What Tune does not provide

• It is not a model library.
• It does not make a single trial faster. It runs more trials at once.
• It does not guarantee a better optimum than a smaller search tool. It makes large searches practical.

Where it came from

Ray Tune grew out of research on distributed systems and machine learning at a university lab, in the same period that produced Ray itself in the late 2010s. A paper describing the Tune project, titled "Tune: A Research Platform for Distributed Model Selection and Training", appeared in 2018. The goal was to give researchers one place to run many model selection algorithms at scale without rewriting training code.

The ideas it packages come from earlier research on tuning:

• Random search and grid search, the simple baselines.
• Bayesian optimization, which builds a model of scores to choose new trials.
• Successive halving and Hyperband, which give small budgets to many configurations and promote the best.
• ASHA, an asynchronous variant of successive halving built for parallel systems.
• Population based training, which evolves a set of models while training them.

Since then Tune has added a long list of integrations with search libraries and training frameworks. Names and interfaces change between releases, so check the documentation for your installed version before copying older examples.

How it works

Step 1: describe the trainable

A trainable is the code that trains one model. In current versions, it can be a plain function that receives a config dictionary, trains, and reports metrics. It may also save checkpoints so that Tune can pause, resume or clone it.

Step 2: describe the search space

You describe each hyperparameter with helpers such as uniform, log-uniform, choice and integer ranges. You can also use grid values. Tune samples from this space to create the configs for trials.

Step 3: choose a search algorithm

The search algorithm proposes configs. The default is random sampling. You can plug in model-based searchers, such as Optuna's TPE, HyperOpt-style searchers, Bayesian optimization or others, through integrations.

Step 4: choose a scheduler

The scheduler decides, while trials run, which to continue, pause or stop. This is where Tune saves most of its compute. Examples:

Step 5: allocate resources

You tell Tune how much each trial needs, for example two CPUs and half a GPU. Ray then packs trials onto the cluster. If you have eight GPUs and each trial needs one, eight trials run at once. If each needs half a GPU, sixteen can run.

Step 6: run and report

Each trial runs in its own process. Inside the training loop, it reports metrics such as validation loss. Tune collects these, feeds them to the search algorithm and the scheduler, and writes logs.

Step 7: fault tolerance and checkpoints

If a worker dies, Tune can restart the trial from its last checkpoint. If a scheduler pauses a trial to make room for a better one, the trial can resume later from its checkpoint. This makes long jobs on shared or cheap machines practical.

Step 8: analyse results

At the end, you get a result grid with every trial's config and metrics. You can sort, filter and export it to a data frame, then pick the best trial and its checkpoint.

An analogy for ASHA

Picture a talent show with hundreds of acts. In round one, each act performs for one minute. The best third go on to a five minute round. The best third of those go on to a twenty-five minute final. You see many acts briefly and only invest time in the strong ones. ASHA does this without waiting for all acts in a round to finish before moving the first good act forward, which keeps stages busy.

An analogy for population based training

Imagine a team of runners training together. Every week, the slowest runner copies the training plan of one of the fastest and then tweaks it a little. Over time the whole team improves, and the plan itself changes as training goes on. Population based training does the same with models: weak trials copy weights and settings from strong ones, then perturb the settings.

Key concepts and terms

A worked example

The example below tunes a small neural network on a table, using a function trainable. It is a sketch with placeholders, not a drop-in script. Imports and call names can differ between versions, so confirm them in the current documentation.

import ray
from ray import tune
from ray.tune.schedulers import ASHAScheduler

def train_fn(config):
    model = build_model(
        width=config["width"],
        depth=config["depth"],
        dropout=config["dropout"],
    )                                            # placeholder
    optimizer = make_optimizer(
        model, lr=config["lr"], weight_decay=config["wd"]
    )                                            # placeholder
    for epoch in range(40):
        train_one_epoch(model, optimizer)        # placeholder
        val_loss, val_auc = evaluate(model)      # placeholder
        tune.report({"val_loss": val_loss, "val_auc": val_auc})

search_space = {
    "width": tune.choice([64, 128, 256, 512]),
    "depth": tune.randint(1, 6),
    "dropout": tune.uniform(0.0, 0.5),
    "lr": tune.loguniform(1e-4, 1e-2),
    "wd": tune.loguniform(1e-6, 1e-2),
}

scheduler = ASHAScheduler(
    max_t=40,
    grace_period=5,
    reduction_factor=3,
)

ray.init(address="auto")        # connect to a running cluster

tuner = tune.Tuner(
    tune.with_resources(train_fn, {"cpu": 2, "gpu": 0.5}),
    param_space=search_space,
    tune_config=tune.TuneConfig(
        metric="val_auc",
        mode="max",
        scheduler=scheduler,
        num_samples=100,
    ),
)

results = tuner.fit()
best = results.get_best_result(metric="val_auc", mode="max")
print(best.config)
print(best.metrics["val_auc"])

What the settings mean

• num_samples=100 asks for one hundred trials.
• grace_period=5 gives every trial at least five epochs before ASHA may stop it.
• reduction_factor=3 means about one third of trials survive each rung.
• max_t=40 caps training at forty epochs.
• gpu: 0.5 lets two trials share one GPU, which suits small models.

A numeric walk-through of ASHA

Take one hundred trials, grace period of five, reduction factor of three and max of forty-five epochs.

• Rung 1 at epoch 5: all 100 trials run. About 33 continue.
• Rung 2 at epoch 15: those 33 run to epoch 15. About 11 continue.
• Rung 3 at epoch 45: those 11 run to the end.

Total epochs used, roughly: 100 x 5 + 33 x 10 + 11 x 30 = 500 + 330 + 330 = 1,160 epochs. Running all 100 trials to 45 epochs would cost 4,500 epochs. The schedule uses about a quarter of that and still gives the best trials their full training.

The caveat is that early scores must predict final ranking reasonably well. If a configuration starts slowly but finishes strongly, such as a very small learning rate, ASHA may cut it too soon. Increase the grace period when you suspect this.

Using Optuna as the searcher

from ray.tune.search.optuna import OptunaSearch

tune_config = tune.TuneConfig(
    metric="val_auc",
    mode="max",
    search_alg=OptunaSearch(),
    scheduler=ASHAScheduler(grace_period=5, reduction_factor=3),
    num_samples=200,
)

Here the proposal step uses a model-based sampler while the scheduling step uses ASHA. This is a common combination. The import path above may differ by version.

Starting a small cluster

# cluster.yaml (illustrative)
cluster_name: tune-demo
max_workers: 4
provider:
  type: aws
  region: us-east-1
head_node_type: head
available_node_types:
  head:
    node_config:
      InstanceType: m5.xlarge
  gpu_worker:
    min_workers: 0
    max_workers: 4
    node_config:
      InstanceType: g4dn.xlarge

Cluster launchers, Kubernetes operators and managed services all exist. The file above is only a sketch to show the shape. Use your own platform's documented method and keep credentials in a secrets manager.

Strengths and limitations

Strengths

• Scales from a laptop to a large cluster with the same code.
• Strong schedulers, including ASHA and population based training.
• Works with many search libraries and training frameworks.
• Fractional GPU support improves hardware use for small models.
• Fault tolerance and checkpointing support long jobs on spot or shared machines.
• Good logging and result analysis.
• Can tune not only learning settings but also data pipeline options and model architectures.

Limitations

• More moving parts than a single-process tuner. A cluster must be set up and monitored.
• Overhead for tiny, quick trials. Starting processes costs time that can exceed the trial itself.
• Early stopping assumes early performance predicts final performance, which is not always true.
• Parallel trials can run in different orders, so exact repeatability is harder.
• Model-based searchers learn less when many trials run at once with little feedback.
• API changes between versions mean older tutorials can break.
• Costs can grow quickly. A large search on many GPUs is expensive if unchecked.

Parallelism and model-based search

A model-based searcher learns from finished trials. If you launch sixty-four trials at once, the first sixty-four are chosen with almost no information, nearly at random. The more parallel you go, the less each choice benefits from history. This is a real trade-off between wall-clock time and search efficiency. A common compromise is to run in waves or to limit concurrency for the model-based searcher.

How it compares with alternatives

Ray Tune versus Optuna

Think of it as a division of labour. Optuna focuses on how to choose settings, with a clean API and pruning. Ray Tune focuses on where and when trials run, with a rich scheduler set and cluster tools. If your trials are cheap and run on one machine, Optuna alone may be enough. If your trials are heavy, need GPUs, or must survive failures, Ray Tune adds real value. Using Optuna inside Ray Tune gives both.

Ray Tune versus managed cloud tuning

Managed services remove cluster management but tie you to a vendor's interface. Ray Tune is portable across environments, at the cost of operating Ray yourself or choosing a Ray hosting option.

Role in tabular foundation models

Tabular foundation models are neural networks pre-trained on very many tables so they can handle a new table with little or no extra training. They learn to read rows and columns of mixed types, to cope with missing values and to work even when column names are absent. Building such models is a large-scale training problem, and that is where distributed tuning fits.

Tuning the pre-training recipe

Pre-training a table model involves a long list of choices: model width and depth, how attention is split across rows and columns, learning rate schedule, batch composition, masking rate for masked-cell objectives, and the mix of generated or real tables. Each full run is costly. Teams therefore tune on scaled-down versions, running many short trials in parallel. Ray Tune style scheduling suits this well: ASHA can cut poor recipes after a short training budget, and population based training can adapt learning rates as a run progresses. The resulting settings are then carried to the large run. For more on the structure of such models, see the architecture page.

Tuning the data side

For models pre-trained on synthetic tables, the generator has settings: number of rows and columns, share of categorical columns, noise level, rate and pattern of missing values, and complexity of the hidden functions that link columns. These behave like hyperparameters of the data. Each trial generates a data mix, trains a small model and measures a downstream score on held-out real tables. Parallel tuning makes such studies feasible. The study design needs care to avoid tuning to a benchmark that is later used as evidence. See the benchmarks page for how comparisons are framed.

Benchmarks need fair baselines

When pre-trained table models are compared with gradient boosting, LightGBM or CatBoost, the baselines should receive a strong tuning budget. Running hundreds of trials for each of dozens of datasets is a perfect job for a cluster scheduler. It lets benchmark authors state a clear protocol, such as a fixed number of trials and a fixed time budget per dataset, and apply it evenly.

In-context models need less per-dataset tuning

Models such as TabPFN and TabICL predict in context, so the heavy per-dataset search is replaced by one forward pass. A tuning cluster may still help with light settings such as ensemble size, context subsampling and preprocessing, but the demand is much smaller. In practice, the cluster time shifts from per-dataset tuning to pre-training and evaluation.

Continual learning

Continual learning asks a model to absorb new data without forgetting earlier knowledge. Settings such as replay buffer size, regularisation strength and learning rate decide the balance. These interact in complex ways, and the right values depend on the data stream. Population based training is a natural fit because it adapts settings over time, and multi-objective tuning can track both new-task accuracy and retention of old tasks.

Imputation, anomaly detection and semantic heads

Specialised heads on top of a shared table encoder, such as those for imputation, anomaly detection or classifying the sector or meaning of a table, each have their own settings. Because heads are small, trials are cheap, and Tune can sweep through many combinations while the encoder stays frozen and shared. Careful validation is crucial. For imputation, use fixed masks for hidden cells. For anomaly detection, use a mix of real and injected anomalies and report both. For semantic classification, use balanced metrics because some sectors are rarer than others.

Data loading and the table pipeline

Tabular training is often limited by data loading, not by compute. Tune lets you treat pipeline settings such as batch size, number of reader processes and column sampling as part of the search. The best throughput setting is often not obvious and differs between machines.

Governance for tuning experiments

Large searches generate many artifacts. Store configs, metrics, seeds, code versions and data snapshot identifiers with each trial. Without this record, results from a month ago cannot be reproduced. A tracking tool such as MLflow pairs well with Tune for this purpose.

Practical guidance and best practices

Start on one machine

Debug your trainable with a single trial and a tiny budget. Then scale to many trials on one machine. Only then move to a cluster. Most bugs are found cheaply this way.

Make the trainable resumable

Save and load checkpoints reliably, including the optimizer state and the epoch. Test a resume by stopping a trial and restarting it.

Report metrics often

Schedulers need frequent reports to make good decisions. Report at least once per epoch, or more often for long epochs.

Choose the grace period carefully

Give every trial enough budget to show its quality. Too short and good slow starters are lost. Too long and you save little compute.

Match resources to the model

Do not give a small model a whole GPU if half is enough. Do not oversubscribe so far that trials run out of memory.

Limit concurrency for model-based searchers

If a searcher needs history, cap the number of simultaneous trials, at least at the start.

Control costs

Set a total time limit and a trial limit. Use spot or preemptible machines only with solid checkpointing. Monitor spending daily for long runs.

Seed for repeatability where possible

Set seeds in your trainable. Remember that parallel execution can still change the order of results.

Keep a clean evaluation

As with all tuning, keep a test set that the search never sees. Report its score once.

Secure the cluster

Do not expose the dashboard or the head node to the open internet. Place the cluster in a private network, restrict access with network rules, and keep secrets in a manager. Use placeholder hosts such as head.example.internal in documentation and config examples.

A scaling plan

1. One trial, tiny data, one CPU.
2. Ten trials, small data, one machine.
3. One hundred trials with a scheduler on one machine.
4. Same job on a small cluster with checkpoints.
5. Full job with cost limits and monitoring.
6. Re-evaluate top configurations with several seeds.

Common mistakes

• Going to a cluster before the trainable works on one machine.
• Forgetting to checkpoint and losing hours after a failure.
• Reporting metrics only once at the end, which disables early stopping.
• Using a grace period so short that slow starters are always cut.
• Launching too many parallel trials with a model-based searcher and getting near-random search.
• Giving every trial a full GPU when fractions would do.
• Tuning with a leaky validation split and trusting the result.
• Reporting the best validation score as the expected performance.
• Ignoring costs and leaving a large cluster running after the job ends.
• Copying old tutorials without checking the current API.
• Letting trials write to the same output path and overwrite each other.
• Exposing the cluster dashboard publicly.

The slow starter problem

Suppose a configuration with a very low learning rate improves slowly but reaches the best final accuracy. At epoch five it ranks near the bottom, so ASHA stops it. You never learn that it would have won. If you suspect this pattern, raise the grace period, use a learning rate warm-up that makes early scores more comparable, or run a second study focused on slower settings with a different scheduler.

Search space design for Ray Tune

Common sampling helpers

Mixing grid and random

You can grid over a few key options and sample the rest at random. For example, grid over three model types while sampling learning rate and dropout. Each grid value then gets its own random draws, which gives a fair comparison across types.

Conditional spaces

Use a custom function or a searcher that supports conditional spaces when a parameter only applies to one branch. This avoids wasting trials on settings that do nothing.

Choosing a scheduler in practice

Signs a scheduler is hurting

• The best final trials were often paused or cut at early rungs in earlier studies.
• Rankings at early epochs do not match rankings at the end.
• Learning rate warm-up dominates early behaviour.

In these cases, lengthen the grace period or switch to FIFO for a pilot study.

Monitoring a running experiment

Watch these signals while a study runs.

• How many trials are running, pending, paused and finished.
• Whether GPUs are actually busy. Low use often means data loading is the bottleneck.
• Whether memory is close to its limit.
• Whether the best score keeps improving.
• Whether errors repeat in many trials, which points to a bug rather than bad luck.

Set an alert for a trial that fails the same way many times. Stopping early on a bug saves a lot of money.

A tuning journal template

Keep a short record for each study. A shared text file works.

Study name:      table-encoder-sweep-01
Goal:            lower validation loss on held-out tables
Search space:    width, depth, dropout, lr, wd
Scheduler:       ASHA, grace 5, factor 3
Budget:          100 trials, 6 hours
Data snapshot:   snapshot-A
Code version:    commit label
Result:          best config and score, plus the spread over 3 seeds
Decision:        what was changed because of this study

Writing the decision down is the most valuable line. It turns a pile of numbers into knowledge that a colleague can use later.

Frequently asked questions

What is Ray Tune used for?

Ray Tune is used to search for good hyperparameters by running many training trials in parallel on one or many machines. It adds schedulers that stop weak trials early, handles resource allocation and recovers from failures, which makes large tuning jobs practical.

How is Ray Tune different from Optuna?

Optuna focuses on how to choose hyperparameter values and when to prune a trial, and it runs best on a single machine or a few workers sharing a database. Ray Tune focuses on running many trials across a cluster with resource management, fault tolerance and many schedulers. The two can be combined, with Optuna proposing values inside Ray Tune.

What is the ASHA scheduler?

ASHA stands for Asynchronous Successive Halving Algorithm. It gives each trial a small budget, keeps the best fraction at each stage and stops the rest, without waiting for every trial to finish a stage. It saves compute when early results predict final results.

Do I need a cluster to use Ray Tune?

No. Ray Tune runs on a single laptop or server and uses the available cores and GPUs. A cluster becomes useful when trials are heavy, numerous or need more hardware than one machine has.

What is population based training?

Population based training trains a group of models at the same time. Periodically, weaker models copy the weights and settings of stronger ones and then slightly change the settings. The result is a schedule of hyperparameters that adapts as training proceeds, rather than one fixed setting.

Can Ray Tune tune gradient boosted trees?

Yes. It can tune any code that reports a metric, including XGBoost, LightGBM and CatBoost. For small tables the overhead may outweigh the benefit, but for large data or many datasets it can speed up the search considerably.

How does Ray Tune handle failures?

It can restart a failed trial from its last checkpoint and can resume an interrupted experiment. This requires that your trainable saves checkpoints and can load them. Without checkpoints, a failed trial must start again from the beginning.

How many trials should I run?

There is no universal number. Start with a modest count to understand the landscape, check how the best score improves with more trials and stop when gains flatten. The needed number grows with the number of hyperparameters and the noise in the metric.

Is Ray Tune useful for tabular foundation models?

It is most useful during their construction, where it can tune the pre-training recipe and data generator settings on scaled-down runs, and during benchmarking, where it gives classical baselines a strong and fair tuning budget. In-context table models themselves need much less per-dataset tuning.

How do I keep tuning results trustworthy?

Use a validation design that matches how the model will be used, keep a separate test set, repeat the best configurations with several seeds, and record settings, seeds and code versions. Be wary of the best score from a large search because it is biased upwards.

Glossary

• ASHA: an asynchronous method that stops weak trials early.
• Checkpoint: a saved state from which a trial can resume.
• Cluster: several machines working together under one scheduler.
• Config: the set of hyperparameter values for one trial.
• Fault tolerance: the ability to continue after a failure.
• Fractional GPU: assigning part of a GPU to a trial.
• Grace period: the minimum budget before a trial may be stopped.
• Head node: the coordinating node of a cluster.
• Hyperband: a method that runs several successive halving brackets.
• Hyperparameter: a setting chosen before training.
• Population based training: an evolutionary method that adapts settings during training.
• Reduction factor: the ratio by which trials are cut at each stage.
• Scheduler: the component that decides which trials continue.
• Search algorithm: the component that proposes new configs.
• Trainable: the code that trains and reports for one trial.
• Trial: one training run with one config.

Summary

Ray Tune makes large-scale hyperparameter tuning practical. It runs many trials in parallel, packs them efficiently onto available CPUs and GPUs, stops weak trials early with schedulers such as ASHA, adapts settings with population based training and recovers from failures through checkpoints. It combines well with search libraries such as Optuna, so you can pair a smart proposer with a strong scheduler.

For tabular foundation models, distributed tuning matters mainly in building the models and in running fair benchmarks against tuned classical baselines such as XGBoost and CatBoost. In-context models like TabPFN shift most of the cost away from per-dataset tuning. Start small, make trials resumable, control costs, keep a clean test set and record everything. For more background, see Bayesian optimization, PyTorch and the model research page.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: