Torchtune

Overview

Torchtune is a PyTorch library for fine-tuning and evaluating large language models with as little abstraction as possible. Instead of hiding the training loop inside a large trainer class, it gives you short, readable recipes written in plain PyTorch. Each recipe is a single file that you can read from top to bottom, copy, and edit. Settings live in YAML config files, and a small command line tool launches the recipes.

The design choice matters. Many fine-tuning tools are powerful but opaque, which makes it hard to change a loss, a data format or a sharding rule. Torchtune trades some convenience for clarity and control. For teams that build custom models, including models that work with tables, that clarity is valuable. It also pairs naturally with native PyTorch tools such as FSDP and PyTorch itself, and it is a good contrast to wrapper toolkits like LLaMA-Factory and speed-focused libraries like Unsloth.

What it is

Torchtune is organized around a few simple ideas.

• Recipes. Each recipe is a Python file that implements one training or evaluation workflow, such as full fine-tuning, LoRA fine-tuning, or preference tuning. The code is meant to be copied and modified.
• Configs. A YAML file chooses the model, tokenizer, dataset, optimizer, scheduler and other settings. Values can be overridden from the command line.
• Components. Reusable building blocks, such as model builders, tokenizers, dataset classes and parameter-efficient modules. They are plain Python functions and classes.
• A command line. A tool to list recipes, copy them to your folder, download model weights and run jobs, locally or across several GPUs.

Design principles

What it covers

Typical workflows include supervised fine-tuning on instruction data, parameter-efficient tuning with LoRA and QLoRA, preference tuning such as DPO, distillation from a larger teacher model, and evaluation through integration with lm-evaluation-harness. It supports several popular open model families. The exact list changes, so confirm it in the current documentation.

What it is not

Torchtune is not a web interface and not a hosted service. It is also not a performance-trick library. It does not ship hand-written kernels in the manner of Unsloth. It aims to be fast enough through standard PyTorch features such as compiled graphs, mixed precision and sharding. If you want a point-and-click experience, a toolkit with a graphical board is easier. If you want to understand and own your training code, Torchtune is a strong choice.

Where it came from

The need for a library like this follows the path of modern fine-tuning. Early open model releases came with research code that was hard to reuse. Community tools grew to fill the gap, many built on the Hugging Face Trainer. They were convenient, but their layers of callbacks and configuration made it hard to see what the training loop actually did.

The PyTorch team responded with a library that follows the PyTorch philosophy: simple building blocks, explicit code, and few magic hooks. The goal was to let researchers and engineers fine-tune models using only PyTorch, with recipes small enough to read in one sitting. Along the way, features from core PyTorch matured, such as sharded data parallel training in FSDP, activation checkpointing, mixed precision and graph compilation. Torchtune used those directly.

The ideas it applies come from well-known work: the LoRA paper in 2021, QLoRA in 2023, and preference methods like DPO in 2023. Torchtune does not invent new algorithms. It makes them transparent and easy to run in native PyTorch. Project status and feature plans evolve, so check the repository for the current state before building a long-term workflow on it.

How it works

Think of two kinds of kitchens. In one, a robot chef takes an order form and produces a dish, and you have no idea what happens inside. In the other, there is a clear recipe card pinned above a normal stove. You can read each step, swap an ingredient and taste as you go. Torchtune is the recipe card.

Step by step

1. Install and list recipes. The command line shows which recipes and configs exist.
2. Download weights. A command fetches the model files and tokenizer from a model hub into a local folder.
3. Pick a config. You choose a ready-made config for your model and method, such as single-device LoRA.
4. Copy if you want to edit. A copy command places the recipe or config in your folder so you can change it.
5. Set overrides. You change dataset, batch size, learning rate or output directory from the command line or in the file.
6. Run. The launcher starts the recipe on one or several GPUs.
7. Checkpoint and log. The recipe saves checkpoints and writes metrics to a chosen logger.
8. Evaluate and generate. Separate recipes evaluate the model and produce text for spot checks.

The anatomy of a recipe

A recipe is a class with a small number of methods. The names vary between recipes, but the shape is consistent.

Because there is no deep inheritance, you can follow any value from the config to the line that uses it.

Configs and overrides

Here is a trimmed example of a config for LoRA fine-tuning. Field names follow the project's conventions but verify them against the version you use.

output_dir: /tmp/example-run

model:
  _component_: torchtune.models.example_model.lora_example_8b
  lora_attn_modules: ['q_proj', 'v_proj']
  lora_rank: 16
  lora_alpha: 32

tokenizer:
  _component_: torchtune.models.example_model.example_tokenizer
  path: /tmp/models/example-8b/tokenizer.model
  max_seq_len: 2048

dataset:
  _component_: torchtune.datasets.instruct_dataset
  source: json
  data_files: data/table_qa.json

batch_size: 2
epochs: 2
gradient_accumulation_steps: 8

optimizer:
  _component_: torch.optim.AdamW
  lr: 2e-4

lr_scheduler:
  _component_: torchtune.training.get_cosine_schedule_with_warmup
  num_warmup_steps: 50

loss:
  _component_: torch.nn.CrossEntropyLoss

dtype: bf16
enable_activation_checkpointing: true

The _component_ field names the Python object to build. This is a neat pattern: the config is a description of objects, so any function with matching arguments can be swapped in, including your own.

Overrides look like this.

tune run lora_finetune_single_device \
  --config example_8b/lora_single_device \
  batch_size=4 epochs=1 output_dir=/tmp/other-run

Multi-GPU training

For several GPUs, a recipe designed for distributed training uses native sharding. The launcher builds on PyTorch's standard distributed launch tool, so you pass the number of devices.

tune run --nproc_per_node 4 full_finetune_distributed \
  --config example_8b/full

Behind the scenes, the recipe wraps the model with sharded data parallelism, as described on the FSDP page. Parameters, gradients and optimizer states are split across ranks. Communication uses NCCL on NVIDIA GPUs. Options such as activation checkpointing, CPU offload of optimizer states, and low-precision settings help fit larger models.

Memory levers

Datasets

Datasets are classes that turn raw records into token IDs and labels. Ready-made ones handle instruction data, chat data, text completion and preference pairs. Writing a custom dataset means producing a dictionary of token IDs and label IDs for each example, with prompt tokens masked in the labels when you want to train only on answers. Because the dataset is an ordinary class, adding a new tabular-aware format is a small task.

Key concepts and terms

A worked example

Goal: fine-tune a small open model so that it labels the semantic type of a table column from its header and sample values. This is a narrow, measurable task. We will go from data to a trained adapter and check results.

Step 1: build the dataset

Create a JSON file where each record has an input and an output field.

import json, random

random.seed(11)

raw = [
    ("cust_dob", ["1984-02-11", "1991-07-30"], "date_of_birth"),
    ("amt_usd", ["12.50", "99.00"], "currency_amount"),
    ("prod_nm", ["Blue mug", "Steel bottle"], "product_name"),
    ("zip", ["10001", "94105"], "postal_code"),
    ("email_addr", ["a@example.internal", "b@example.internal"], "email_address"),
]

rows = []
for header, values, label in raw * 40:  # in real work use many distinct examples
    rows.append({
        "input": f"Header: {header}\nValues: {', '.join(values)}\nSemantic type:",
        "output": label,
    })

random.shuffle(rows)
split = int(0.9 * len(rows))
json.dump(rows[:split], open("data/col_types_train.json", "w"))
json.dump(rows[split:], open("data/col_types_eval.json", "w"))

In real use, never duplicate rows to reach a size. The loop above is only to keep the example short.

Step 2: copy and edit a config

tune ls
tune cp example_8b/lora_single_device my_col_types.yaml

Edit the dataset block to point to your JSON file and set output_dir. Adjust sequence length to be short, since these prompts are tiny. This frees memory for a larger batch.

Step 3: run

tune run lora_finetune_single_device --config my_col_types.yaml

Step 4: inspect the loop when something looks wrong

Because the recipe is plain PyTorch, you can add a print or a check in the training step. For example, to verify that prompt tokens are masked:

# inside the recipe, after a batch is loaded
tokens, labels = batch["tokens"], batch["labels"]
masked = (labels == -100).float().mean().item()
print(f"fraction of masked label positions: {masked:.2f}")

If this fraction is zero, the model is learning to copy prompts, which is rarely what you want.

Step 5: evaluate

def accuracy(preds, golds):
    ok = sum(p.strip() == g.strip() for p, g in zip(preds, golds))
    return ok / len(golds)

Generate a label for each evaluation example, compare to the gold label and compute accuracy. Also look at the confusion between similar labels, such as postal_code and phone_number. A confusion table tells you which examples to add.

Step 6: a back-of-the-envelope memory check

So a single 24 GB card can be enough for short sequences with LoRA. QLoRA lowers the weight part further. Full fine-tuning of the same model needs weights, gradients and Adam states for all parameters, which is many times larger, and is where sharding comes in.

Step 7: scale up if needed

If you want to run full fine-tuning on a larger model, switch to the distributed recipe, set the number of GPUs, enable activation checkpointing and use bf16. Test with a few steps to see memory per GPU, then raise batch size until you are near the limit.

Step 8: add knowledge distillation

If a large model gives good labels but is too costly to serve, train a smaller student to imitate it. The teacher produces outputs or probability distributions for your inputs, and the student learns to match them. In a recipe, this adds a second model that runs in evaluation mode and a loss that compares the two outputs. The benefit is a small model with much of the teacher's behavior on your narrow task. The risk is copying the teacher's mistakes, so check the student on gold labels and not only on teacher agreement.

Step 9: a simple results table

Keep a table like this for every batch of runs.

Fill the measured column with your own numbers. Do not copy numbers from other projects, since they depend on data and model.

Reading a loss curve like a story

A healthy fine-tuning curve has three parts. At the start, loss falls quickly as the model learns the format. In the middle, the fall slows as it learns the content. Near the end, it flattens. If eval loss turns upward while train loss keeps falling, the model has started to memorize. Stop at the turning point or add data. Think of a student: the first hour of study brings the biggest jump, and cramming the same page ten more times brings none.

Using your own model builder

Because configs name components by path, you can plug in a custom model. Write a function that returns a module, point _component_ at it, and the recipe will build it. This is how research teams test a new layer, such as a different attention pattern or a new embedding for numeric cells, while reusing the rest of the loop: data loading, sharding, checkpointing and logging.

import torch.nn as nn

def tiny_table_encoder(d_model: int = 128, n_layers: int = 4) -> nn.Module:
    layer = nn.TransformerEncoderLayer(d_model=d_model, nhead=4, batch_first=True)
    return nn.TransformerEncoder(layer, num_layers=n_layers)

The function is only a sketch. A real table encoder would add cell embeddings, row and column attention and a masking path for missing values.

Strengths and limitations

Strengths

• Readable code. A recipe fits in one file, so you can understand and change it.
• Few dependencies. It leans on PyTorch rather than a stack of wrappers.
• Good defaults for memory. LoRA, QLoRA, checkpointing and sharding are built in.
• Config as objects. The _component_ pattern makes swaps easy.
• Native distributed training. Uses the same FSDP and launcher you would use in your own code.
• A good teaching tool. You see every step of a modern fine-tuning loop.

Limitations

• Fewer conveniences. No graphical interface, and fewer built-in data format helpers than broader toolkits.
• Smaller model and method menu. Coverage may lag behind wrapper toolkits when new models appear.
• Not the fastest for tiny setups. Kernel-level libraries can beat it on a single GPU for LoRA.
• You own the code. Copying a recipe means you also own its maintenance.
• Project direction can change. As with any fast-moving open project, check its current status before committing to it for a long program.

How it compares with alternatives

A good rule: pick Torchtune when you expect to modify the training logic, or when you want your fine-tuning code to match what you would write by hand in PyTorch.

Role in tabular foundation models

Tabular foundation models learn from raw tables. Their inputs are rows and columns of mixed types, with missing values and often meaningless or absent column names. Models such as TabPFN are pre-trained on very many synthetic tables sampled from a prior over data-generating processes, and then solve new tasks through in-context learning: the labeled rows are placed in the context, and the model predicts for new rows in a single forward pass. Related models like TabICL, TabSTAR and FT-Transformer explore other encoders and training ideas.

Torchtune is a language model tool, so it does not train those architectures out of the box. Its connection to tabular work is more about method and about the surrounding system.

Why its style suits table model research

Tabular foundation models need custom components: a cell encoder for numbers and categories, attention across rows and across columns (compare axial attention and Perceiver IO), a masking scheme for missing values, and unusual training targets. Native PyTorch code is the right base for that work. The habits Torchtune teaches, such as small composable components, config-described objects, explicit loops, and sharding through FSDP, are the same habits used when building a table model. A team that has read and modified Torchtune recipes will find it easy to write a pre-training loop for a table transformer.

Pre-training on synthetic and real tables

Pre-training uses a stream of generated tables. Each batch holds many small tasks, with a training split used as context and a test split used for the loss. The model also needs masked cell objectives, see masked cell modeling. Recipes of Torchtune style, with a dataset class that samples a synthetic task, a loss that predicts masked cells or labels, and a sharded optimizer, are a close fit for such a stack. The library itself will not generate the tables. You supply the generator and the dataset class. See also synthetic data generation and self-supervised learning.

Language models around table models

Fine-tuning a language model to write SQL, to label column semantics from headers and values, or to narrate model outputs is a direct use of Torchtune. This supports tasks such as sector or semantic classification of columns, which can feed metadata into the table model. It also helps when headers are absent: a text model can propose names that help downstream users. Related ideas appear in semantic sector heads.

Imputation, anomalies and continual learning

For imputation, tabular models fill missing cells by conditioning on observed ones, see imputation. For anomaly detection, they can flag rows with low likelihood or large reconstruction error, see anomaly detection. A Torchtune-tuned language model can complement these by explaining a flagged row or filling free-text cells. For continual learning, you can keep a frozen base and train separate LoRA adapters per domain, a pattern that limits forgetting and avoids the complexity of regularizers such as EWC.

Where to look next

For architecture ideas see the architecture page. For the research background see the research page. For measured comparisons see the benchmarks page.

Practical guidance and best practices

Start from a shipped config

• Run the list command to see what exists for your model.
• Copy the closest config and change as little as possible.
• Keep your copy under version control, with the exact library version noted.

Keep the data side clean

• Check label masking in the first batch.
• Make sure the tokenizer and special tokens match the model.
• Hold out a fixed evaluation set and never tune on it.
• Remove personal data before training, for example with Presidio.

Choose settings sensibly

Debug in the open

• Add prints in the recipe for shapes, label masks and loss values.
• Overfit ten examples first. If the loss does not go near zero, something is broken.
• Compare the first-step loss with a known reasonable value for your vocabulary size.
• Check memory with a short run before a long one.

Scale step by step

1. Get a single-GPU run working.
2. Move to one node with several GPUs.
3. Add multi-node only when needed, and verify network speed.
4. Save sharded checkpoints for speed and consolidate at the end, see FSDP.

Track and serve

• Log metrics to a tracker like MLflow or Weights & Biases.
• Evaluate with a harness for general abilities and a custom script for your task.
• Export and serve with an engine such as vLLM or convert to GGUF for local runtimes.

Testing your changes

• Write a unit test that builds the model from your config and runs one forward pass on random tokens.
• Add a test that a saved checkpoint reloads to identical outputs.
• Compare your modified recipe against the original on a tiny dataset; loss curves should match until your change takes effect.
• Pin a small regression set and run it after each library upgrade.

Choosing between single-device and distributed recipes

Documentation habits

Write a short readme next to each copied recipe. State why you copied it, what you changed, which library version it was based on, and how to reproduce the best run. Future you, or a new teammate, will need this when the upstream recipe changes.

Common mistakes

1. Editing a shipped recipe in place. You lose track of what changed. Copy first.
2. Skipping the overfit test. A broken loop can look fine for a while.
3. Unmasked prompts. The model learns to repeat the prompt.
4. Wrong tokenizer or special tokens. Quality drops without any error message.
5. Mixing config and override sources without recording them. Runs become hard to reproduce.
6. Using full tuning when LoRA is enough. It costs far more memory and time.
7. Ignoring activation memory. Sharding does not shard activations; use checkpointing.
8. Large batch with a small learning-rate schedule mismatch. Scale the rate with care.
9. Treating loss as the only metric. Always add a task metric.
10. Not checking the project status. Confirm that the features you rely on are maintained.

Frequently asked questions

What is Torchtune?

Torchtune is a PyTorch library of recipes for fine-tuning and evaluating language models. Each recipe is a readable Python file, and settings live in YAML configs. It aims to keep abstraction minimal so that you can understand and modify the training code.

Who should use Torchtune?

It suits researchers and engineers who want control and clarity. If you plan to change the loss, the data pipeline or the sharding approach, or you want to learn how modern fine-tuning works, it is a good fit. If you only want a quick result through a graphical interface, a toolkit with a web board may be easier.

How is it different from the Hugging Face Trainer?

The Hugging Face Trainer provides a large, general training class with callbacks and many options. Torchtune provides short recipes where the loop is visible and there is little inheritance. The first favors convenience and breadth, while the second favors transparency and hackability.

Does it support LoRA and QLoRA?

Yes, it includes LoRA and QLoRA variants for supported models, along with options for activation checkpointing and mixed precision. These make it possible to fine-tune sizable models on a single GPU. Check the documentation for the current list of models and methods.

Can it train across multiple GPUs?

Yes. Distributed recipes use native sharded data parallel training with a standard launcher, so parameters, gradients and optimizer states are split across devices. You pass the number of processes and tune the memory options.

Can I use it to pre-train a tabular foundation model?

It is not built for that, but its style is a good template. Table models need custom encoders and objectives, which you would write in plain PyTorch. Torchtune's approach to small components, config-described objects and FSDP sharding transfers well. You would supply the dataset class that generates or loads tables.

Does it make training faster than other tools?

It uses standard PyTorch features, such as mixed precision, sharding and graph compilation, so it can be efficient, but it does not focus on hand-written kernels. Libraries built around custom kernels can be faster on a single GPU for LoRA. Measure on your own workload.

How do I add my own dataset format?

Write a dataset class that returns token IDs and labels for each example, register it through a config entry with the _component_ field, and test with a few batches. Verify label masking and sequence length. Because the code is plain Python, this is usually a short change.

How do I evaluate a model after tuning?

Use the evaluation recipe with a harness for general abilities, and add your own script for the task metric. For column labeling, accuracy and a confusion table are useful. For SQL, compare result sets after execution. Compare to a baseline from the base model.

Is Torchtune good for production pipelines?

It can be, if you treat the recipe copy as your own code, pin versions, keep configs under version control and add tests. Its transparency helps with audits. Review the project's current status and plan for maintenance of your copy.

Glossary

• Activation checkpointing: Recomputing activations in the backward pass to save memory.
• Builder: A function that creates a model of a chosen variant.
• Component: An object described in a config by its Python path.
• Config: A YAML file with settings and component choices.
• DPO: Direct preference optimization.
• FSDP: Fully sharded data parallel training in PyTorch.
• Label masking: Ignoring some token positions when computing the loss.
• LoRA: Low-rank adaptation using small trainable matrices.
• Override: A command line value that replaces a config entry.
• QLoRA: LoRA on a 4-bit quantized base model.
• Recipe: A single-file workflow for training or evaluation.
• Sharding: Splitting tensors across devices.
• Tokenizer: The component that maps text to token IDs.

Summary

Torchtune is a PyTorch-native library that treats fine-tuning as readable code. It offers recipes, YAML configs and a small command line, with built-in support for LoRA, QLoRA, mixed precision, activation checkpointing and sharded training. Its main value is clarity: you can see and change every part of the loop.

For tabular foundation models, it is not the trainer of the table model itself, but its habits fit that work well. Table models need custom encoders, masked-cell objectives and sharded pre-training on synthetic and real tables, all of which are written in plain PyTorch with tools like FSDP. Torchtune is also a practical way to tune the language models that surround tables, from SQL writers to column labelers. Start from a shipped config, test with a tiny overfit run, keep a baseline, and move to larger setups only after the small one is correct.

Related

It belongs to Training tooling in my stack, under LLMs, retrieval, and evaluation. Others in the same group: