Axolotl

Overview

Axolotl is an open-source framework that fine-tunes language models from a single YAML configuration file. You describe the base model, the dataset, the tuning method and the training settings in one text file. Axolotl then builds the data pipeline, wraps the model, launches training and saves the result. You do not write a training loop.

The main value is repeatability. A config file can be stored in version control, reviewed, shared and rerun. Changing a technique, such as moving from full fine-tuning to LoRA or QLoRA, is usually a few lines in the file. This makes Axolotl a practical choice for teams that run many fine-tuning experiments and want each one to be easy to reproduce. For readers who work with structured data, the same ideas matter when adapting large models to tables, as covered in /model/research.

What it is

Axolotl is a wrapper and orchestrator. It sits on top of well-known libraries such as Hugging Face Transformers, PEFT, bitsandbytes and PyTorch. It does not replace them. It connects them behind one consistent interface.

The core idea

Fine-tuning scripts tend to look alike. They load a model, load a tokenizer, format a dataset, set hyperparameters and train. Most of the differences between projects are choices, not code. Axolotl turns those choices into configuration keys.

Think of a kitchen with a standard recipe card. The cook (Axolotl) knows how to run the stove and the oven. You only fill in the card: which ingredients, how long, what temperature. Two people with the same card get the same dish.

What you can configure

• The base model and its tokenizer.
• The tuning method: full fine-tuning, LoRA, QLoRA and related adapter methods.
• The datasets, their formats and how to turn rows into prompts.
• Sequence length, batching and packing.
• Optimizer, learning rate schedule and number of epochs.
• Precision, memory savings and multi-GPU strategy.
• Preference tuning methods such as DPO.
• Logging and checkpoint options.

What it is not

Axolotl is not a model, not an inference server and not a hosted service. It is a training-side tool. For serving the tuned model you would use something else, such as vLLM.

Where it came from

Axolotl grew out of the open-source community around fine-tuning open-weight language models. After open models became widely available, many hobbyists and small labs wanted to adapt them to chat and instruction data. Each person wrote their own script, and the scripts drifted apart.

A shared, config-driven tool solved a real problem: people wanted to compare recipes without rewriting code. Axolotl became popular for that reason. Its growth followed the wider arrival of parameter-efficient methods such as LoRA and the 4-bit quantization behind QLoRA, which made fine-tuning possible on one or a few consumer or workstation GPUs.

The technology it builds on

Because it reuses these parts, Axolotl improves when they improve. It also inherits their quirks, which is worth knowing when debugging.

How it works

The workflow has a clear order. Each step is a place where a setting in the YAML file takes effect.

Step 1: Read the config

Axolotl parses the YAML file and checks the keys. It fills in defaults and warns about combinations that do not make sense. Think of a pre-flight checklist: before the plane moves, someone confirms the fuel and the route.

Step 2: Load the tokenizer and model

It loads the base model named in the config. If quantization is requested, it loads the weights in a low-bit format. If an adapter is requested, it attaches small trainable matrices to chosen layers and freezes the rest.

The analogy is a heavy textbook with sticky notes. The book (the base model) stays unchanged. You write on the notes (the adapter). Later you can peel the notes off or keep them.

Step 3: Prepare the dataset

Each dataset entry has a type. The type says how rows become text. For example, a chat dataset has turns with roles. An instruction dataset has an instruction, an optional input and an output. Axolotl applies a template, tokenizes the text and builds labels.

A key detail is label masking. In many setups the loss is computed only on the answer part, not on the prompt. This tells the model to learn how to respond, not to memorize questions.

Step 4: Pack sequences

Short examples waste compute because they must be padded to equal length. Sample packing joins several short examples into one long sequence, with attention masks that keep them separate. It is like loading a moving truck: instead of one small box per trip, you fill the truck.

Step 5: Choose the memory strategy

Several switches reduce memory:

• Mixed precision, usually bf16 on modern GPUs. See mixed precision.
• Gradient checkpointing, which recomputes activations instead of storing them.
• Low-bit weights as in QLoRA.
• Sharding the training state with DeepSpeed ZeRO or FSDP.

Step 6: Train and save

The training loop runs through the Hugging Face trainer machinery. Checkpoints and logs are written to the output directory. At the end, the adapter weights, or the full model, are saved. Adapters can be merged back into the base model for simple deployment.

Step 7: Evaluate and iterate

Many users hold out a validation split and track loss. Loss alone is a weak signal for quality, so a task-specific check is also needed. The config can be edited and the run repeated.

Step 8: Know the folder layout

A run leaves a predictable set of files. Knowing them saves time when something goes wrong.

• The output directory holds checkpoints, logs and the final adapter or model.
• A prepared-data cache holds the tokenized dataset, so a second run starts faster.
• A copy of the config is often saved next to the results, which helps later audits.
• Tokenizer files are saved so the tuned model loads without extra steps.

Step 9: Understand the data path in more detail

The data path is where most problems start, so it is worth a closer look.

1. Raw files are read from disk or from a hub. Common shapes are JSON lines, parquet and CSV.
2. Each row is mapped into a conversation or an instruction record.
3. A template turns the record into one string with role markers.
4. The tokenizer converts the string into integer ids.
5. A label array is created. Prompt tokens get an ignore value so they do not count in the loss.
6. Examples longer than the sequence length are dropped or truncated.
7. If packing is on, examples are grouped to fill each sequence.
8. A collator builds batches with the right masks.

A useful analogy is a mail room. Letters arrive in many shapes. A clerk puts each in a standard envelope, writes a label, and sorts them into trays. The training loop only sees neat trays. If the clerk uses the wrong envelope, nothing downstream can fix it.

Multiple datasets

A config can list several datasets. Each can have its own type and its own template. They are mixed into one stream. This is handy when you want to blend general chat data with a small amount of task data so the model keeps its broad skills. The mix ratio matters. Too much task data can narrow the model. Too little may not move it.

Hardware and launch behavior

On a single GPU the tool starts one process. On several GPUs it starts one process per device through a launcher. Each process holds a slice of the data and, with sharding, a slice of the model state. Logs from the first process are the main source of truth. If one process fails, the others often hang, so check every rank when a run stalls.

Key concepts and terms

Effective batch size, in plain words

Imagine you want to taste-test 32 soup samples before changing the recipe, but your table only fits 4 bowls. You test 4 bowls eight times, note the results, then adjust once. That is gradient accumulation. The effective batch is 32 even though only 4 fit at once.

A worked example

Below is a small QLoRA config. The model name and dataset path are placeholders.

base_model: example-org/example-7b
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true

datasets:
  - path: ./data/train.jsonl
    type: alpaca

val_set_size: 0.05
output_dir: ./outputs/example-qlora

sequence_len: 2048
sample_packing: true

micro_batch_size: 2
gradient_accumulation_steps: 8
num_epochs: 3
learning_rate: 0.0002
lr_scheduler: cosine
warmup_steps: 20
optimizer: paged_adamw_8bit

bf16: auto
gradient_checkpointing: true
logging_steps: 10
save_strategy: epoch

Launch it from a shell. The exact command form varies by version, so check the installed help text.

axolotl train config.yml

Reading the config

• load_in_4bit and adapter: qlora say the base weights are stored in 4-bit form and a LoRA adapter is trained on top.
• lora_r is the rank. A larger rank means more trainable capacity and more memory.
• lora_alpha scales the adapter output. A common habit is to set it to about twice the rank, though this is a convention, not a law.
• lora_target_linear applies adapters to all linear layers, not only attention.
• sequence_len and sample_packing control how text is arranged into batches.
• The effective batch size here is 2 times 8, so 16 per GPU.

A numeric check

Suppose the dataset has 8,000 examples averaging 400 tokens. With packing at 2,048 tokens, each packed sequence holds about five examples. That gives roughly 1,600 packed sequences per epoch. With an effective batch of 16, an epoch is about 100 optimizer steps. Three epochs is about 300 steps. Knowing this helps you set warmup_steps and save_strategy sensibly.

Merging the adapter

After training, a merge step folds the adapter into the base model so serving does not need the adapter code path. A typical command looks like this, with placeholder paths.

axolotl merge-lora config.yml --lora-model-dir ./outputs/example-qlora

A DPO-style variation

Preference tuning needs paired data: a prompt, a preferred answer and a rejected answer. Axolotl exposes this through an rl style key in newer versions. Details change, so read the current documentation for exact names. The principle matches DPO: the model learns to prefer one response over another without a separate reward model, a simpler path than full RLHF.

Strengths and limitations

Strengths

• One file describes a whole run, which supports review and reproducibility.
• Broad coverage of models, adapters and dataset formats.
• Good defaults for packing, masking and chat templates.
• Works with multi-GPU strategies such as ZeRO and FSDP.
• Active community, so new models and methods are often supported quickly.
• Lets non-experts try recipes without touching the training loop.

Limitations

• The YAML surface is large. Some keys interact in ways that are not obvious.
• Behavior depends on versions of Transformers, PEFT, bitsandbytes and CUDA. Mismatches cause confusing errors.
• It is a thin layer, so deep debugging often requires reading the libraries below it.
• If you need a custom loss or a new training algorithm, a config is not enough and you will edit code.
• Speed is generally good but not always the fastest. Some specialized kernels in other tools can beat it on single-GPU runs. See Unsloth.
• Documentation can lag behind rapid changes.

When Axolotl is the wrong tool

• You only need prompting or retrieval. Fine-tuning adds cost and upkeep, so try advanced RAG or better prompts first.
• You need a brand new training objective. Write a custom trainer instead.
• You have fewer than a few hundred clean examples. Results will be noisy and hard to judge.
• You cannot evaluate the output. Without a test, you cannot tell if the run helped.
• The task is purely numeric prediction on a table. Use a model built for tables.

A decision checklist

1. Is the goal clear and measurable?
2. Is there a held-out test set?
3. Was a simpler method tried?
4. Is the data clean, licensed for use and free of private details?
5. Is the config saved with the data version?

If all five answers are yes, a run is likely to teach you something useful.

How it compares with alternatives

How to choose

If you value a shareable, declarative file, pick Axolotl or LLaMA-Factory. If you want to read and edit every line, pick Torchtune or a hand-written script. If memory and speed on one GPU are the top concern, look at Unsloth. These tools are not exclusive. Some can be combined with ZeRO or FSDP for scale.

Role in tabular foundation models

Axolotl is built for language models, so its link to tabular foundation models is indirect. It is still useful to understand the link, because the same adaptation ideas apply when a model must learn from raw tables.

Why tables are different

A table has rows and columns. Columns can be numbers, categories, dates or text. Values can be missing. Column names may be absent or meaningless. A tabular foundation model, such as TabPFN, is pretrained on a large collection of synthetic tables so that it can read a new table at inference time and predict without gradient updates. This is in-context learning: the training rows are shown to the model as context, and it predicts the labels of new rows in one forward pass. The architecture side of this is discussed in /model/architecture.

Where a fine-tuning framework fits

1. Text-serialized tables. One approach turns each row into text, such as "age is 41, income is 52 thousand, city is Leeds", and fine-tunes a language model to predict a target. Axolotl can run this kind of supervised fine-tuning. The dataset type would be a simple instruction format where the input is the serialized row.
2. Semantic and sector classification. Column headers and cell text often carry meaning. A language model tuned with Axolotl can classify a column into a semantic type, or label a record by industry sector, using short text prompts. This helps when headers are missing and the model must infer meaning from values.
3. Imputation by generation. A tuned model can be asked to fill a missing cell given the rest of the row. This is useful for text-like or categorical fields. For purely numeric imputation, a model built for tables usually does better.
4. Anomaly explanation. A tabular model can flag an unusual row. A tuned language model can then explain in words why the row looks odd, using a prompt that contains the row and the flagged columns.
5. Continual learning. When new data arrives, adapters offer a way to add knowledge without retraining the whole model. A new adapter can be trained for a new domain, which limits forgetting of earlier skills. The risk of overwriting old behavior remains and must be measured.

Honest limits for table work

Language models struggle with exact arithmetic and with very wide tables. Serializing rows into text can lose numeric precision and makes sequences long. For core prediction on a table, a model designed for tables is usually a better first choice. Fine-tuning a language model is best viewed as a complement, for the text-heavy and semantic parts of table work. You can compare approaches on the standards listed in /model/benchmarks.

Feeding results back into a table workflow

A tuned language model is rarely the whole pipeline. A common layout in table work looks like this:

1. A tabular model such as TabICL or TabPFN makes the numeric predictions and finds odd rows.
2. A tuned language model labels columns, normalizes text values and writes short explanations.
3. A rules layer checks types and ranges, and rejects bad output.
4. A human reviews the cases flagged as uncertain. See human in the loop.

In this layout, Axolotl's job is step 2 only. Keeping the roles separate makes errors easy to trace. If a number is wrong, look at the tabular model. If a label is wrong, look at the tuned language model and its data.

Building training data for column work

Good data for column typing and sector labels has a few properties:

• Many column types, including rare ones.
• Examples with and without headers, so the model learns to rely on values.
• Mixed formats for the same type, such as several date styles.
• Missing and malformed values, since real tables have them.
• Balanced classes, or at least known imbalance that you plan for.

Synthetic examples can fill gaps. Generate values by rule, then check a sample by eye. See synthetic data generation. Keep a real hold-out set so you do not grade the model on its own training recipe.

A small example of a table prompt

{"instruction": "Classify the column type. Answer with one label.",
 "input": "values: 2021-03-04, 2021-03-09, 2022-01-17",
 "output": "date"}

A few thousand such examples, covering many column types, can teach a model to label unseen columns even when the header is missing.

Practical guidance and best practices

Start small

Begin with a short run on a small slice of data. Confirm the pipeline works, the prompts look right and the loss falls. Then scale up.

Inspect the formatted data

Print a few tokenized examples with the template applied. Check that special tokens and role markers are correct. A wrong template is one of the most common silent causes of poor results.

Pick the method by budget

Tune a few knobs first

• Learning rate. Adapter methods usually tolerate higher rates than full fine-tuning.
• Rank and alpha.
• Number of epochs. More epochs can overfit small datasets.
• Sequence length. Do not set it far above your real data length.

Keep runs reproducible

• Store the config with the output.
• Record library versions.
• Fix the seed.
• Log the dataset version or hash.

Evaluate beyond loss

Build a small held-out test set that matches the real task. Check outputs by reading them, and use automatic metrics where the task allows. Compare against the untuned base model so you know the tuning helped.

Watch for data leakage

If the evaluation rows also appear in the training data, scores will look better than reality. Split by group, such as by customer or by time, when rows are related.

Plan the deployment

Decide early whether to serve the adapter separately or merge it. Merged models are simpler to serve. Separate adapters allow swapping tasks on one base model.

Choosing a LoRA rank, with a rough feel

Rank sets how many numbers the adapter has per target layer. A rank of 8 is light. A rank of 64 is heavy. The adapter holds two thin matrices, so the count grows linearly with rank.

These are habits, not rules. Test two ranks on your own task before settling.

Sizing a run before you start

A quick estimate prevents surprises.

• Weights: a 7 billion parameter model needs about 14 GB in 16-bit form and about 4 GB in 4-bit form.
• Adapter and optimizer state: small for LoRA, large for full fine-tuning, where the optimizer can need several times the weight size.
• Activations: grow with batch size and sequence length. Gradient checkpointing trades compute for lower activation memory.
• Headroom: leave room for the CUDA context and fragmentation.

If the estimate is close to the limit, reduce micro batch size first, then sequence length, then consider a smaller rank or a shorter model.

Logging and monitoring

Connect a logging service if you have one, and watch these signals:

• Training loss: should fall quickly, then slowly.
• Validation loss: if it rises while training loss falls, the model is overfitting.
• Gradient norm: spikes hint at an unstable learning rate.
• Tokens per second: a sudden drop points to a data or hardware problem.
• GPU memory: steady use is good; a steady climb suggests a leak.

A short debugging routine

1. Run ten steps on a tiny dataset. Confirm that it finishes.
2. Print one decoded training batch. Read it as text.
3. Check that the prompt part is masked and the answer part is not.
4. Run a generation test on the saved adapter with a prompt from training. The answer should look close to the training answer.
5. Run a generation test on an unseen prompt.
6. Compare with the base model on the same prompts.

If step 3 fails, fix the template or the dataset type before anything else.

Working in a team

• Keep configs in version control with a short name that says what changed.
• Use one config per experiment instead of editing one file again and again.
• Store data under a version tag. A model is only as reproducible as its data.
• Write down the question each run answers. Runs without a question pile up and confuse everyone.

Common mistakes

• Wrong chat template. The model sees different markers in training and inference.
• Training on the prompt. Not masking prompt tokens can waste capacity on copying instructions.
• Sequence length too small. Examples are truncated and the answer is cut off.
• Sequence length too large. Memory and time are wasted on padding.
• Learning rate too high. Loss spikes, or the model forgets its general skills.
• Too many epochs on small data. The model memorizes examples.
• Ignoring versions. A newer library breaks an older config key.
• Skipping evaluation. A falling loss is mistaken for a good model.
• Assuming packing is free. With some setups, packing changes effective batch behavior and needs checking.
• Merging into a quantized base. Merging low-bit weights needs care. Merge into a full-precision copy when quality matters.
• Mixing up the effective batch size. Forgetting to multiply by the number of GPUs changes learning dynamics.

Frequently asked questions

What is Axolotl used for?

Axolotl is used to fine-tune language models from a YAML configuration. It supports full fine-tuning and adapter methods such as LoRA and QLoRA, and it handles data formatting, packing and multi-GPU launch so that you do not need to write a training loop.

Do I need to know Python to use Axolotl?

Basic use needs only the config file and a command line. You do need some comfort with installing packages, reading logs and preparing a dataset file. Python skills become important when you want custom data processing or a new training method.

Can Axolotl run on a single GPU?

Yes. With QLoRA, gradient checkpointing and a modest sequence length, models in the multi-billion parameter range can be tuned on one GPU with enough memory. The right choice depends on model size and your memory budget.

How is Axolotl different from LLaMA-Factory?

Both are config-driven wrappers around the same core libraries. LLaMA-Factory puts more emphasis on a built-in web interface for trying models and methods, while Axolotl is mostly used through YAML files and the command line. The best choice depends on whether you prefer a visual or a file-based workflow.

Does Axolotl support multi-GPU training?

Yes. It can launch distributed runs and works with sharding approaches such as DeepSpeed ZeRO and FSDP. These split the training state across devices so larger models fit and training runs faster.

What is sample packing and should I use it?

Sample packing joins several short examples into one long sequence to reduce padding waste. It often speeds up training when your examples are much shorter than the sequence length. Check that attention masking keeps examples separate and compare results with an unpacked run if quality matters.

Can Axolotl be used for preference tuning?

Yes. Recent versions support preference-based methods such as DPO, which train on pairs of preferred and rejected answers. The exact config keys change between versions, so consult the current documentation.

Can I fine-tune a model on tabular data with Axolotl?

You can, by turning rows into text prompts and training a language model to answer. This suits tasks such as column type labeling, record classification and text-heavy fields. For core numeric prediction, models built for tables, such as TabPFN, are often a stronger starting point.

What is the difference between LoRA and QLoRA in an Axolotl config?

LoRA trains small adapter matrices on top of frozen weights stored in normal precision. QLoRA does the same but stores the frozen weights in 4-bit form, which cuts memory a lot. In the config this is mostly the difference between setting the adapter to lora or qlora and enabling 4-bit loading.

How do I know the fine-tune worked?

Compare the tuned model with the base model on a held-out set that matches your real task. Read sample outputs, use task metrics where possible, and check that general abilities did not collapse. Training loss alone is not enough.

Glossary

• Adapter: A small set of trainable weights added to a frozen model.
• Base model: The pretrained model used as the starting point.
• bf16: A 16-bit floating point format with a wide range, popular for training.
• Checkpoint: A saved snapshot of training state or weights.
• Chat template: Text rules that mark roles and turns in a conversation.
• Config: A file that sets all options for a run.
• DPO: Direct preference optimization, a method that trains on preferred and rejected answers.
• Epoch: One full pass over the training data.
• Fine-tuning: Continuing training of a pretrained model on a narrower dataset.
• Gradient accumulation: Summing gradients over several mini-batches before one update.
• LoRA: Low-rank adaptation, a parameter-efficient tuning method.
• Merge: Combining adapter weights into base weights.
• Packing: Putting several short examples in one sequence.
• QLoRA: LoRA applied to a 4-bit quantized base model.
• Rank: The inner size of the LoRA matrices, which sets their capacity.
• Sharding: Splitting training state across devices.
• Tokenizer: The component that converts text to integer tokens and back.
• Validation split: Data held out to monitor generalization.

Summary

Axolotl turns language model fine-tuning into a configuration task. One YAML file names the model, data, method and hardware strategy, and the framework handles loading, formatting, packing, distributed launch and saving. Its strengths are repeatability, breadth and sensible defaults. Its limits are version sensitivity, a large option surface and the need to edit code for novel algorithms.

For work around tables, Axolotl is a complement, not a replacement for models built for tabular data. It helps with text-serialized rows, semantic column typing, sector labels and explanation tasks, and its adapter approach supports adding domains over time. Pair it with careful evaluation, compare it with sibling tools such as Unsloth, LLaMA-Factory and Torchtune, and see /model/research and /model/benchmarks for how adaptation fits the wider picture.

Related

It belongs to Training tooling in my stack, under LLMs, retrieval, and evaluation. Others in the same group: