LLaMA-Factory

Overview

LLaMA-Factory is an open-source toolkit that lets you fine-tune many different language models through one consistent interface. You can drive it from a command line with a configuration file, or from a web interface where you pick a model, a dataset and a method with a few clicks. It supports a wide range of training approaches, from plain supervised fine-tuning to preference tuning, with full tuning, LoRA and QLoRA as options.

The point of the tool is speed of experimentation. Instead of writing a new training script for every model family, you change a few fields in a config and run. That makes it a handy way to try a model, compare tuning methods, and get a first working result. For people who work with tables, it offers a quick path to a language model that writes queries, explains data or labels columns, sitting next to specialist models such as TabPFN and TabICL.

What it is

LLaMA-Factory is a Python project built on top of the Hugging Face stack. It wraps Transformers, PEFT, TRL and related libraries behind a unified set of options. It also integrates optional speedups and memory savers, which can include bitsandbytes quantization, FlashAttention and sharding tools such as DeepSpeed ZeRO.

Its main parts are:

• A command line. A single entry command reads a YAML file and runs training, evaluation, chat or export.
• A web interface. A browser-based board where you fill in forms, start a run, watch loss curves and test the model in a chat box.
• A dataset registry. A small JSON file describes your datasets and how their fields map to prompt, response and other roles.
• Prompt templates. Named templates that format conversations the way each model family expects.
• Training stages. Pre-training, supervised fine-tuning, reward modeling and preference-based methods.
• Export tools. Merge adapters into base weights and write the result for use in other runtimes.

Training stages in plain words

The list of supported methods changes as the project grows, so check the current documentation for the exact set.

What it is not

LLaMA-Factory is not a foundation model and not a hosting platform. It does not replace experiment tracking or data engineering. It is also a wrapper, so when something fails deep inside a dependency, you may need to read the underlying library to understand why. For code you can read from top to bottom, Torchtune takes a different path.

Where it came from

The recipe for adapting a pretrained model grew from a few ideas. The LoRA paper in 2021 made tuning small adapter matrices popular. QLoRA in 2023 combined that with 4-bit base weights. Preference methods such as DPO, introduced in 2023, offered a simpler alternative to full reinforcement learning from human feedback. RLHF had been the standard approach before.

As open model families multiplied, a new problem appeared. Every family had its own chat format, its own tokenizer details and its own quirks. Researchers and hobbyists kept rewriting similar scripts. Projects like LLaMA-Factory responded by collecting these into one codebase with a common configuration surface. The name points to the early open LLaMA family, but the toolkit has grown to cover many other architectures.

Over time it added a graphical board, dataset conventions, more training stages, and hooks for faster kernels and distributed training. Today it is often used in classrooms, small labs and early-stage product teams as a fast way to get from idea to tuned model.

How it works

Imagine a workshop with one workbench and a wall of labeled tool drawers. You bring a piece of wood (the base model) and a plan (the dataset). The labels on the drawers (the config fields) tell the workbench which tool to use. You do not rebuild the workbench for each project.

Step by step

1. Choose a base model. You give a model name or a local path. The toolkit loads the tokenizer and weights.
2. Choose a template. The template tells the toolkit how to turn a conversation into the exact text the model was trained on, with special tokens and role markers.
3. Register data. In the dataset registry, you describe your file: its format and which columns hold the instruction, input and output.
4. Pick a method. You set the stage, such as supervised fine-tuning, and the tuning type, such as LoRA or full.
5. Set hyperparameters. Learning rate, epochs, batch size, sequence length, scheduler and precision.
6. Launch. Run the command or press the start button. The toolkit builds a trainer and begins training.
7. Monitor. Loss logs and optional dashboards show progress. The web board plots curves live.
8. Evaluate and chat. Try the model in a chat panel or run a batch evaluation on a held-out set.
9. Export. Merge adapters if needed and save the model for your serving stack.

The configuration file

Most users interact with a YAML file. Here is a small example for LoRA-based supervised fine-tuning. Field names follow the common conventions of the project, but verify them against the current version.

### model
model_name_or_path: example-org/small-instruct-model

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 16
lora_target: all

### dataset
dataset: table_qa_demo
template: default
cutoff_len: 2048
max_samples: 5000
overwrite_cache: true

### output
output_dir: saves/table-qa-lora
logging_steps: 10
save_steps: 200
plot_loss: true

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 2.0
lr_scheduler_type: cosine
warmup_ratio: 0.05
bf16: true

You run it with a command like llamafactory-cli train config.yaml. The same command pattern handles chat, export and evaluation with different config files.

The dataset registry

Datasets are described in a small JSON file. The idea is to map your column names to roles.

{
  "table_qa_demo": {
    "file_name": "table_qa_demo.json",
    "columns": {
      "prompt": "instruction",
      "query": "input",
      "response": "output"
    }
  }
}

A record in the data file might look like this.

{
  "instruction": "Write a SQL query for the question below.",
  "input": "Schema: orders(id, customer_id, total, created_at). Question: total sales per customer.",
  "output": "SELECT customer_id, SUM(total) AS total_sales FROM orders GROUP BY customer_id;"
}

The web board

The web interface is a form-based view of the same options. You choose the language, model, method, dataset and hyperparameters, then press start. A preview panel can show the command it will run. After training you can load the adapter into the chat tab, type a question, and see the answer. The board is a good way to learn what each option means, since you can see how the options map to the config.

Distributed and memory options

For larger models, the toolkit can pass settings to sharding backends. You can enable DeepSpeed ZeRO with a config file, or use FSDP through the launcher integration. For low memory, you can turn on quantization and gradient checkpointing. Mixed precision options such as bf16 use the ideas described under mixed precision.

Key concepts and terms

A worked example

Goal: build a small assistant that turns a plain question about a sales table into a SQL query. We use placeholder names and a tiny dataset to show the workflow.

Step 1: prepare data

Create a JSON file with a few hundred records in the shape shown above. Make sure each schema line appears in the input and that the SQL is valid. Split off 10 percent as a held-out set.

import json, random

records = json.load(open("data/table_qa_all.json"))
random.seed(7)
random.shuffle(records)
cut = int(len(records) * 0.9)
json.dump(records[:cut], open("data/table_qa_demo.json", "w"), indent=2)
json.dump(records[cut:], open("data/table_qa_eval.json", "w"), indent=2)
print(len(records[:cut]), "train,", len(records[cut:]), "eval")

Step 2: register the dataset

Add the entry to the dataset registry as shown earlier, using the name table_qa_demo.

Step 3: train

llamafactory-cli train configs/table_qa_lora.yaml

Watch the loss. For a small dataset with two epochs, you should see it fall in the first hundred steps and then flatten.

Step 4: check results by execution

A good SQL test is to run the generated query. Here is a simple check using an in-memory database.

import sqlite3

def run(sql, conn):
    try:
        return conn.execute(sql).fetchall()
    except Exception as e:
        return ("error", str(e))

conn = sqlite3.connect(":memory:")
conn.executescript("""
CREATE TABLE orders(id INTEGER, customer_id INTEGER, total REAL, created_at TEXT);
INSERT INTO orders VALUES (1, 10, 25.0, '2025-01-02'), (2, 10, 15.0, '2025-01-03'), (3, 11, 40.0, '2025-01-04');
""")

gold = "SELECT customer_id, SUM(total) FROM orders GROUP BY customer_id ORDER BY customer_id;"
pred = "SELECT customer_id, SUM(total) AS s FROM orders GROUP BY customer_id ORDER BY customer_id;"

print(run(gold, conn) == run(pred, conn))  # True: same result set

Comparing result sets, not query text, is fairer. Two different queries can be equally correct.

Step 5: export

Create an export config that points to the base model and the adapter directory, then run the export command. The result is a standalone model folder that can be loaded by common serving engines such as vLLM, or converted to GGUF for llama.cpp.

Step 6: compare methods

A neat feature of a unified tool is that the same data can be used with different methods. A simple comparison plan:

Keep the data, seed and evaluation identical so the comparison is fair.

Step 7: add a preference stage

After supervised tuning, you may still see queries that run but give wrong answers. Preference tuning can push the model away from common errors. Build pairs from real model outputs: the chosen answer is a query whose result matches the gold result, and the rejected answer is a query that fails or returns a wrong result.

{
  "instruction": "Write a SQL query for the question below.",
  "input": "Schema: orders(id, customer_id, total, created_at). Question: orders placed in January 2025.",
  "chosen": "SELECT * FROM orders WHERE created_at >= '2025-01-01' AND created_at < '2025-02-01';",
  "rejected": "SELECT * FROM orders WHERE created_at = '2025-01';"
}

This pairing is cheap to produce automatically, because you can run both queries and let the database decide which is right. Keep the number of pairs modest at first and check that the model does not become overly cautious or lose fluency.

Reading the training log

A small project plan

Short cycles beat long ones. Each cycle should end with a number you can compare to the last one.

Handling several languages and domains

If your users write in more than one language, include examples in each of them, and evaluate each separately. A model can look strong on average while failing on one language. The same applies to domains: a model tuned on retail tables may stumble on logistics tables. Tag each example with its language and domain so you can slice the results.

Working with long schemas

Real databases can have hundreds of columns. Putting the entire schema in every prompt wastes tokens and can confuse the model. Two common fixes are to select only the relevant tables with a retrieval step, and to describe columns in a compact form. See hybrid search and naive RAG for retrieval ideas. When training, make the prompt look like the prompt you will use at serving time, including the retrieval step output.

Strengths and limitations

Strengths

• One interface for many models. Less script rewriting when you switch families.
• Low barrier to entry. The web board helps newcomers.
• Broad method coverage. Supervised, preference and reinforcement stages in one place.
• Config-driven. YAML files are easy to version and share.
• Good for quick trials. You can test a base model and a method in an afternoon.
• Integrations. Works with common acceleration and sharding tools.

Limitations

• Abstraction cost. Wrapping many libraries makes some failures harder to trace.
• Moving target. Option names and supported models change between versions. Pin the version you use.
• Not the fastest per GPU. Specialized libraries such as Unsloth can be faster on a single card for LoRA work.
• Less control than custom code. If you need a novel loss or an unusual data pipeline, you may outgrow it.
• Data quality is still on you. The toolkit does not clean or validate your data.
• Evaluation is basic. For serious evaluation, add a harness such as lm-evaluation-harness.

How it compares with alternatives

A common path is to begin in LLaMA-Factory to find a working recipe, then move to a code-first library when you need finer control, or to a sharding setup when you scale up.

Role in tabular foundation models

Tabular foundation models learn directly from raw tables. They must handle row and column structure, a mix of numeric, categorical, date and text columns, missing values, and headers that may be missing or meaningless. Models such as TabPFN are pre-trained on large numbers of synthetic tables, then apply in-context learning at inference time: you give the labeled rows as context and the model predicts for new rows without gradient updates. Others, like TabICL and TabSTAR, explore related designs, including text-aware encoders.

LLaMA-Factory does not train these models. Its home is the language model world. But it plays several real roles around them.

Language front ends for table models

A user rarely wants to craft tensors. They want to ask a question. A tuned language model can read a question and a schema, choose a task, and write the call that runs a tabular model or a SQL query. LLaMA-Factory makes it easy to tune such a front end on pairs of schema, question and tool call. See function calling and structured outputs for the patterns.

Semantic understanding of columns

When headers are cryptic, such as c_17 or amt_x, a model that understands text can guess their meaning from sample values. Tuning a language model to label columns with semantic types supports sector or domain classification, unit detection and sensitive-field flagging. This complements semantic sector heads built into table models. You can also use such labels as extra features or as metadata to improve a tabular model's input.

Synthetic table design

Pre-training data for tabular models often comes from synthetic generators. A language model tuned to propose schemas, plausible column names and realistic value ranges can enrich the variety of generated tables. See synthetic data generation. The language model does not produce the final numbers. It sets up the structure that a statistical generator fills in.

Imputation and anomaly explanations

For missing values in numeric or categorical cells, a tabular foundation model that conditions on all observed cells is the better tool. For text cells, a tuned language model can propose fills. For anomalies, a tabular detector flags rows, and a tuned language model can write a short reason that a human can read. The tools imputation and anomaly detection cover the core methods, and masked cell modeling shows how table models learn to fill gaps.

Continual learning

Teams often want an assistant that adds new skills without losing old ones. With adapters, you can train a new LoRA for each new domain and keep the earlier ones intact, which sidesteps much of the forgetting that methods like EWC and LwF address. LLaMA-Factory makes it simple to train, name and export several adapters from the same base.

Where it is not the right tool

For pre-training a transformer on millions of tables, you need a training stack built for scale, such as PyTorch with FSDP, Megatron-LM or Colossal-AI. For background on how such models are built and measured, see the architecture page, the research page and the benchmarks page.

Practical guidance and best practices

Before you start

• Define the task and metric in one sentence.
• Pick the smallest model that could plausibly work.
• Check the model license for your use.
• Build a held-out set first, then the training set.

Data preparation

• Keep the format consistent. A mix of styles in one dataset confuses small models.
• Include the schema in every prompt for text-to-SQL tasks.
• Use realistic, varied questions. Avoid repeating the same pattern thousands of times.
• Remove duplicates, see deduplication.
• Remove or mask personal data before training.

Configuration

Run hygiene

• Save every config file with the run output.
• Use a fixed seed and record versions.
• Log to a tracker such as MLflow or Weights & Biases.
• Compare to a base model baseline on the same held-out set.

Moving to production

• Merge and export, then test the exported model, not only the adapter.
• Serve with an engine built for throughput, such as vLLM or SGLang.
• Add guardrails for outputs that touch real systems. See guardrails and prompt injection defense.
• Execute generated SQL in a read-only, sandboxed account. See sandboxing.

Security notes

A model that writes SQL is a path from text to your data. Treat it like any other untrusted input source. Use a read-only database role, set row and time limits, allow only a safe subset of statements, and log every query. Never place secrets in training data or prompts. Test with hostile questions that try to read other users' rows, and review the results. The OWASP guidance on injection issues is a good starting point.

Team workflow tips

• Keep configs, the dataset registry and evaluation scripts in one repository.
• Review data changes the way you review code.
• Name runs by what they test, not by date alone.
• Share a short results table after every batch of runs, with the baseline in the first row.
• Agree on a stop rule in advance: for example, stop when two variants in a row fail to beat the best score by a meaningful margin.

Checklist before calling a run done

• The held-out score beats the base model baseline by a margin you care about.
• The result holds on a second, fresh set of examples.
• The exported model gives the same answers as the adapter on ten spot checks.
• The prompt format used in serving matches the training template.
• Errors were reviewed by a person, and the top error types are written down.
• Config, data version, seed and library versions are saved with the model.
• Generated queries run only under a restricted account.
• The license of the base model allows your intended use.

Cost control

Training cost is mostly GPU hours. Reduce it by trimming the cutoff length, packing short examples, using a smaller base model, and stopping early when evaluation stops improving. Use spot or preemptible instances only if your runs save checkpoints often and can resume. Measure cost per point of metric gain, not only total cost, so you know when extra spending stops paying off.

Reading model cards

Before choosing a base model, read its card. Look for the intended use, the training data description in general terms, the context length, the license, the recommended chat format, and known limits. A model that is strong at code and structured output will usually start closer to text-to-SQL than a model tuned mainly for casual chat.

Keeping your toolkit current

The field moves fast. New models arrive, and option names change. Subscribe to release notes, test upgrades in a separate environment, and re-run your fixed evaluation set after any upgrade. If scores shift without any change to your data, the toolkit or a dependency changed behavior, and you should find out why before deploying.

Sharing results

When you report results, include the dataset size, the split method, the metric definition, the baseline and the number of runs. A single run can mislead, so repeat the best setting with two other seeds when the margin is small.

Common mistakes

1. Using the wrong template. The model sees a format it was not trained on, and quality drops.
2. Trusting the web board defaults. Defaults are generic. Check learning rate, cutoff and epochs.
3. Not holding out data. Without a test set you cannot see overfitting.
4. Comparing runs with different data or seeds. Changes in results then mean little.
5. Training too long on a small set. The model memorizes answers.
6. Ignoring the cutoff length. Long schemas get cut and answers lose context.
7. Letting version drift break reproducibility. Pin the toolkit and its dependencies.
8. Judging SQL by text match. Use execution-based checks.
9. Expecting facts to be learned. Use lookups for changing facts.
10. Skipping safety checks on generated queries. Never run generated SQL with write access.

Frequently asked questions

What is LLaMA-Factory?

LLaMA-Factory is an open-source toolkit that provides one interface for fine-tuning many language models. It offers a command line driven by YAML files and a web board for point-and-click use. It supports several training stages and tuning types, including LoRA, QLoRA and full tuning.

Do I need to know how to code to use it?

Basic use through the web board needs little code, mainly preparing a dataset file in the right format. For repeatable work, most people move to config files and the command line. Understanding data formats and evaluation matters more than writing training code.

Which methods does it support?

It covers supervised fine-tuning, continued pre-training, reward modeling and preference-based methods, and it can run full, partial or adapter-based tuning. The exact list changes with versions, so check the current documentation before you plan a project.

How is it different from Axolotl or Unsloth?

All three help with fine-tuning. LLaMA-Factory stresses a unified interface with a web UI over many models. Axolotl stresses YAML recipes for repeatable runs. Unsloth stresses speed and memory savings on a single GPU through optimized kernels. They can be used at different stages of one project.

Can it train on several GPUs?

Yes, it can use distributed launchers and sharding backends such as DeepSpeed ZeRO or FSDP. You still need to understand the memory trade-offs of those tools, and you should test a small run before launching a large one.

Can it train a tabular foundation model?

Not directly. Tabular foundation models are pre-trained from scratch on synthetic or real tables with architectures that read rows and columns. LLaMA-Factory is for language models. It is useful for building language front ends and column-understanding helpers around table models.

How do I evaluate a tuned model?

Hold out examples, define a task metric, and compare against the base model with a good prompt. For SQL, compare execution results. For open-ended text, use human review or a judge model, and watch for regressions on general tasks.

Can I export the result for local use?

Yes. You can merge the adapter into the base model and save a standalone folder, which other tools can convert to quantized formats such as GGUF. Always test the exported file in the runtime where it will be used.

Is the web interface suitable for production?

It is best seen as a lab tool for learning and experiments. For production training, use config files, version control, a tracker and a scripted pipeline so that runs are repeatable and reviewable.

What are good first projects?

Good first projects are narrow and easy to measure: text-to-SQL for a small schema, column semantic type labeling, short table summaries, or a format-conversion assistant. These have clear right answers and let you see whether tuning helps.

Glossary

• Adapter: Small trainable weights added to a frozen model.
• Base model: The pretrained starting point.
• Cutoff length: The maximum token count kept per example.
• DPO: Direct preference optimization, a way to learn from preference pairs.
• Freeze tuning: Training only a chosen subset of layers.
• LoRA: Low-rank adaptation of weight matrices.
• Packing: Joining short examples into long sequences.
• PEFT: Parameter-efficient fine-tuning.
• Preference pair: A better and a worse answer to the same prompt.
• QLoRA: LoRA on a 4-bit quantized base.
• Registry: A file that describes datasets and their fields.
• SFT: Supervised fine-tuning.
• Template: The text format of a conversation for a given model.
• Web board: The browser interface for configuring and running jobs.

Summary

LLaMA-Factory is a unified fine-tuning toolkit with a command line and a web interface. It lets you try many base models and many tuning methods through one set of options, which makes it a strong choice for quick experiments, teaching and early product work. Its costs are abstraction and version drift, so pin versions and keep configs under version control.

For tabular work, it is a supporting tool. Tabular foundation models are trained elsewhere, on synthetic or real tables, with architectures built for rows, columns, mixed types and missing values. LLaMA-Factory helps build the language layer around them: query writers, schema readers, column labelers and explainers. Start narrow, measure by execution or by a held-out set, keep a baseline, and move to code-first or sharded tools when your needs grow.

Related

It belongs to Training tooling in my stack, under LLMs, retrieval, and evaluation. Others in the same group: