SFT

Overview

SFT, short for supervised fine-tuning, is the step that teaches a pre-trained model how to behave by showing it examples of good behavior. You collect prompts, pair each one with an ideal answer, and train the model to reproduce those answers. After this step a raw base model that only continues text becomes a model that follows instructions, answers in a chosen style, and performs a specific task.

SFT is usually the first stage of post-training. It is cheap compared with pre-training, it needs far less data, and it gives large gains in usefulness. It is often followed by preference tuning such as RLHF, DPO or KTO, which refine the model further. This article explains how SFT works, what a good dataset looks like, how to run it, and how the same idea applies to models that learn from raw tables.

What it is

Supervised fine-tuning means continuing the training of a model with labeled examples. "Supervised" means every example has a target: the correct output for a given input. In the language model case, the input is a prompt and the target is a response. The model is trained to make the target response more likely.

A helpful picture is an apprentice chef. A base model is like someone who has read every cookbook in the world but has never worked in a kitchen. They know a lot of words about food, but if you say "make me a soup" they might continue with a paragraph about soup history. SFT is the apprenticeship. The head chef shows a few thousand finished dishes next to the orders that asked for them. The apprentice learns what a good response to an order looks like.

What SFT changes and what it does not

SFT mostly changes the format and the style of behavior. It teaches the model:

• To treat the input as an instruction and not as text to continue.
• To answer in a structure, such as a short paragraph, a list, a JSON object or a code block.
• To use a tone, such as formal, concise or friendly.
• To perform a narrow task, such as classification, extraction or summarization.

SFT is weaker at adding large amounts of new knowledge. Facts and skills come mostly from pre-training. If a model has never seen a domain, a small SFT set will not fill the gap. That job belongs to continued pre-training. This split of labor, knowledge from pre-training and behavior from SFT, is one of the most useful rules of thumb in the field.

Variants of SFT

Where it came from

The idea of fine-tuning a pre-trained network on a labeled task is older than modern language models. In computer vision, practitioners took networks trained on large image collections and fine-tuned them on small datasets. That practice is a form of transfer learning.

In language, the recipe took shape in the late 2010s. Pre-trained transformers such as GPT and BERT were fine-tuned on classification and question answering sets. Each task got its own copy of the model and its own labeled data. The next shift was to fine-tune one model on many tasks written as natural language instructions. Work on multitask prompted training and on instruction tuning, around 2021 and 2022, showed that a model tuned this way could follow instructions for tasks it had not seen during tuning.

The InstructGPT work in 2022 combined SFT on human-written demonstrations with reinforcement learning from human feedback. That three-stage pipeline became the standard template for assistants: pre-train, then SFT, then preference optimization. Later work showed that a small, carefully curated SFT set can be enough to get strong chat behavior, which pushed the field toward data quality over data volume. Open tooling such as Hugging Face, TRL, Axolotl, LLaMA-Factory, torchtune and Unsloth then made SFT easy to run on a single machine.

How it works

SFT uses the same next-token prediction objective as pre-training. What changes is the data and the way the loss is computed. The steps below show the full flow.

Step 1: Collect or write the examples

Each example is a prompt and a target response. For multi-turn chat, it is a list of messages with roles such as system, user and assistant. Sources include human-written answers, curated public sets, outputs of a stronger model that are then filtered, and logs from real use after review.

The analogy is a driving school. The instructor does not describe every rule of the road. They show good drives in many conditions. The quality of the drives matters more than the count.

Step 2: Apply a chat template

Models expect text in a specific layout, with special tokens that mark where the system message, the user message and the assistant message begin and end. This layout is the chat template. During SFT you must format training text with exactly the same template that will be used at inference time. If the template differs, the model sees a format at test time that it never practiced.

Step 3: Tokenize

The text is split into tokens by the model's own tokenizer, see BPE and SentencePiece. Never swap in a different tokenizer. The special tokens in the template must map to the ids the model already knows.

Step 4: Mask the loss

This step is easy to overlook. In pre-training, the model learns from every token. In SFT, you usually want the model to learn only from the response tokens, not from the prompt. So you set the label of prompt tokens to an ignore value, and the loss is computed only on the assistant part.

Think of a student copying model essays. You want them to practice writing the answers, not memorize the exam questions. Masking the prompt keeps the model focused on producing answers.

Step 5: Pack or batch sequences

Examples have different lengths. Padding wastes compute. Packing joins several short examples into one long sequence, with attention masks that stop them from seeing each other. Both approaches work. Packing is faster, but you must get the attention boundaries right.

Step 6: Train with a small learning rate

SFT starts from a model that is already good. A large learning rate can wipe out its prior knowledge. Typical learning rates are much lower than in pre-training, and training runs for a small number of epochs, often one to three. A warmup and a cosine or linear decay are common. See mixed precision and gradient checkpointing for memory savers.

Step 7: Evaluate

You hold out a validation set and watch the loss. Loss alone is not enough. You also run task metrics, spot checks by humans, and automated judges, see LLM-as-judge and evals. Over-training makes the model rigid and can reduce general ability.

Step 8: Save and serve

You save either the full weights or a small adapter. Then you serve the model with a runtime such as vLLM or SGLang, or convert it to a compact format such as GGUF.

The loss in one line

For a response with tokens y1 to yT given a prompt x, the SFT loss is the average negative log probability of each response token given everything before it:

L = - (1/T) * sum over t of log p(y_t | x, y_1 ... y_{t-1})

Lower loss means the model assigns higher probability to the reference answers.

Key concepts and terms

A worked example

This example uses Hugging Face TRL to run SFT on a small instruction dataset. Names and paths are placeholders.

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

# Each row has a "messages" list: [{"role": "user", ...}, {"role": "assistant", ...}]
dataset = load_dataset("json", data_files="data/support_chats.jsonl", split="train")
split = dataset.train_test_split(test_size=0.05, seed=7)

config = SFTConfig(
    output_dir="out/sft-support",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=1e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    max_length=2048,
    packing=False,
)

trainer = SFTTrainer(
    model="example-org/base-model-7b",
    args=config,
    train_dataset=split["train"],
    eval_dataset=split["test"],
)
trainer.train()
trainer.save_model("out/sft-support/final")

What one training row looks like

{
  "messages": [
    {"role": "system", "content": "You are a concise support assistant."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Open Settings, choose Security, then select Reset password. A link will be sent to your email."}
  ]
}

A small numeric walk-through of the loss

Suppose the target answer is "Yes, it is" and the model gives these probabilities to the correct next token at each step: 0.5 for "Yes", 0.8 for ",", 0.4 for "it", 0.9 for "is". The loss is the average of the negative natural logs.

The sum is about 1.937 and the average over four tokens is about 0.484. After training, the same tokens might get probabilities of 0.9, 0.95, 0.8 and 0.95. The average loss would then fall to roughly 0.1. The model has become more confident in the reference answer. If the prompt tokens were not masked, the loss would also include the cost of predicting the user question, which is not what you want to optimize.

Checking the result

After training, test on prompts that were not in the data. Check three things. Does the model follow the format? Does it stay accurate on facts it knew before? Does it still handle general prompts, not just the narrow task? A model that does well on the first and poorly on the other two has been over-tuned.

A second worked example: building the dataset

The code above assumes a ready dataset. In practice, most effort goes into building it. This section walks through a realistic data pipeline for a table-related instruction set.

Step A: Define the task and the output format

Write down, in one paragraph, what the model must do and what the output must look like. For example: "Given one row of a table, return the most likely sector as a single lowercase label from a fixed list." A precise definition tells you what to collect and how to check it.

Step B: Draft a seed set by hand

Write 50 to 200 examples yourself or with domain experts. This forces you to find the hard cases early. Include easy, medium and hard rows, and rows where the right answer is "unknown".

Step C: Expand with a stronger model, then filter

A stronger model can draft more examples from your seeds. Treat these as candidates, not as truth. Filter them with rules (format checks, label in allowed list), with a second model acting as a reviewer, and with random human spot checks.

import json, random

ALLOWED = {"retail", "energy", "health", "finance", "unknown"}

def valid(example):
    msgs = example["messages"]
    if [m["role"] for m in msgs][-2:] != ["user", "assistant"]:
        return False
    answer = msgs[-1]["content"].strip().lower()
    return answer in ALLOWED

with open("data/candidates.jsonl") as f:
    rows = [json.loads(line) for line in f]

clean = [r for r in rows if valid(r)]
random.Random(7).shuffle(clean)
print(len(rows), "candidates,", len(clean), "kept")

Step D: Balance the classes

If 80 percent of the answers are one label, the model will learn to say that label. Down-sample the common classes or add more rare ones. Check the final label counts in a small table before training.

Step E: Split by group

If several rows come from the same source table, keep all of them on one side of the split. A model that has seen a table's other rows can guess the label of a row from the same table, so a random row split overstates quality.

Step F: Version and document

Give the dataset a version id, store the filtering rules, and record the counts. When quality changes later, you will know whether the data or the training caused it.

Choosing hyperparameters

The values below are common starting points, not rules. Always tune on a validation set.

Reading the loss curve

A healthy run shows a fast drop in the first few dozen steps and then a slow decline. A flat curve from the start can mean the learning rate is too low or the labels are masked wrongly. A curve that falls to near zero in training while validation rises means memorization. A loss that spikes or becomes not-a-number often means the learning rate is too high or mixed precision settings need a check.

Data formats and templates

Several formats are common. Pick one and be consistent.

• Prompt and completion. Two fields. Simple, good for single-turn tasks.
• Messages. A list of role and content items. Standard for chat and tool use.
• Instruction, input, output. Three fields, popular in early public sets.

When a dataset uses roles, the trainer applies the model's chat template to build the final text. Print one fully formatted example and read it before every training run. Look for stray whitespace, missing end tokens and doubled special tokens.

Multi-turn conversations

For a conversation with several assistant turns, there are two choices. You can train on only the last assistant reply, or on every assistant reply in the thread. The second gives more signal per example, but it needs correct masking for each turn.

Tool-use examples

For agents, include the tool call, the tool result and the final answer. The model must learn when to call a tool, how to write valid arguments, and how to use the result. See function calling and structured outputs.

Quality control for demonstrations

Good demonstrations share traits. They are correct, direct, complete for the question, and written in the target voice. A short checklist helps reviewers.

1. Is the answer factually right, as far as the reviewer can verify?
2. Does it answer exactly the question that was asked?
3. Is the format the one that the product needs?
4. Does it avoid claims the model cannot support?
5. Is the tone consistent with other examples?
6. Would a domain expert accept it without edits?

Disagreement between reviewers is a signal. If two experts give different answers to the same row, the task is ambiguous, and the model will learn a blurry rule. Resolve the ambiguity in the task definition first.

Strengths and limitations

Strengths

• Simple. It reuses the pre-training objective and a standard training loop.
• Data efficient. Hundreds to tens of thousands of good examples can change behavior a lot.
• Predictable. The target is explicit, so you can reason about what the model will learn.
• Composable. It works with adapters, quantization and distillation.
• A strong base for later stages such as preference tuning and reinforcement learning, see PPO and reinforcement learning.
• Fast to iterate. A small run can finish in hours on one accelerator.

Limitations

• It copies the data. If the examples are wrong, biased or inconsistent, so is the model.
• It teaches imitation, not judgment. The model learns what a good answer looks like, not why. It cannot tell which of two valid answers a user would prefer unless you add preference data.
• It adds little new knowledge. Facts must come from pre-training or retrieval, see naive RAG and advanced RAG.
• It can encourage confident guessing. If the data has no examples that say "I do not know", the model may always answer.
• It can cause forgetting if run too long or at too high a learning rate.
• Quality is capped by the data author. A model trained on weaker answers will not exceed them unless other methods are used.

How it compares with alternatives

A good rule: try prompting first, add retrieval for facts, use SFT for behavior, use continued pre-training for deep domain language, and use preference tuning to polish.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns. Columns can hold numbers, categories, text, dates and missing values, and column names may be absent or meaningless. Models in this area, such as TabPFN, TabICL, TabSTAR and XTab, are pre-trained on many tables, often synthetic ones generated from random causal structures, and then used with in-context learning. SFT has several concrete roles here. See also /model/research and /model/architecture.

SFT where the model is a language model

When a language model reads tables serialized as text, SFT teaches it the task. Typical examples are: a table fragment and a question as the prompt, and the answer as the target. Other examples ask the model to return a column type, fill a missing cell, flag an anomalous row or map a column to a semantic sector. The prompt holds the row values and the target holds the label. This makes the model follow table-aware instructions without a new architecture.

SFT as a supervised stage for table-native models

A table-native model does not read text. It reads cells as numbers and categories and uses row and column attention, see axial attention and FT-Transformer. Its pre-training on synthetic tables is already supervised in a sense: each synthetic table comes with features and a target, and the model is trained to predict held-out targets from context rows. The same supervised signal can be reused later as a fine-tuning stage on real tables. That stage is SFT in spirit, with the table as the prompt and the target column as the answer.

Tasks where supervised tuning helps

For imputation, you can create supervised data for free. Take a complete table, hide some cells at random, and train the model to recover them. This is close to masked cell modeling. The mask pattern should match real missingness, whether values are missing at random or for a reason that depends on the value.

No column names

Many real tables have headers like "col1", "x7" or no header at all. If you train with names always present, the model may lean on them. A good SFT set includes cases with names removed, shuffled or replaced with neutral ids, so the model learns to use the values and their statistics. This also protects against leakage from a revealing name.

Continual learning and forgetting

Fine-tuning on one sector can harm another. A model tuned on retail tables may get worse on industrial sensor tables. Remedies include mixing a replay set of earlier data into SFT, using small adapters per sector, and regularizing weights as in EWC, synaptic intelligence and LwF. SFT with a replay mix is the simplest of these.

Where SFT is not the right tool

A table-native model that already does in-context learning often needs no weight update. You pass labeled rows as context and it predicts for new rows. Use SFT only when the gain over in-context use is clear and measured, for example for a fixed schema that appears again and again with a large labeled set. See /model/benchmarks for how such comparisons are framed.

Practical guidance and best practices

Data

• Prefer quality over quantity. A few thousand clean, diverse examples often beat a large noisy set.
• Cover the real range of inputs, including short, long, messy and adversarial ones.
• Include examples where the right behavior is to refuse, ask a question or say the answer is unknown.
• Keep style consistent. Mixed tones in the answers produce a mixed tone in the model.
• Deduplicate and remove near copies, see deduplication. Duplicates cause memorization and leak into validation.
• Split by group, not by row, when rows share an entity. Otherwise the validation score is too optimistic.
• Keep a private test set that is never used for tuning.

Training

• Start from the instruction or base model that best matches your need.
• Use a low learning rate, a short warmup, and one to three epochs.
• Use the exact chat template that you will use in production.
• Mask prompt tokens unless you have a reason not to.
• Watch both training and validation loss. A rising validation loss with a falling training loss means overfitting.
• Save checkpoints and compare them, the last one is not always best.
• Use LoRA or QLoRA when memory is tight, and full updates when you have the budget, see DeepSpeed ZeRO and FSDP for scaling.

Evaluation

• Build a fixed evaluation set before training.
• Track a general benchmark in addition to your task, so forgetting is visible, see lm-evaluation-harness.
• Review samples by hand. Aggregate numbers hide failure patterns.
• Use an automated judge as a helper, and check it against human labels.

Deployment

• Version the data, the code, the config and the weights together, see MLflow.
• Add guardrails for output format and safety, see guardrails.
• Monitor live outputs and feed reviewed cases back into the next data version.

Compute and memory planning

Memory use in full SFT has four parts: the weights, the gradients, the optimizer states and the activations. With a common adaptive optimizer, the optimizer states alone can take twice the size of the weights in extra memory. This is why parameter-efficient methods are popular. They train a tiny share of parameters, so gradients and optimizer states shrink to almost nothing.

A staged plan for a new project

1. Run the base model with a good prompt and record the failures.
2. Build a small seed set and an evaluation set from those failures.
3. Run a quick LoRA SFT on the seed set to see if the idea works at all.
4. Grow the data where the model still fails, and rerun.
5. Compare against a prompt-only baseline and a few-shot baseline.
6. If the gap is worth the cost, try full SFT or a larger base model.
7. Add preference tuning only after SFT is stable.

Safety and bias checks

Demonstrations can carry hidden bias or unsafe content. Scan the set for personal data, see GDPR and data tokenization, and for harmful instructions. Test the tuned model with adversarial prompts, see red teaming and prompt injection defense. Tuning can weaken safety behavior that the base model had, so repeat safety tests after every tuning run.

Common mistakes

• Wrong chat template. Training with one layout and serving with another is the most common cause of odd behavior.
• No loss masking. The model spends capacity on predicting prompts and can drift toward echoing them.
• Learning rate too high. The model forgets general ability and becomes brittle.
• Too many epochs. The model memorizes the set and repeats training phrases.
• Contaminated validation. Near duplicates across splits give a false sense of quality.
• Expecting new knowledge. SFT on a few hundred facts will not make a model reliably recall them. Use retrieval or continued pre-training.
• Teaching hallucination. Answers in the data that cite facts the model cannot know teach it to invent facts.
• Ignoring the end-of-sequence token. If it is missing from targets, the model never learns to stop.
• Only positive examples. Without refusals or unknown cases, the model will always try to answer.
• Judging by loss only. A lower loss does not guarantee better answers. Check outputs.
• Leaky tabular prompts. Putting the target or a near copy of it into a serialized row makes the task trivial and the test misleading.
• Mismatched missingness. Training imputation on random masks and deploying on structured gaps gives poor results.

Frequently asked questions

What does SFT stand for?

SFT stands for supervised fine-tuning. It means continuing the training of a pre-trained model on labeled examples, where each example pairs an input with the desired output, so the model learns to produce that kind of output.

How is SFT different from pre-training?

Pre-training uses huge amounts of raw, mostly unlabeled data to teach general knowledge and language patterns. SFT uses a much smaller, curated set of prompt and answer pairs to teach behavior, format and task skills. Both use next-token prediction, but SFT usually computes the loss only on the answer tokens and uses a much lower learning rate.

How much data do I need for SFT?

It depends on the goal. A narrow format or style change can work with a few hundred clean examples. A broad assistant needs tens of thousands of varied examples. In practice, quality, diversity and consistency matter more than raw count, so start small, measure, and add data where the model fails.

Does SFT teach a model new facts?

Only to a limited degree. SFT is good at shaping how a model answers, not at storing large amounts of new knowledge. For new domain knowledge, use continued pre-training on domain text or add retrieval so the model can look facts up at answer time.

What is the difference between SFT and RLHF?

SFT trains the model to imitate given answers. RLHF trains it to maximize a learned reward that reflects human preferences, which lets it choose between several valid answers. SFT usually comes first and gives a good starting point, and preference methods such as RLHF or DPO then refine the model.

Should I use full fine-tuning or LoRA for SFT?

Use full fine-tuning when you have enough memory, plenty of data and want maximum adaptation. Use LoRA or QLoRA when memory is limited, when you need many task-specific variants, or when you want to keep the base weights untouched. For many tasks the quality gap is small, so test both on your own evaluation set.

Why do I need to mask the prompt tokens?

Masking tells the trainer to ignore the prompt when computing the loss, so the model learns to write responses and not to memorize questions. Without masking, part of the training signal is spent on text the model will never need to generate, which can slightly hurt quality.

Can SFT be used for tabular data?

Yes. With language models, you serialize rows as text and train on prompt and answer pairs for tasks such as classification, imputation or anomaly flags. With table-native models, a supervised stage on real tables with hidden cells or labeled targets plays the same role. In both cases you should test whether in-context learning alone is already enough.

How do I avoid catastrophic forgetting during SFT?

Use a low learning rate, train for few epochs, mix some earlier or general data into the training set, and consider adapters or weight regularization. Also track a general benchmark during training so you can see forgetting early and stop or roll back.

How do I know when SFT is finished?

Stop when validation loss stops improving and your task metrics plateau, and when manual review shows the outputs meet your bar. Compare several checkpoints, because the best one is often earlier than the last. If problems remain after that, the cause is usually the data and not the number of steps.

Glossary

• Adapter. A small trainable module added to a frozen model.
• Base model. A pre-trained model that has not been tuned to follow instructions.
• Batch size. The number of examples used for one weight update.
• Catastrophic forgetting. Loss of earlier abilities after training on new data.
• Chat template. The text layout with role markers used for conversations.
• Checkpoint. A saved copy of model weights during training.
• Demonstration. An example of a prompt and a good response.
• Distillation. Training a model on outputs of a stronger model.
• Epoch. One full pass over the training data.
• Gradient accumulation. Summing gradients over several small batches before an update.
• Instruction tuning. SFT on many tasks written as instructions.
• Learning rate. The size of each weight update step.
• Loss masking. Excluding prompt tokens from the loss.
• Overfitting. Fitting the training data too closely and generalizing poorly.
• Packing. Placing several short examples in one sequence.
• Post-training. Training stages after pre-training, such as SFT and preference tuning.
• Rejection sampling. Keeping only the best of several generated answers.
• Token. A unit of text or a special marker that the model reads.
• Validation set. Held-out data used to monitor training.

Summary

SFT, or supervised fine-tuning, trains a pre-trained model on prompt and answer pairs so it follows instructions, uses a chosen format and performs a task. It reuses next-token prediction, masks the prompt, uses a low learning rate and a few epochs, and depends far more on clean, diverse data than on volume. It shapes behavior more than knowledge, so pair it with continued pre-training or retrieval for facts and with preference tuning for polish.

For models that learn from raw tables, the same idea appears as a supervised stage: tables with hidden cells, labeled anomalies or known sectors serve as the prompt, and the true values or labels serve as the answer. Include cases without column names, match real missingness, mix in replay data to avoid forgetting, and check whether in-context learning already meets the need before changing any weights. For deeper background, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: