Prefix tuning

Overview

Prefix tuning is a parameter-efficient way to adapt a large frozen model. Instead of changing the model weights, you learn a short sequence of continuous vectors, called a prefix, and attach it to the attention keys and values at every layer. The model reads the prefix as if it were extra context. Only the prefix is trained, so the stored file per task is very small.

The method is useful when you want one shared backbone and many tasks, and you want to keep training cost low. It sits in the same family as prompt tuning, adapters, LoRA, IA3 and BitFit, all under the umbrella of PEFT. It works best on generation tasks and mid-size models, and it needs care with training stability. For table-native models, the same idea can steer a pre-trained tabular transformer toward a domain without touching its weights.

What it is

Prefix tuning prepends trainable vectors to the hidden states that attention layers use as keys and values. These vectors are not words. They are free parameters in the model's continuous space. During training the model weights are frozen, and gradients update only the prefix.

A helpful picture: imagine a skilled translator who works from a briefing sheet. You cannot change the translator, but you can hand over a sheet of notes before each job. The notes steer the style and the topic. Prefix tuning learns the best possible notes, written in a code that only the translator understands.

Where the prefix lives

In a transformer, each layer has attention with queries, keys and values. In prefix tuning, each layer gets its own learned pair of extra key vectors and value vectors. Every real token can attend to these extra positions. Because the prefix is present in every layer, it has strong influence on the output.

Virtual tokens

The prefix has a length, for example 10 or 20 positions. These positions are called virtual tokens, because they behave like tokens but do not correspond to any word. The length is the main size knob.

Frozen model, tiny file

Only the prefix is saved. For a model with billions of weights, the prefix may be well under one percent of the size. You keep one copy of the backbone and swap prefixes per task or per user.

Prefix tuning versus nearby ideas

Where it came from

Prefix tuning was proposed by Li and Liang in 2021 in the paper "Prefix-Tuning: Optimizing Continuous Prompts for Generation". The authors were looking for a lightweight alternative to full fine-tuning for text generation tasks such as table-to-text and summarization. They showed that learning a small continuous prefix could reach quality close to full fine-tuning while training a tiny fraction of the parameters, and that it did well in low-data settings.

The work built on earlier ideas:

• Prompting large language models with text, which showed that context can steer behavior.
• Automated search for discrete prompts, which tried to find the best words.
• Adapter methods, which showed that small add-on modules can carry a task.

Soon after, prompt tuning (Lester and colleagues, 2021) simplified the idea by learning soft tokens only at the input layer. The authors reported that the gap to full fine-tuning shrinks as model size grows. Later work on P-tuning v2 showed that deep prompts, applied at every layer in the spirit of prefix tuning, make the approach work better on smaller models and on harder tasks such as sequence labeling.

Prefix tuning is now available in common libraries such as the PEFT package from the Hugging Face ecosystem. For the research context of reusing one backbone, see /model/research.

How it works

Step 1: Take a frozen pre-trained model

You start with a pre-trained transformer. All weights are frozen. The model may be a decoder for text generation, an encoder-decoder, or any attention-based network.

Step 2: Create a trainable prefix

You create a tensor of learned vectors. Its shape is roughly: number of layers, two (for keys and values), prefix length, and hidden size per head and head count. This tensor is the only thing that will be trained.

Step 3: Prepend it to keys and values

In each attention layer, the real keys and values are computed from the input tokens. The prefix keys and values are concatenated in front of them. Queries come from real tokens only.

So when a real token asks "what should I pay attention to," it can look at the real context and at the prefix. The prefix acts like a permanent set of hints available at every depth of the network.

Step 4: Reparameterize for stability

Training the prefix directly can be unstable. The original paper used a small helper network, usually a multilayer perceptron, that maps a smaller matrix into the full prefix. During training, the helper produces the prefix. After training, you can throw the helper away and keep only the final prefix vectors. This trick smooths optimization.

Analogy: instead of writing every note by hand, you teach a small assistant to write the notes. The assistant is only needed while you learn. Once the notes are good, you keep the notes and let the assistant go.

Step 5: Train with the usual loss

You run normal batches through the frozen model with the prefix attached. For language models, the loss is next-token cross entropy. For classification, it is cross entropy on the head output. Gradients flow through the entire frozen network back to the prefix.

Step 6: Save and serve

You save the prefix, and the helper network can be discarded. At serving time, load the shared backbone and attach the prefix for the requested task. Different tasks can share the same GPU memory for the backbone.

What the math looks like

Let the real keys be K and values be V. Let the prefix keys be Pk and prefix values be Pv. Attention then uses:

K' = concat(Pk, K)
V' = concat(Pv, V)
Attention(Q, K', V') = softmax(Q K'^T / sqrt(d)) V'

The prefix does not change the sequence the model sees. It only extends the memory that attention reads from.

The cost of the prefix

The prefix takes sequence positions inside attention. A prefix of length 20 means every token attends to 20 extra positions. This adds a small amount of compute and memory. It also uses part of the available context window, which matters for long inputs.

Choosing the prefix length

• Short prefixes (around 5 to 10) are cheap and may be enough for simple tasks.
• Medium prefixes (around 20 to 50) suit most generation tasks.
• Very long prefixes can overfit and may hurt quality.

Tune the length on validation data, and watch the context budget.

Initialization

The prefix can start from random values, or from the activations of real words. Starting from real word activations often helps, because the model has seen those regions of the space before. For small data, good initialization makes a visible difference.

Variants and relatives

Key concepts and terms

A worked example

This sketch shows the core idea in plain PyTorch: a single attention layer where a learned prefix extends the keys and values. It is a teaching sketch, not a full model.

import torch
import torch.nn as nn
import torch.nn.functional as F


class PrefixAttention(nn.Module):
    def __init__(self, d_model=64, n_heads=4, prefix_len=8):
        super().__init__()
        self.h = n_heads
        self.dh = d_model // n_heads
        self.q = nn.Linear(d_model, d_model)
        self.k = nn.Linear(d_model, d_model)
        self.v = nn.Linear(d_model, d_model)
        self.o = nn.Linear(d_model, d_model)
        # trainable prefix for keys and values
        self.pk = nn.Parameter(torch.randn(prefix_len, d_model) * 0.02)
        self.pv = nn.Parameter(torch.randn(prefix_len, d_model) * 0.02)

    def split(self, x):
        b, t, _ = x.shape
        return x.view(b, t, self.h, self.dh).transpose(1, 2)

    def forward(self, x):
        b = x.size(0)
        q = self.split(self.q(x))
        k = self.k(x)
        v = self.v(x)
        pk = self.pk.unsqueeze(0).expand(b, -1, -1)
        pv = self.pv.unsqueeze(0).expand(b, -1, -1)
        k = self.split(torch.cat([pk, k], dim=1))
        v = self.split(torch.cat([pv, v], dim=1))
        att = F.scaled_dot_product_attention(q, k, v)
        att = att.transpose(1, 2).reshape(b, -1, self.h * self.dh)
        return self.o(att)


layer = PrefixAttention()
for name, p in layer.named_parameters():
    p.requires_grad = name in ("pk", "pv")

x = torch.randn(2, 10, 64)
y = layer(x)
print(y.shape)  # torch.Size([2, 10, 64])
print(sum(p.numel() for p in layer.parameters() if p.requires_grad))

Reading the example

• The real projections q, k, v and o are frozen.
• Only pk and pv receive gradients.
• With prefix length 8 and width 64, the trainable count is 1,024.
• The frozen projections hold about 16,640 weights, so the prefix is a small share.

Numeric walk-through of attention with a prefix

Suppose one query attends to three real positions and two prefix positions. Raw attention scores might be:

1. Prefix 1: 0.9
2. Prefix 2: 0.4
3. Real 1: 1.2
4. Real 2: 0.1
5. Real 3: 0.6

After softmax, the weights are roughly 0.25, 0.15, 0.34, 0.11 and 0.14. The two prefix positions together take about 40 percent of the attention mass. Their value vectors then shape the output. Training moves the prefix values so that this 40 percent pushes the output toward the target behavior.

Using a library

With a library, the pattern is short: load the base model, create a prefix config with the number of virtual tokens, wrap the model, train as usual, and save the small prefix file. Check the library documentation for the exact config names, since these can change between versions. The PEFT package and trainers such as TRL follow this pattern.

# Pseudocode shape of a typical workflow
base = load_pretrained("base-model-name")
cfg = PrefixConfig(num_virtual_tokens=20, prefix_projection=True)
model = wrap_with_peft(base, cfg)
train(model, train_data, lr=3e-3)
model.save_prefix("prefix_task_a")

Strengths and limitations

Strengths

• Tiny per-task storage. A prefix is a small file.
• Base weights stay untouched, so there is no forgetting of pre-training.
• One backbone can serve many tasks and users with different prefixes.
• Works well with little data, in several reports on generation tasks.
• Training memory is low, since optimizer states exist only for the prefix.
• Prefixes can be batched across users in some serving setups.
• Easy to remove: dropping the prefix restores the original model.

Limitations

• The prefix consumes part of the context window.
• Training can be unstable and sensitive to learning rate and initialization.
• Quality can fall behind LoRA and full fine-tuning on harder tasks and on smaller models.
• The prefix is not mergeable into the weights, so some overhead stays at inference.
• It is less interpretable, because prefix vectors do not map to words.
• Performance depends on the base model and may not transfer between models.
• Tooling has moved toward LoRA-style methods in many production stacks.

A note on scaling

Reports on soft prompts show that the gap to full fine-tuning tends to shrink as the base model grows. For small and mid-size models, deep prefixes at every layer usually do better than input-only prompts. Always test on your own model size.

How it compares with alternatives

Reading the comparison

Prefix tuning gives more steering power than input-only prompts, because it touches every layer. It gives up merging and uses context length. If those costs matter, LoRA or IA3 may be a better pick. If you want pure prompt-like behavior that is easy to explain, prompt tuning is simpler.

Prefix tuning and in-context examples

Few-shot prompting places real examples in the input. Prefix tuning learns a compact replacement for those examples. The learned prefix can hold more task information per position than text can, and it does not need to be re-sent as text, though it still occupies attention positions.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types, missing cells and often no column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on large numbers of synthetic or real tables so that they can predict on a new table by reading it as context. Prefix tuning gives a way to steer such a backbone toward a domain. See /model/architecture for how table-native backbones are organized.

A prefix as learned context rows

In-context tabular models predict by attending to labeled example rows. A learned prefix can play the role of extra, synthetic "memory rows" that carry domain knowledge. Instead of storing thousands of real rows in the context, a short prefix at each layer can bias the model's reading of the table toward a domain prior. This is an analogy, and its benefit has to be tested on real data.

Row and column structure

Tabular transformers attend across rows and across columns. A prefix could be attached in either attention path. A column-path prefix can bias how the model weighs features, which is useful when certain columns matter in a domain. A row-path prefix can bias how the model compares records. Because the prefix is separate from the weights, you can try each path and keep the better one.

Mixed types and missing values

Cells may be numbers, categories, dates, text or missing. Pre-trained encoders handle each type. A domain prefix can help the model interpret domain-specific patterns, such as a missing value that signals "not applicable" in one table family and "not measured" in another. The type encoders themselves stay frozen.

No column names

When a table has no useful header, the model depends on value patterns and structure. When names exist, text-aware models such as TabSTAR can use them. A prefix can be trained to compensate for the lack of names within a domain, by encoding typical value ranges and relationships that the header would normally supply.

Pre-training on synthetic and real tables

Synthetic pre-training teaches generic structure: noise, interactions, causal graphs and missingness. Real-table pre-training adds realistic semantics. A prefix is a post-hoc steering tool: it does not replace pre-training, it sits on top of it. A good use is to tune a prefix for each customer or data source while the backbone stays shared. Compare against the plain in-context result, and against tuned tree ensembles such as XGBoost, as shown in the suites summarized at /model/benchmarks.

Continual learning

New domains arrive over time. With prefix tuning, each domain gets its own prefix, and the backbone never changes. This isolates knowledge and avoids the forgetting that regularization methods such as EWC or replay methods such as experience replay try to manage. The cost is a routing decision at serve time: which prefix to attach, or how to combine several.

Imputation and anomaly detection

For imputation, a backbone pre-trained with masked cell modeling predicts hidden cells. A prefix can adapt it to a table family with its own gap patterns, such as sensor streams with long outages. For anomaly detection, a prefix can encode what normal looks like for a system, which sharpens reconstruction or density scores. See also imputation networks.

Sector and semantic classification

Assigning a sector or semantic label to a table or entity requires combining values and names. A prefix plus a semantic head can specialize the backbone for a label set. When the label set changes, retrain the prefix and head, not the model.

Honest caveats

Prefix tuning was designed for text generation. Its use on tabular backbones is an application of the same mechanism, not a guarantee of gain. In-context tabular models already adapt by reading context, so the gain from a prefix is largest when the domain is stable, labeled data is available, and zero-shot quality is not yet enough. Methods like LoRA or adapters are common alternatives for the same job.

Practical guidance and best practices

Start simple

• Try the frozen model with in-context prompts first.
• If that is not enough, try prompt tuning as the cheapest soft method.
• Move to prefix tuning when you need more steering power.
• Compare with LoRA as a strong general baseline.

Hyperparameters to tune

Keep the context budget in mind

A prefix of 30 positions in a model with a short window removes space for real input. Count tokens or rows carefully. For long tables, a shorter prefix or a mergeable method may be a better fit.

Validate properly

On tabular data, use k-fold cross-validation with leakage-safe preprocessing. Fit scalers and encoders inside each fold. Report the mean and spread, not a single run. Repeat with several seeds, because soft-prompt methods can vary between runs.

Track and version

• Save the prefix with the backbone version and tokenizer or encoder version.
• Record the training data snapshot and metrics.
• Use an experiment tracker such as MLflow.
• Test that a prefix refuses to load on the wrong backbone.

Serving

• Keep the backbone resident and swap prefixes per request group.
• Batch by prefix to reduce overhead.
• Measure the added latency from the extra attention positions.
• Cache prefix key and value tensors so they are not recomputed.

Combine with quantization

The frozen backbone can be stored in lower precision using tools such as bitsandbytes, and the prefix can stay in higher precision. Always re-check quality after quantization.

A short pre-flight checklist

• Print the number of trainable parameters. It should equal the prefix size.
• Confirm that no backbone parameter has gradients.
• Check that the loss is finite on the first batch.
• Run a quick overfit test on a tiny set to confirm the prefix can learn.
• Fix the evaluation split before tuning.

Debugging a prefix that does not learn

If the loss stays flat, work through these checks in order:

1. Confirm that the prefix tensor is registered as a parameter and is passed to the optimizer.
2. Confirm that the learning rate is not too small. Try a tenfold increase.
3. Confirm that gradients reach the prefix. Print the gradient norm after one backward pass.
4. Try a shorter or longer prefix to see whether capacity is the issue.
5. Turn on reparameterization if training is noisy.
6. Check that labels and inputs are aligned, since a data bug looks like a method failure.
7. Try initializing from real activations rather than random values.

Debugging a prefix that overfits

• Reduce the prefix length.
• Add dropout on the prefix.
• Use weight decay and early stopping.
• Add more labeled data, or use stronger cross-validation.
• Check for duplicate rows shared between train and validation sets.

Everyday analogy for the length trade-off

A prefix is like a briefing note handed to a worker before every job. A one-line note is quick to read but may miss details. A ten-page note covers everything but slows the worker down and may include confusing detail. The best note is the shortest one that still gets the job done.

Per-user and per-segment prefixes

One practical use is personalization. A shared backbone serves all users, and each user or segment has a small prefix learned from their data. Storage stays low, and the backbone can be updated centrally. Privacy needs care: a prefix is trained on personal data and should be treated as personal data, with access control and deletion on request.

How to decide when it is worth it

Ask these questions:

• Do you have many tasks that must share one backbone?
• Is the backbone too large to fine-tune fully for each task?
• Is the context window long enough to spare some positions?
• Do you accept some training sensitivity?
• Have simpler options, such as in-context examples, fallen short?

If most answers are yes, prefix tuning is a reasonable candidate. If latency or merging matters most, look at LoRA or IA3 first.

When prefix tuning meets quantized or sharded models

Large backbones are often split across devices or stored in low precision. A prefix is small, so it can live on every device. Make sure its dtype matches what the attention layers expect, or cast it at the boundary. Mismatched precision is a quiet source of poor results, so compare outputs with and without quantization on a small test.

What to log in each run

• Prefix length, learning rate and batch size.
• Random seed and data split identifier.
• Training and validation loss per epoch.
• Final metric on the held-out set.
• Peak memory and time per step.

Reporting results fairly

• State the base model, the prefix length and the number of trainable parameters.
• Report the same metrics for the frozen baseline and for full fine-tuning.
• Give the mean and spread over several seeds.
• Describe the data split and any preprocessing.
• Avoid claiming interpretability that was not tested.

Common mistakes

• Training the backbone by accident. If any base parameter has gradients, you are no longer doing prefix tuning.
• Skipping reparameterization. Direct training of the prefix can diverge on some models.
• Keeping the helper network in production. After training, only the final prefix is needed.
• Setting the prefix too long. It wastes context and can overfit.
• Using a low learning rate. Prefixes start untrained and often need a higher rate than full fine-tuning.
• Reusing a prefix on a different model. A prefix is tied to its exact backbone.
• Ignoring seeds. Results can vary, so report several runs.
• No baseline. Without the zero-shot and tree ensemble baselines, gains are hard to judge.
• Data leakage. Features that encode the target inflate scores, especially in tabular tasks.
• Over-trusting interpretability. Prefix vectors do not translate to words, so avoid making claims about what a prefix "says".
• Forgetting preprocessing parity. Training and serving must use identical encoders and missing-value rules.

Frequently asked questions

What is prefix tuning?

Prefix tuning is a method that adapts a frozen model by learning a short sequence of continuous vectors attached to the attention keys and values at every layer. Only these vectors are trained. The result is a very small file per task that steers the shared backbone.

How is prefix tuning different from prompt tuning?

Prompt tuning learns soft tokens only at the input layer, while prefix tuning learns vectors at every layer. The deeper placement gives more steering power, especially for smaller models and harder tasks. Prompt tuning is simpler and cheaper, and it tends to catch up as the base model grows.

Does prefix tuning change the model weights?

No. All base weights stay frozen. Only the prefix parameters, and a temporary helper network if you use one, are updated. After training you can discard the helper and keep just the prefix.

How long should the prefix be?

A length of 10 to 30 virtual tokens is a common starting range. Shorter prefixes are cheaper and may suffice for simple tasks, while longer ones add capacity but can overfit and use up context. The best value depends on the task, so choose it with validation data.

Why is a helper network used during training?

Directly optimizing a long prefix can be unstable, so a small network that produces the prefix from a smaller matrix is used to smooth training. When training ends, the final prefix is stored and the helper is no longer needed. This keeps the serving file small.

Can prefix tuning match full fine-tuning?

On some generation tasks and in low-data settings, reports show quality close to full fine-tuning with a tiny share of trainable parameters. On harder tasks or with smaller models, a gap can remain. Test against full fine-tuning and LoRA on your own data before choosing.

Does prefix tuning add inference cost?

Yes, a small one. Every token attends to the extra prefix positions, which adds compute and memory, and the prefix uses part of the context window. The cost is modest for short prefixes, but it is not zero, and the prefix cannot be merged into the weights.

Can prefix tuning be used for tabular models?

The mechanism applies to any attention-based backbone, including models pre-trained on tables. A prefix can steer such a model toward a domain while the backbone stays shared. Treat it as an experiment, and compare with the zero-shot model, adapters, LoRA and tuned tree ensembles.

Is prefix tuning still widely used?

It is well known and supported in common libraries, but many production teams now prefer LoRA-style methods because they can be merged and often give strong quality. Prefix tuning remains useful for research, for per-user steering and for cases where mergeable updates are not needed.

Can I use several prefixes together?

You can keep many prefixes for different tasks and select one per request. Combining two prefixes at once is less standard and needs testing, since interactions are hard to predict. A routing layer that picks the right prefix is the usual approach.

Glossary

• Attention: The operation that lets each position read from other positions.
• Backbone: The large pre-trained model that stays frozen.
• Continuous prompt: A prompt made of learned vectors rather than words.
• Deep prompt: A learned prompt applied at many layers.
• Fine-tuning: Further training of a pre-trained model on a target task.
• Foundation model: A large model pre-trained on broad data and reused for many tasks.
• Frozen weights: Weights that receive no updates.
• In-context learning: Predicting from examples in the input without weight updates.
• Key and value: The vectors that attention uses as memory.
• Masked cell modeling: Pre-training by hiding table cells and predicting them.
• PEFT: Parameter-efficient fine-tuning.
• Prefix: A learned set of key and value vectors prepended in attention.
• Reparameterization: Producing parameters through a helper network for stable training.
• Soft prompt: Another name for a learned continuous prompt.
• Virtual token: A prefix position not tied to a real word.

Summary

Prefix tuning freezes a pre-trained model and learns a small set of continuous key and value vectors that are prepended in attention at every layer. It produces tiny per-task files, protects the original model and works well on generation tasks and in low-data settings. Its costs are context usage, some training sensitivity, and no merging into the base weights.

For tabular foundation models, the same mechanism offers a way to steer a table-native backbone toward a domain, a source or a label set, while the backbone is shared and unchanged. It fits continual and multi-tenant setups through per-domain prefixes. As with all PEFT methods, validate against the zero-shot model, LoRA, adapters and a tuned tree ensemble. For more, see prompt tuning, IA3, BitFit, /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: