Prompt tuning

Overview

Prompt tuning is a parameter-efficient method that adapts a large frozen model by learning a short sequence of vectors, called a soft prompt, placed in front of the input embeddings. The model weights never change. Only the soft prompt is trained with gradient descent, and the file you store per task can be tiny.

The method is the simplest member of the PEFT family. It is easier to implement than prefix tuning, adapters or LoRA, and it keeps one shared backbone for many tasks. Reports show that it works increasingly well as the base model gets larger. On small models it can lag behind richer methods. For tabular foundation models, the same idea offers a lightweight way to steer a table-native backbone toward a domain, which is worth testing against in-context prediction and tree ensembles.

What it is

A soft prompt is a block of trainable vectors that look like token embeddings but do not correspond to any word. They are concatenated in front of the real input embeddings, and the combined sequence goes through the frozen model. Training adjusts the soft prompt so that the frozen model produces the right output for the task.

A simple picture: a hard prompt is a note written in plain words, such as "Summarize the text below." A soft prompt is a note written in the model's own private code. The code can say things no sentence can say, because it is not limited to the vocabulary. You do not need to read it. You only need it to work.

Hard prompts and soft prompts

Where it sits in the model

The soft prompt exists only at the input layer. Higher layers see its effect through the usual flow of hidden states. That is different from prefix tuning, where learned vectors are added in the attention of every layer.

Why it is called tuning

The prompt is treated as a set of parameters and tuned by optimization. The word "tuning" here means gradient-based training of the prompt, not manual wording changes.

What gets saved

Only the soft prompt tensor is saved. Its size is the prompt length times the embedding size. A prompt of 20 positions and embedding width of 4,096 has about 82,000 values. That is tiny compared with a model that has billions of weights.

Where it came from

Soft prompt tuning was described by Lester, Al-Rfou and Constant in 2021 in the paper "The Power of Scale for Parameter-Efficient Prompt Tuning". The authors trained soft prompts on frozen text-to-text models of different sizes and found that, as the model grew, prompt tuning closed the gap with full fine-tuning. They also showed that one frozen model could serve many tasks by swapping prompts, which makes serving cheaper.

Related work came around the same time:

• Prefix tuning (Li and Liang, 2021) learned deep continuous prefixes for generation.
• Earlier work on automated discrete prompt search tried to find the best words by search.
• P-tuning explored continuous prompts for understanding tasks.
• P-tuning v2 applied deep prompts at every layer and showed that this helps smaller models.
• Adapters (2019) showed that small add-on modules can carry a task.

Prompt tuning became a standard reference for parameter-efficient methods, and it is available through common libraries such as the PEFT package in the Hugging Face ecosystem. Methods like LoRA later became more popular for large language model fine-tuning because they can be merged into the weights. For the research setting around reusing one backbone, see /model/research.

How it works

Step 1: Freeze the model

Load a pre-trained model and set all of its weights so that they do not receive updates. This includes embeddings, attention, feed-forward layers and normalization.

Step 2: Create the soft prompt

Create a trainable matrix of shape prompt length by embedding size. Initialize it. There are three common choices:

• Random values drawn from a small range.
• Embeddings copied from the vocabulary, for example from the most frequent tokens.
• Embeddings of words related to the task, such as the names of the output classes.

The original paper found that initializing from real vocabulary and class label embeddings worked better than random, especially for smaller models.

Step 3: Prepend to the input

For each training example, convert the input tokens to embeddings in the usual way. Then concatenate the soft prompt in front. The attention mask is extended so that real tokens can attend to prompt positions.

Analogy: before each exam question, you slip a short coded hint card in front of the paper. The student reads the card first, then the question. The card is trained so that the student answers well.

Step 4: Run the frozen model

The combined sequence passes through the model. The prompt vectors interact with real tokens through attention in every layer, even though the prompt exists only at the input. The model then produces an output, for example the next tokens of a label word.

Step 5: Compute the loss

For classification, the usual approach is to cast the task as text generation: the model should output the label word. The loss is cross entropy on those output tokens. For regression and other tasks, you can add a small head or format the target as text.

Step 6: Update only the prompt

Gradients flow back through the whole frozen model to the input embeddings. Only the soft prompt is updated by the optimizer. The optimizer state exists only for these few parameters, which keeps memory low.

Step 7: Save and swap

Save the soft prompt. To serve several tasks, load the shared model once and attach the prompt for each request. Because prompts are tiny, many can sit in memory together.

Prompt ensembling

The paper also described ensembling: train several prompts with different seeds, run them in a mixed batch through one frozen model, and combine predictions. This gives gains similar to ensembling several full models, at the price of only extra prompts.

Choosing the prompt length

Reports on large models show that quality tends to plateau at modest lengths. Test on validation data.

The math in brief

Let X be the embedded input of shape n by d, and P the soft prompt of shape m by d. The model sees:

input = concat(P, X)         # shape (m + n) by d
output = frozen_model(input)

Only P is trainable. The gradient of the loss with respect to P is computed by backpropagation through the frozen network.

Training tips from the research

• Use a relatively high learning rate compared with full fine-tuning, because the prompt starts from scratch.
• Use more training steps. Prompt tuning often needs longer training than other methods.
• Use prompt lengths that fit the context budget.
• Prefer larger base models when possible, since the method works better as size grows.

Variants and relatives

Key concepts and terms

A worked example

This PyTorch sketch shows a soft prompt in front of a frozen toy encoder for classification. It illustrates the mechanics, not a production system.

import torch
import torch.nn as nn


class SoftPromptClassifier(nn.Module):
    def __init__(self, vocab=1000, d=64, prompt_len=10, n_classes=3):
        super().__init__()
        self.embed = nn.Embedding(vocab, d)
        layer = nn.TransformerEncoderLayer(d_model=d, nhead=4, batch_first=True)
        self.encoder = nn.TransformerEncoder(layer, num_layers=2)
        self.head = nn.Linear(d, n_classes)
        # trainable soft prompt
        self.prompt = nn.Parameter(torch.randn(prompt_len, d) * 0.02)

    def forward(self, ids):
        x = self.embed(ids)                         # (b, n, d)
        p = self.prompt.unsqueeze(0).expand(x.size(0), -1, -1)
        z = torch.cat([p, x], dim=1)                # (b, m+n, d)
        h = self.encoder(z)
        return self.head(h[:, 0])                   # read first prompt slot


model = SoftPromptClassifier()
for name, p in model.named_parameters():
    p.requires_grad = name in ("prompt", "head.weight", "head.bias")

params = [p for p in model.parameters() if p.requires_grad]
opt = torch.optim.AdamW(params, lr=1e-2)
loss_fn = nn.CrossEntropyLoss()

ids = torch.randint(0, 1000, (16, 20))
y = torch.randint(0, 3, (16,))
for step in range(50):
    loss = loss_fn(model(ids), y)
    opt.zero_grad()
    loss.backward()
    opt.step()
print(sum(p.numel() for p in params))

What to notice

• The embedding table and the encoder are frozen.
• The soft prompt and a small head are the only trainable parts.
• The prompt has 10 positions of width 64, so 640 values.
• The head adds 195 values, so the trainable total is 835.

Numeric intuition

Suppose the frozen model has 1.5 billion parameters. A prompt of 20 positions with width 2,048 has 40,960 values. That is about 0.003 percent of the model. Even with a larger prompt, the share stays far below one percent.

Initializing from class names

A useful trick is to initialize the first prompt slots from the embeddings of the class label words. For a sentiment task with the labels "positive" and "negative", copy those embeddings into the prompt. The model starts closer to the answer space and trains faster.

Using a library

# Pseudocode shape of a typical workflow
base = load_pretrained("base-model-name")
cfg = PromptTuningConfig(num_virtual_tokens=20, init_from_text="Classify the label:")
model = wrap_with_peft(base, cfg)
train(model, train_data, lr=3e-2, epochs=20)
model.save_prompt("prompt_task_a")

Exact class names differ by library version, so check the documentation of the PEFT package you use.

Strengths and limitations

Strengths

• The simplest PEFT method, with only one tensor to learn.
• Tiny storage per task, so thousands of tasks can share one model.
• No change to the backbone, so no forgetting of pre-training.
• Mixed-task batches are possible, because prompts are just inputs.
• Cheap to train in memory terms.
• Strong results on large models, as shown in the original scale study.
• Easy to understand and explain to non-experts.

Limitations

• Weaker on small and mid-size models than richer methods.
• Can need many training steps and a high learning rate.
• Sensitive to initialization and random seeds.
• Uses context positions, which reduces space for real input.
• Not mergeable into the model weights.
• The learned prompt is not human readable.
• Often below LoRA and full fine-tuning on hard tasks.

Scale matters

The central finding of the scale study is that quality improves with model size. When the backbone has many billions of parameters, a small soft prompt can steer it very well. When the backbone is small, the prompt may lack the leverage it needs. If you use a smaller model, consider deep prompts, adapters or LoRA.

How it compares with alternatives

Prompt tuning versus manual prompts

Manual prompts need human effort and are often brittle: a small wording change can swing results. Prompt tuning replaces that search with optimization, and gives a prompt that is optimized for the exact model. The price is the need for labeled data and training, and the loss of readability.

Prompt tuning versus fine-tuning

Full fine-tuning changes the model and can reach higher accuracy, but it needs a model copy per task and carries forgetting risk. Prompt tuning keeps one model and one tiny file per task. If you serve many tasks or tenants, the saving is large.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types, missing cells and often no useful column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on large numbers of synthetic or real tables, and they predict on new tables by reading labeled rows as context. Prompt tuning offers a light way to steer such a model. See /model/architecture for how table-native backbones are built.

A soft prompt as learned context

In-context tabular models read example rows and then predict for query rows. A soft prompt can act as a learned set of "virtual rows" or "virtual column tokens" that carry domain information. Instead of putting thousands of real rows into the context, you place a few learned positions that bias the model toward the domain prior. This is an analogy. It must be validated on real data, because the table encoders and attention patterns differ from text models.

Row and column structure

Tabular transformers often treat cells as tokens and attend across rows and across columns. A soft prompt can be added as extra column-like tokens, extra row-like tokens, or both. Column-like prompt tokens can encode which features matter. Row-like prompt tokens can encode typical records. Which one helps depends on the model and the domain.

Mixed types and missing values

A cell encoder maps numbers, categories, dates and text into a shared space, and a special embedding marks missing cells. A soft prompt works after this step, in embedding space. It can help the model interpret domain patterns, such as a missing value that signals "not applicable" in one table family. The encoders stay frozen.

No column names

When there are no headers, the model relies on values and structure. When names exist, text-aware models such as TabSTAR can use them. A prompt can learn to compensate in a domain where names are missing, by encoding typical value ranges and relations that the header would supply.

Synthetic and real pre-training

Synthetic pre-training teaches generic structure, such as interactions, noise and missingness. Real-table pre-training adds semantics. A soft prompt sits on top: it is a cheap domain adapter and not a substitute for pre-training. A practical approach is one prompt per data source, with a shared backbone. Compare it with the plain in-context result and with tuned trees such as XGBoost, as summarized in the evaluation overview at /model/benchmarks.

Continual learning

When new domains arrive, learn a new prompt and leave the backbone alone. This isolates knowledge and avoids forgetting without replaying old data as in experience replay or constraining weights as in EWC. You do need a way to select the right prompt at serve time, such as a domain tag or a small router.

Imputation and anomaly detection

For imputation, a backbone trained with masked cell modeling predicts hidden cells. A prompt can adapt it to table families with specific gap patterns. For anomaly detection, a prompt can encode what normal looks like for a system, improving reconstruction or density scores. See also imputation networks.

Sector and semantic classification

A prompt plus a semantic head can specialize a backbone for a label set such as industry sectors. If labels change, retrain only the prompt and head.

Caveats for tables

Prompt tuning was designed for text models, and its best results are on very large ones. Many tabular foundation models are much smaller. That suggests richer methods, such as adapters or LoRA, may be a better first try. Also, in-context tabular models already adapt through the context rows, so a prompt adds value mainly when the domain is stable and labeled data is available.

Practical guidance and best practices

A sensible order of experiments

1. Run the frozen model with in-context examples and record the score.
2. Add a short soft prompt and train it. Record the score.
3. Try a longer prompt and class-name initialization.
4. Compare with LoRA or adapters.
5. Compare with a tuned tree ensemble such as LightGBM.
6. Keep the simplest method that meets your target.

Hyperparameters

Validate with care

On tabular tasks use k-fold cross-validation with leakage-safe preprocessing. Fit scalers and encoders inside each fold. Report mean and spread across folds and seeds.

Version and track

• Store the prompt with the backbone version.
• Log hyperparameters and metrics with a tracker such as MLflow.
• Keep a small test that confirms the prompt loads on the right model only.

Serving

• Keep the backbone loaded and attach prompts per request.
• Use mixed-task batches where the framework supports them.
• Measure latency, since the prompt adds positions.
• Cache the prompt embeddings.

Combine with quantization

A frozen backbone can be stored in lower precision with tools such as bitsandbytes, while the prompt stays in higher precision. Recheck quality after quantization.

Pre-flight checklist

• Print the trainable parameter count and confirm it equals the prompt plus any head.
• Confirm that no backbone parameter has gradients.
• Overfit a tiny batch to prove that the prompt can learn.
• Fix the evaluation split before tuning.
• Check that preprocessing at train and serve time is identical.

Everyday analogy for prompt transfer

If a student already learned a related subject, a short refresher note is enough to start a new one. Prompt transfer works like that. You start a new prompt from one trained on a related task, so training begins closer to a good solution and needs less data.

Privacy and governance

A prompt trained on private data can encode traces of it. Treat prompt files as sensitive artifacts, control access, and delete them when the data right expires. Test fairness metrics per prompt when it serves different user groups.

Debugging a prompt that does not learn

1. Confirm that the prompt is a registered parameter and is passed to the optimizer.
2. Print the gradient norm of the prompt after one backward pass.
3. Raise the learning rate by a factor of ten and watch the loss.
4. Try class-name initialization instead of random values.
5. Check that labels and inputs are aligned.
6. Try a longer prompt if the loss plateaus high.
7. Confirm that the attention mask covers the prompt positions.

Debugging a prompt that overfits

• Shorten the prompt.
• Add weight decay and early stopping.
• Use more data or stronger cross-validation.
• Look for duplicate rows across the train and validation sets.
• Average several prompts trained with different seeds.

Planning the context budget

Count the positions you need. A text model may have a window of thousands of positions, so a 20-position prompt is cheap. A tabular model that reads many example rows may use its window more fully. In that case every prompt position competes with example rows. Measure accuracy with different splits of the budget, for example fewer example rows and a longer prompt, or the reverse.

Task design for classification

• Map each class to a short label word or token.
• Keep label words distinct from each other.
• Balance classes in the training batches, or weight the loss.
• Check calibration of predicted probabilities, since soft prompts can be over-confident.
• Use a small validation set to choose a decision threshold if classes are imbalanced.

What to log in each run

• Prompt length, initialization method and learning rate.
• Random seed and data split identifier.
• Loss curves for training and validation.
• Final held-out metric and its spread across seeds.
• Peak memory and time per step.

Multi-tenant serving note

When many tenants each have a prompt, keep a registry that maps tenant to prompt version. Roll out new prompts to a small share of traffic first. Keep the previous version available for fast rollback.

Reporting results fairly

• State the base model, the prompt length and the trainable parameter count.
• Report the frozen baseline and a full fine-tuning reference.
• Describe the split and the preprocessing.
• Avoid claims about what the prompt means.

Common mistakes

• Using too low a learning rate. Prompts start from scratch and need larger steps than full fine-tuning.
• Stopping too early. Prompt tuning tends to converge slowly.
• Random initialization on small models. Initializing from real embeddings often helps.
• Expecting it to work equally on every model size. Gains are strongest on large backbones.
• Making the prompt too long. It wastes context and can hurt quality.
• Forgetting to freeze the backbone. Then you are fine-tuning and lose the benefits.
• Single-seed conclusions. Variance is high, so use several seeds.
• No baseline. Compare with the frozen model and a tuned tree ensemble.
• Data leakage. Target-encoding features outside the fold inflate scores.
• Reusing a prompt on another model. A prompt is tied to its exact backbone and embedding space.
• Trying to read the prompt. Soft prompts are not words, so avoid claims about their meaning.

Frequently asked questions

What is prompt tuning?

Prompt tuning is a method that adapts a frozen model by learning a short sequence of vectors, called a soft prompt, placed before the input embeddings. Only the soft prompt is trained. This gives a very small file per task and leaves the model weights unchanged.

How is prompt tuning different from prefix tuning?

Prompt tuning adds learned vectors only at the input layer, while prefix tuning adds learned keys and values at every attention layer. Prefix tuning has more steering power and works better on smaller models. Prompt tuning is simpler and becomes competitive as the model grows.

Is prompt tuning the same as prompt engineering?

No. Prompt engineering is the manual craft of writing good text prompts. Prompt tuning trains vectors by gradient descent, and the result is not readable text. The two can be combined, for example by initializing a soft prompt from a hand-written instruction.

Does prompt tuning work on small models?

It can, but results are usually weaker than on very large models. Reports on text models show that the gap to full fine-tuning shrinks as model size increases. For small models, deep prompts, adapters or LoRA often give better quality.

How many parameters does prompt tuning train?

Only the prompt length times the embedding size, plus any small head. For a prompt of 20 positions and width 4,096, that is about 82,000 values. This is a tiny share of a model with billions of weights.

Can one model serve many prompts at once?

Yes. Because a prompt is just a prefix on the input, a batch can mix examples that use different prompts, and the shared model processes them together. This is one of the main serving advantages of the method.

Can prompt tuning be applied to tabular models?

The mechanism applies to any model that takes embeddings as input, including transformers pre-trained on tables. A soft prompt can steer such a model toward a domain. Treat it as an experiment and compare with in-context prediction, adapters, LoRA and tuned tree ensembles.

Does prompt tuning add latency?

A little. The prompt adds positions to the input, so attention works on a longer sequence. For short prompts the cost is small. Because the prompt cannot be merged into the weights, the cost remains at inference time.

What is prompt ensembling?

Prompt ensembling trains several prompts with different seeds and combines their predictions. Since all prompts share one frozen model, the ensemble costs far less than training and storing several full models. It can improve accuracy and stability.

When should I choose LoRA instead?

Choose LoRA when you want higher quality on a mid-size model, when you need to merge updates for fast serving, or when your task is hard. Choose prompt tuning when the model is very large, when you need the smallest possible file, or when you want mixed-task batches.

Glossary

• Backbone: The large pre-trained model that is kept frozen.
• Context window: The maximum number of positions a model can attend to.
• Embedding: A vector that represents a token, a cell or a feature.
• Ensembling: Combining several predictions to improve accuracy.
• Fine-tuning: Further training of a model on a target task.
• Foundation model: A large model pre-trained on broad data for reuse.
• Hard prompt: A prompt written in natural language.
• In-context learning: Predicting from examples in the input without weight updates.
• Masked cell modeling: Pre-training by hiding table cells and predicting them.
• PEFT: Parameter-efficient fine-tuning.
• Prompt length: The number of soft prompt positions.
• Prompt transfer: Initializing a prompt from one trained on another task.
• Soft prompt: A learned block of vectors placed before the input.
• Text-to-text: A framing in which all tasks use text as input and output.
• Virtual token: A prompt position that is not tied to a word.

Summary

Prompt tuning learns a short soft prompt in front of the input of a frozen model. It is the simplest PEFT method, gives tiny per-task files, allows mixed-task batches, and approaches full fine-tuning quality as the base model gets larger. Its limits are weaker results on small models, slow and sensitive training, use of context positions, and no merging into the weights.

For tabular foundation models, a soft prompt can act as learned context that steers a table-native backbone toward a domain, source or label set, while the backbone is shared and unchanged. It suits per-domain and continual setups, yet the gain must be proven against the in-context baseline, adapters, LoRA and a tuned tree ensemble. For related methods see prefix tuning, IA3, BitFit and full fine-tuning, and read more at /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: