Few-shot learning

Overview

Few-shot learning is the ability of a model to pick up a new task from only a handful of labelled examples, often between one and a few dozen. Instead of collecting thousands of rows and training from scratch, the model uses what it already knows from earlier training and adapts quickly. The adaptation can happen through a small weight update, through a distance comparison to a few reference examples, or simply by reading the examples in its input.

The idea matters because labels are expensive. A hospital may have only twenty confirmed cases of a rare condition. A factory may see a new fault type once a month. A product team may launch a new category with ten sold items. Few-shot learning is the family of methods that make these situations workable. It is closely related to meta-learning, transfer learning and zero-shot learning, and it is the idea behind in-context learning in large models, including tabular foundation models such as TabPFN and TabICL.

What it is

Few-shot learning is a problem setting first, and a set of techniques second. The setting is simple: you are given a tiny labelled set for a task, and you must predict well on new items from the same task.

The vocabulary of the setting

• Support set: the small group of labelled examples you are allowed to see for the new task.
• Query set: the new, unlabelled items you must predict.
• N-way: the number of classes in the task. A 5-way task has five possible labels.
• K-shot: the number of labelled examples per class in the support set. A 5-way 1-shot task has five support items in total, one per class.
• Episode: one small task made of a support set and a query set. Training often runs over many episodes.

An everyday analogy

Imagine you are shown a photo of a bird you have never seen, with its name, and then asked to spot it in a field guide. You do not need a thousand photos. You already know what wings, beaks and feathers look like, so one example is enough to tell this bird from others. Few-shot learning tries to give models the same background knowledge, so that a small number of examples is enough to fix the details.

What few-shot learning is not

• It is not simply training on a small dataset. A model trained from scratch on twenty rows usually overfits badly.
• It is not the same as zero-shot learning, where no examples of the task are given.
• It is not a single algorithm. It covers metric learning, meta-learning, fine-tuning with few labels, and prompting with examples.

Where it came from

The roots are older than modern deep learning. Psychologists had long noted that people learn concepts from very few instances. In machine learning, early work on one-shot learning of object categories showed in the mid 2000s that a model could use prior knowledge from other categories to learn a new one from a single example.

A few milestones, described in general terms:

• Siamese networks for verification, from the early 1990s and revived in the 2010s, compared pairs of items rather than classifying single ones. This is the seed of metric-based few-shot methods.
• Matching Networks (2016) framed one-shot classification as a weighted vote over a support set, and popularised episodic training.
• Prototypical Networks (2017) simplified the idea: average the embeddings of each class into a prototype and classify by nearest prototype.
• MAML, model-agnostic meta-learning (2017), trained a starting point for weights that could be fine-tuned in a few gradient steps on any new task.
• Language Models are Few-Shot Learners (2020) showed that a large language model could perform new tasks by reading a few examples in its prompt, with no weight updates. This made "few-shot" a household term for prompting.
• TabPFN (2022 onward) brought the same in-context idea to tables: a transformer pre-trained on many synthetic datasets that classifies a new table's rows in a single forward pass.

The shift over this timeline is worth noticing. Early methods learned a good embedding space. Later methods learned good starting weights. The most recent methods learn to learn inside the forward pass, so that the examples themselves act as the training data. See /model/research for how this line of work connects to tabular foundation models.

How it works

There are four main routes. Real systems often mix them.

Route 1: Metric-based methods

The model maps every item into an embedding space where similar items sit close together. A new class is then handled by comparing distances.

1. Pre-train an encoder on many classes so that its embeddings separate classes well.
2. For a new task, embed each support example.
3. Average the embeddings of each class to get a prototype.
4. Embed each query item and assign it to the nearest prototype.

The analogy is a map. Once you know where the landmarks are, finding which landmark a new point is closest to needs no new surveying. The hard work was done when the map was drawn.

Route 2: Optimisation-based meta-learning

Here the model learns an initial set of weights that is easy to adapt. Training runs over many small tasks. In an inner loop, the model takes a few gradient steps on a support set. In an outer loop, the initial weights are updated so that, after those few steps, the model does well on the query set.

The analogy is a cook who has practised many cuisines. Given a new recipe and a few tasting rounds, the cook adjusts quickly. The skill is not any one recipe but the habit of adapting fast. See meta-learning for more.

Route 3: Fine-tuning a pre-trained model

This is the most common practical route. Start from a large pre-trained model, then update a small part of it using the few labels. To avoid overfitting, people freeze most layers, use a low learning rate, or use parameter-efficient methods such as LoRA. This route leans on transfer learning.

Route 4: In-context learning

A large pre-trained model is given the examples as part of its input, along with the new query. It outputs the answer in one forward pass. No weights change. The model has learned during pre-training to treat the examples as evidence.

Why can this work? During pre-training, the model sees a huge number of tasks, each presented as a list of examples followed by a query. To score well, it must learn a general procedure for inferring a rule from examples and applying it. At test time, the new task is just another instance of that procedure.

The analogy is an experienced detective. Show a detective five short case files and a sixth open case, and they reason from the pattern without being retrained. The experience is already in their head.

A step-by-step view of an episode

1. Sample a task from a large pool of tasks.
2. Pick a small support set and a query set from that task.
3. Show the support set to the model, either as training for an inner loop or as context.
4. Ask the model to predict the query labels.
5. Compute the loss on the query predictions.
6. Update the model, then move to the next task.

The key design choice is that training episodes mimic the test situation. If you want the model to work with five examples at test time, train it on episodes with about five examples.

Related variants of the setting

Few-shot learning has several close relatives, and it helps to know the names.

• One-shot learning: exactly one labelled example per class. It is the hardest case and leans most on the prior.

• Few-shot with generalised labels: the test set may contain both new classes and old ones, so the model must not forget the old classes.

• Cross-domain few-shot: the pre-training data and the new task come from different domains, such as photos versus sketches or retail tables versus sensor tables.

• Few-shot regression: predicting a number rather than a class from a handful of examples. In-context tabular models handle this by reading numeric targets in the context.

• Few-shot detection: finding objects or events of a new type from a few labelled instances.

• Few-shot generation: producing new items in the style of a few examples, which links to diffusion models and GANs.

Why the prior matters more than the algorithm

A common surprise is that the choice among prototype, meta-learning and fine-tuning methods matters less than the quality of the pre-trained model. A strong encoder trained on diverse data with a simple nearest-prototype rule often beats a clever algorithm on a weak encoder. The practical lesson is to invest in the prior first, through broad pre-training, good architecture design and rich synthetic data generation, and only then tune the adaptation step.

Typical evaluation protocol

1. Fix a pool of tasks that were not used in training.

2. For each task, draw a support set of the chosen size.

3. Draw a disjoint query set.

4. Predict and record accuracy or another metric.

5. Repeat for hundreds of tasks.

6. Report the mean with a confidence interval.

A simple mental model

Think of the model as a student, the pre-training as years of school, and the support set as a short briefing. A well-schooled student needs a short briefing. A poorly schooled student needs a long course, no matter how good the briefing is. Few-shot learning is therefore less about the briefing and more about the schooling.

Key concepts and terms

A worked example

This example classifies rows from a small table using a prototype approach in Python. It shows the core idea in a few lines: embed, average per class, compare.

import numpy as np
from sklearn.preprocessing import StandardScaler

# Support set: 3 labelled rows per class (a 2-way 3-shot task)
X_support = np.array([
    [5.1, 3.5], [4.9, 3.0], [5.0, 3.4],   # class 0
    [6.7, 3.1], [6.3, 2.9], [6.5, 3.0],   # class 1
])
y_support = np.array([0, 0, 0, 1, 1, 1])

# Query set: new rows to classify
X_query = np.array([[5.2, 3.3], [6.4, 3.2], [5.8, 3.0]])

scaler = StandardScaler().fit(X_support)
S = scaler.transform(X_support)
Q = scaler.transform(X_query)

# Prototype per class = mean of support embeddings
prototypes = np.stack([S[y_support == c].mean(axis=0) for c in (0, 1)])

# Distance from each query to each prototype
dists = np.linalg.norm(Q[:, None, :] - prototypes[None, :, :], axis=2)
pred = dists.argmin(axis=1)
print(pred)

Walking through it

• The support set has six rows: three per class. That is a 2-way 3-shot task.
• We scale the features so that no single column dominates the distance.
• Each prototype is the mean of its class rows in scaled space.
• Each query row is assigned to the closest prototype.
• The first query is near the class 0 rows, the second near class 1. The third sits between them, and the nearest prototype decides.

A numeric check by hand

Suppose class 0 has prototype (-1.0, 0.5) and class 1 has prototype (1.0, -0.5) in scaled space. A query at (-0.6, 0.3) has distance about 0.45 to class 0 and about 1.7 to class 1. It goes to class 0. The decision is simply "which prototype is nearer".

The same task with in-context learning

With a model such as TabPFN, the code is shorter, because the pre-trained network does the work.

from tabpfn import TabPFNClassifier

clf = TabPFNClassifier()
clf.fit(X_support, y_support)   # stores the examples as context
pred = clf.predict(X_query)     # one forward pass, no gradient training

Here fit does not train weights in the usual sense. It hands the support rows to the network as context. The prediction is produced by attention over those rows.

Prompting with examples for a language model

The same idea in text form looks like this:

Classify the sentiment.
Text: "Arrived late and damaged." -> negative
Text: "Works exactly as described." -> positive
Text: "Packaging was fine but the item broke in a week." ->

The two labelled lines are the support set. The last line is the query.

Strengths and limitations

Strengths

• Low label cost. Useful when labelling needs experts or time.
• Fast adaptation. In-context methods adapt in a single forward pass.
• Handles new classes. Metric methods can add a class by adding a prototype, without retraining.
• Good starting point for rare events. Faults, fraud types and rare diseases often have few examples.
• Less engineering per task. One pre-trained model can serve many small tasks.

Limitations

• High variance. With very few examples, the choice of which examples you give can change results a lot.
• Depends on the prior. If the new task is far from what the model saw in pre-training, adaptation fails.
• Capped by context length. In-context methods can only read so many examples at once.
• Class imbalance hurts. A support set with one example of a class gives a weak prototype.
• Noisy labels are costly. One wrong label among five is a large share of the evidence.
• Hard to evaluate. Small test sets give noisy estimates, so claims need many episodes and error bars.
• Not always better than a simple baseline. A regularised linear model or gradient-boosted trees on a few dozen rows can be surprisingly strong. See XGBoost.

When few-shot is the wrong tool

If you have thousands of clean labels and a stable task, classic supervised training is simpler and usually stronger. Few-shot learning earns its place when labels are scarce, when tasks change often, or when you must serve many small tasks.

How it compares with alternatives

How to choose between them

• If you have no labels at all but a good description of the task, try zero-shot.
• If you have a few labels and a strong pre-trained model, try few-shot with in-context learning first.
• If you have a few labels and can label more, add active learning to pick the next ones.
• If you have many unlabelled rows, combine few-shot with semi-supervised methods.

Role in tabular foundation models

This is where few-shot learning has become practical for everyday data work. A tabular foundation model is pre-trained on a very large number of tables so that it can make predictions on a new table without being trained on it. Few-shot learning is the mechanism that makes this possible.

Rows as examples

In a table, each row with a known target is a labelled example. A new row with an unknown target is a query. So the training rows of a new table are the support set, and the rows to predict are the query set. This matches the few-shot setting directly, with the number of shots equal to the number of labelled rows.

In-context learning on tables

Models such as TabPFN and TabICL read the labelled rows and the query rows together. Attention lets each query row look at the labelled rows and infer the pattern. There is no per-table gradient training. This is in-context learning, the fourth route above. It works best when the table is small to medium sized, which is exactly where classical deep learning struggles.

Pre-training builds the prior

The model learns the procedure for inferring rules from examples by seeing millions of tables during pre-training. Because real labelled tables are hard to collect in that volume, many such models use synthetic data generation to create varied tables with known structure. This pre-training is the tabular equivalent of the episodic training in classic few-shot work: each synthetic table is an episode.

Mixed types and missing values

Real tables mix numbers, categories, dates and text, and they have gaps. A model that learns from raw tables must handle these inside the same few-shot loop. Typical choices:

• Numeric columns are scaled and passed as numeric tokens.
• Categorical columns are encoded, either by index or by learned embeddings. See TabSTAR for text-aware handling.
• Missing cells are flagged with a mask so that the model can treat "unknown" as a signal. See masked cell modeling.

Few-shot behaviour depends on this handling. With only a few rows, every missing value removes a large share of the evidence.

No column names needed

A strong tabular model should work even when column names are absent or meaningless, because many real exports have names like col_7. In the few-shot view, the model must infer what a column means from its values and its relations to the target, not from the header. Models that can use names when present and still work without them are more robust.

Imputation and anomaly detection as few-shot tasks

• Imputation. Predicting a missing cell is a prediction task where the other cells of the row are features and the known cells in the column are examples. A few-shot learner can fill gaps with little per-table training. See imputation networks.
• Anomaly detection. With a few labelled anomalies, a model can compare new rows to both normal and anomalous prototypes. See anomaly detection and Isolation Forest.

Sector and semantic classification

Given a handful of labelled examples of a table's domain, a model can learn to tag new tables or columns with a sector or a semantic type, such as "finance" or "postal code". This is few-shot classification at the table level. See semantic sector heads.

Continual learning

Few-shot adaptation is also a gentle way to extend a model. New tasks can be added by providing examples rather than retraining the whole network, which reduces the risk of forgetting. Methods such as experience replay and EWC address the cases where weights do change.

Why it matters for architecture

Few-shot capability puts demands on design. Attention across rows and across columns, such as axial attention, helps the model relate query rows to support rows. Handling many rows needs efficient attention, covered under architecture design. For the architecture of a concrete system, see /model/architecture, and for measured behaviour see /model/benchmarks.

Practical guidance and best practices

Choose the support set with care

• Include every class you expect to see, even if only once.
• Pick examples that are typical, not odd edge cases, unless the edge cases matter.
• Keep the support set balanced where possible.
• If you can choose, use active learning to pick the most informative rows to label.

Measure honestly

• Evaluate over many random draws of the support set, not just one.
• Report the mean and the spread.
• Keep the query set strictly separate from the support set.
• Compare against a simple baseline trained on the same few rows.

Build a strong prior

• Pre-train on data that resembles the target domain.
• Use self-supervised learning when labels are scarce even for pre-training.
• Consider domain adaptation if the target differs from the source.

Control overfitting when you do update weights

• Freeze most of the network.
• Use a small learning rate and few steps.
• Prefer low-rank or adapter updates such as LoRA.
• Use early stopping on a held-out query set.

Prompting tips for language models

• Use examples that match the format of the real query.
• Keep labels consistent in wording and order.
• Vary the order of examples and check that results are stable.
• Include a few hard examples as well as easy ones.

Data hygiene

• Check the labels in the support set by hand. Errors matter more here.
• Remove duplicates between support and query rows. See deduplication.
• Standardise units and formats before showing examples to a model.

Deployment notes

• Cache embeddings of the support set if you reuse it.
• Log which support set produced each prediction, so you can trace changes.
• Monitor drift. If the data shifts, refresh the support set.

A practical workflow from zero to deployed

1. Write down the task in one sentence, including the labels.

2. Collect the few labelled rows you have and check each by hand.

3. Build a simple baseline, such as a regularised linear model or a small tree ensemble.

4. Try a pre-trained model with in-context learning on the same rows.

5. Compare both over many random splits of the small set.

6. Pick the stronger one, and record the spread, not just the mean.

7. Decide how new labels will arrive and when to refresh the support set.

8. Monitor accuracy and calibration after launch.

How many shots are enough

There is no universal answer, but a few patterns hold.

• Going from one example per class to five usually brings the largest gain.
• Going from five to fifty brings a smaller, steady gain.
• Beyond a few hundred rows, classic supervised methods tend to catch up with few-shot methods on tables.
• Hard classes need more shots than easy ones, so spend labelling effort where the model is confused.

Choosing the distance or similarity

For prototype methods, the choice of distance matters.

• Euclidean distance suits well-scaled numeric features.
• Cosine similarity suits embeddings where direction matters more than length.
• A learned metric can weight columns by usefulness.
• For mixed types, convert categories to embeddings first, then compare.

Handling imbalance in the support set

• Use the same number of shots per class when you can.
• If you cannot, weight prototypes or predictions by class frequency from a larger unlabelled pool.
• Report per-class results, not only overall accuracy.
• Consider threshold tuning after the fact on a held-out query set.

Combining few-shot with unlabelled data

Unlabelled rows are often plentiful even when labels are not. Useful tactics:

• Pseudo-label confident query rows and add them to the support set.
• Refine prototypes using the unlabelled cluster structure.
• Pre-train an encoder on all rows with a self-supervised objective first.
• Use the unlabelled rows to estimate drift between support and query.

Few-shot with uncertainty

Because the evidence is thin, uncertainty is important.

• Report predicted probabilities, not only labels.
• Check calibration on held-out episodes.
• Flag low-confidence predictions for human review.
• An ensemble over different support subsets gives a rough error bar.

A short note on language prompts versus table rows

Prompting a language model with examples feeds text. A tabular model feeds numbers and category codes. In both cases the model sees pairs of input and answer, then a new input. The practical rules, such as balanced classes, consistent format and honest evaluation, carry across.

Common mistakes

• Treating one run as proof. A single lucky support set can mislead. Average over many draws.
• Leaking query rows into the support set. This inflates scores sharply.
• Ignoring class coverage. A class absent from the support set cannot be predicted by a prototype method.
• Over-tuning on tiny validation sets. With ten rows, any tuning fits noise.
• Comparing with weak baselines. Compare with a well-tuned simple model.
• Forgetting preprocessing. Unscaled features can dominate distance-based methods.
• Using in-context learning beyond its context limit. Truncation silently drops examples.
• Assuming the prior fits the task. If the task is unlike pre-training, few-shot results can be poor.
• Skipping calibration. Few-shot probabilities can be over- or under-confident. Check calibration.
• Treating prompts as stable. Small wording changes can alter a language model's answers.

Frequently asked questions

What is few-shot learning in simple terms?

Few-shot learning means teaching a model a new task from only a few labelled examples. The model relies on general knowledge it gained earlier, so it does not need thousands of examples to get started. It is like a person who has seen many animals and can recognise a new one after seeing a single photo.

How many examples count as few?

There is no strict number, but the term usually covers one to a few dozen examples per class. One-shot means a single example per class. Some tabular work stretches the idea to a few hundred rows, because compared with the millions used in standard training, that is still small.

What is the difference between few-shot and zero-shot learning?

Few-shot learning gives the model a handful of labelled examples of the task, while zero-shot learning gives none and relies on a description or on general knowledge. Few-shot is usually more accurate when good examples exist. Zero-shot is useful when no labels are available at all.

Is in-context learning the same as few-shot learning?

In-context learning is one way to do few-shot learning. The examples are placed in the model input and the model predicts without changing its weights. Other few-shot methods do change weights, such as fine-tuning or meta-learning with gradient steps, so the two terms overlap but are not identical.

Does few-shot learning work on tabular data?

Yes, and it has become a practical approach for small and medium tables. Pre-trained tabular models read the labelled rows as context and predict new rows in one pass. They often match or beat tuned tree models on small datasets, although results depend on the data and should be checked against a baseline.

Why do few-shot results vary so much between runs?

With so few examples, the particular rows you choose carry a lot of weight. A different draw can shift the prototypes or the context enough to change predictions. This is why careful evaluation averages over many random support sets and reports the spread.

Can few-shot learning replace collecting more data?

Not entirely. It reduces the amount of labelled data needed to reach a decent result, but more clean labels usually still help. A good practice is to start with few-shot to get a working system, then add labels over time, ideally chosen with active learning.

How is few-shot learning related to meta-learning?

Meta-learning is a way of training that prepares a model for few-shot tasks. The model practises on many small tasks so that it becomes good at adapting. Many few-shot methods, including large pre-trained models, can be seen as a form of meta-learning at scale.

What goes wrong when the tasks differ from pre-training?

The model's prior may not apply, so the few examples are not enough to correct it. Predictions can become unreliable and over-confident. In that case, domain adaptation, a better source of pre-training data, or more labels is the safer path.

How do I know whether to use few-shot or fine-tuning?

If you have only a handful of labels and need results fast, try few-shot or in-context learning first. If you have a few hundred or thousand labels and a stable task, light fine-tuning often gives better accuracy. Test both on a held-out set and pick the one that is stronger and simpler to maintain.

Glossary

• Adapter: a small trainable module added to a frozen model.
• Attention: a mechanism that lets each item weigh the importance of other items.
• Baseline: a simple model used for comparison.
• Calibration: how well predicted probabilities match real frequencies.
• Context window: the maximum input a model can read at once.
• Embedding: a vector that represents an item so that similar items are close.
• Episode: a small task with a support set and a query set.
• Few-shot: learning from a handful of labelled examples.
• Fine-tuning: continuing training of a pre-trained model on new data.
• In-context learning: adapting by reading examples in the input.
• K-shot: K labelled examples per class.
• Meta-learning: training a model to learn new tasks quickly.
• N-way: N classes in the task.
• One-shot: a single labelled example per class.
• Overfitting: fitting noise in a small training set.
• Prior: knowledge a model has before seeing the task.
• Prototype: the mean embedding of a class.
• Query set: the items to be predicted.
• Support set: the labelled examples given for the task.
• Task distribution: the family of tasks seen during training.

Summary

Few-shot learning lets a model handle a new task from a small number of labelled examples by relying on a strong prior. It can work through distances to class prototypes, through meta-learned starting weights, through light fine-tuning, or through in-context learning where the examples are read as part of the input. The approach shines when labels are scarce and tasks change often, and it needs careful evaluation because small samples are noisy.

For tables, few-shot learning is the core mechanism of modern tabular foundation models: the labelled rows are the support set, the rows to predict are the query set, and pre-training on many varied tables, often synthetic, supplies the prior. This links directly to imputation, anomaly detection, semantic classification and continual learning on raw tables. Related reading includes meta-learning, zero-shot learning, TabPFN, /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: