Zero-shot learning

Overview

Zero-shot learning is the ability of a model to perform a task or recognise a class that it was never explicitly trained on, with no labelled examples of that task. Instead of examples, the model is given a description, a name, a set of attributes, or simply an instruction. It then relies on general knowledge learned during pre-training to connect that description to the right answer.

The approach is valuable when labels are impossible or too slow to collect. New product categories appear every week. New fault types appear in machines. New columns appear in customer tables. A zero-shot system can give a useful first answer on day one, before anyone has labelled anything. It sits next to few-shot learning and transfer learning on a spectrum of how much task-specific data a model needs, and it is increasingly used with foundation models, including tabular foundation models such as TabPFN that predict on new tables without per-table training.

What it is

Zero-shot learning is a setting in which the classes or tasks seen at test time are not seen at training time. The model must bridge the gap with some shared knowledge.

The core requirement

To work, there must be a bridge between what the model knows and what it is asked to do. That bridge can be:

• A semantic space, where class names, descriptions or attributes are represented as vectors.

• A shared language, where both the task and the answer can be expressed as text.

• A learned procedure, where the model has practised on many tasks and can apply the same procedure to a new one.

An everyday analogy

Suppose someone tells you that a "zebra" looks like a horse with black and white stripes. You have never seen a zebra, yet at the zoo you pick it out at once. You combined two things you already knew, horses and stripes, using a verbal description. Zero-shot learning gives models the same trick: use known concepts and a description to recognise something new.

Variants of the setting

• Conventional zero-shot: test items come only from unseen classes.

• Generalised zero-shot: test items come from both seen and unseen classes. This is harder because models tend to favour the classes they were trained on.

• Zero-shot task transfer: the model is asked to do a new kind of task, such as summarising or translating, from an instruction alone.

• Zero-shot prompting: a language model is given only an instruction, with no examples.

What it is not

• It is not magic. The model has no information about the task other than what the description and pre-training provide.

• It is not the same as unsupervised learning, which finds structure without labels but does not map to named classes.

• It is not always better than a few labelled examples. Adding even a handful of examples often helps a lot.

Where it came from

The idea has two lines of ancestry.

Early attribute-based work

In the late 2000s, researchers proposed learning to recognise classes from attributes. Instead of training a classifier for "zebra", they trained classifiers for "has stripes", "has four legs" and "is black and white", then described each new class as a list of attributes. A paper on learning to detect unseen object classes by between-class attribute transfer (2009) is a well-known example. Related work on zero-data learning appeared around the same time.

Embedding-based methods

Next, researchers mapped images and class names into a shared embedding space. Word vectors, such as those from word2vec (2013), gave class names a numeric meaning. Models like DeViSE (2013) mapped images into the word vector space, so an image of an unseen class would land near the vector of its name.

Large language and vision-language models

Large language models showed in 2019 and 2020 that a single pre-trained network could perform many tasks from an instruction alone. In 2021, CLIP trained an image encoder and a text encoder together on image and caption pairs, which made zero-shot image classification with natural language labels practical. See CLIP. Later, instruction-tuned language models became standard tools for zero-shot prompting.

Tabular data

For tables, zero-shot has a different flavour. A tabular foundation model is pre-trained on many tables, often synthetic, and then predicts on a new table without gradient updates for that table. The model has not seen that table, so in one sense this is zero-shot at the table level. It still reads the labelled rows of the new table as context, so it is few-shot at the row level. See /model/research for how this work developed.

How it works

There are four main mechanisms. They differ in how the bridge between known and unknown is built.

Mechanism 1: Attribute-based classification

1. Define a vocabulary of attributes, such as colour, shape, habitat.

2. Train a model to predict each attribute from the input, using seen classes.

3. Describe each unseen class by its attributes.

4. For a new input, predict its attributes and pick the unseen class whose attribute list matches best.

The analogy is a detective using a suspect description. The witness does not give a name, only features. The detective matches features to candidates.

Mechanism 2: Shared embedding space

1. Choose an encoder for inputs, and an encoder for class descriptions.

2. Train them so that matching pairs sit close together.

3. At test time, embed the input and the descriptions of all candidate classes, including unseen ones.

4. Choose the class whose description is nearest.

The analogy is a library where books and index cards are shelved by meaning. A new card for an unseen topic lands near related books, and a new book finds its card by closeness.

Mechanism 3: Instruction following

A language model is pre-trained on huge amounts of text and then tuned to follow instructions. At test time you describe the task in words, such as "Classify this review as positive or negative", and the model answers. The knowledge of the task comes from the many similar tasks described in its training text.

Mechanism 4: Learned procedures over data

A model is pre-trained on many tasks drawn from a broad distribution, such as many tables generated by simulation. It learns a general procedure for predicting from structured data. At test time, a new table is just one more draw from that distribution, so the model applies the procedure without task-specific training. See synthetic data generation and meta-learning.

Step-by-step view of zero-shot text classification

1. Write labels as short phrases, for example "billing issue" and "delivery delay".

2. Turn each label into a hypothesis or a prompt.

3. Score how well the input text supports each hypothesis.

4. Return the highest score, or all scores as probabilities.

5. Check the margin. If scores are close, route to review.

Why it can fail

• The description is vague or ambiguous.

• The unseen class is far from anything the model has seen.

• The model is biased toward seen classes, which is the generalised zero-shot problem.

• Label wording changes the answer, which makes results fragile.

Common failure patterns in detail

Hubness. In high-dimensional spaces a few label vectors become the nearest neighbour of a large share of inputs. The result is that one or two labels absorb most predictions. Normalising scores per label, or centring embeddings, reduces the effect.

Domain gap. A model trained on web text may not know the jargon of a specialised field. The label "NPL" might mean a loan type to one reader and a language field to another. Adding a short definition to the label helps.

Prompt order and format. Language models can be sensitive to where the label list appears and how it is formatted. Fixing a template and testing a few variants keeps results stable.

Negation and numbers. Embedding models often treat "not late" and "late" as close. Numeric thresholds, such as "over 30 days", are hard to capture in a shared space. Use rules or a model with explicit reasoning for these cases.

When zero-shot is a good first step

• A new category list is being drafted and you want a quick look at its volume.

• You need a baseline to decide whether labelling is worth the cost.

• You want to bootstrap labels, then correct them by hand.

• You need to triage items into rough groups before detailed processing.

When to avoid it

• Decisions with legal or safety impact and no human review.

• Tasks where classes differ in subtle, domain-specific ways.

• Tasks with numeric rules that text similarity cannot express.

Key concepts and terms

A worked example

This example classifies short texts into labels it never saw during training, using a general embedding model and cosine similarity. It illustrates the shared embedding space mechanism.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

labels = [
    "a complaint about billing",
    "a question about delivery time",
    "praise for the product",
]
texts = [
    "I was charged twice this month",
    "When will my parcel arrive?",
    "Fantastic quality, would buy again",
]

L = model.encode(labels, normalize_embeddings=True)
T = model.encode(texts, normalize_embeddings=True)

scores = T @ L.T                 # cosine similarity
pred = scores.argmax(axis=1)
for t, p in zip(texts, pred):
    print(t, "->", labels[p])

Walking through it

• The three labels are plain phrases. The model was never trained on these exact labels.

• Each text and each label becomes a vector in the same space.

• The dot product of normalised vectors equals cosine similarity.

• The label with the highest similarity is the prediction.

A numeric check by hand

Suppose a text vector is (0.8, 0.6) and two label vectors are (0.9, 0.4) and (0.1, 0.99), all roughly normalised. The first similarity is 0.8 times 0.9 plus 0.6 times 0.4, which is 0.96. The second is 0.8 times 0.1 plus 0.6 times 0.99, which is 0.67. The text goes to the first label, since 0.96 is larger.

Improving the same example

• Rewrite labels as full sentences, such as "This message is about a billing problem".

• Average the scores over several phrasings of each label.

• Add a "none of these" label to avoid forcing a bad match.

• Set a threshold on the top score and send low-score items to review.

A zero-shot table example

A pre-trained tabular model can be used without any training on a new dataset's weights.

from tabpfn import TabPFNClassifier

clf = TabPFNClassifier()
clf.fit(X_train, y_train)      # context only, no gradient training
proba = clf.predict_proba(X_test)

This is zero-shot with respect to the table, because the network has never been trained on it. It is not zero-label, since the model still reads the labelled rows. A truly label-free use on tables is different and is covered below.

Strengths and limitations

Strengths

• No labels required for the new task. Useful for cold starts.

• Fast to deploy. You can ship a first version in hours.

• Flexible. Labels can be changed by editing text.

• Scales across many classes. Adding a class means adding a description.

• Good for exploration. It reveals which categories exist before you invest in labelling.

Limitations

• Lower accuracy than supervised models when enough labels exist.

• Sensitive to wording. Small changes in label text can shift results.

• Bias toward seen classes in the generalised setting.

• Weak on domain-specific jargon that the model never met in pre-training.

• Hard to verify. Without labels, you need a small labelled test set just to measure quality.

• Overconfidence. Models may output high scores for wrong answers.

• Limited on structured data. Without examples, a model cannot learn column semantics unique to your table.

The practical rule

Use zero-shot to start and to explore. Measure it on a small labelled sample. If accuracy is not enough, move to few-shot learning or light fine-tuning.

How it compares with alternatives

How to choose

• If you have no labels and a text description, start zero-shot.

• If you can label twenty items, compare zero-shot against few-shot on them.

• If you can label hundreds, train or fine-tune and keep the zero-shot result as a baseline.

• If the domain is specialised, expect zero-shot to need help from examples.

Role in tabular foundation models

Tables are the hardest case for zero-shot learning, because the "description" of a table is its own structure. There is no pretrained vocabulary of column meanings the way there is for words. Still, tabular foundation models use zero-shot ideas in several concrete ways.

Zero-shot at the table level

A tabular foundation model is pre-trained on a very large number of tables so that it can predict on a table it has never seen. No per-table training is needed. In this sense the table is an unseen task and the model is zero-shot on it. Models such as TabPFN and TabICL do this through in-context learning, with the labelled rows given as context. See /model/architecture for how such a model reads rows and columns.

Pre-training on synthetic tables

Because real labelled tables are scarce, much of the pre-training uses generated tables. The generators are built from random causal graphs, random functions and noise. This gives the model a broad prior over how columns relate. See synthetic data generation. The model then treats a real table as another draw from this wide family, which is the learned procedure mechanism above.

Raw tables with no column names

A true zero-shot scenario for tables is a file with unlabelled headers such as c1, c2, c3. The model must handle numbers, categories, dates and gaps with no semantic hints. Strong tabular models infer roles from the values themselves: a column with few distinct integers behaves like a category, a skewed positive column behaves like an amount, and so on. When names are available, a text-aware model such as TabSTAR can use them as extra signal.

Missing values

A zero-shot model must treat missing entries correctly without being told how. Pre-training with random masks, as in masked cell modeling, teaches the model that an empty cell is information, not an error. This also supports zero-shot imputation, where the model fills gaps in a new table with no table-specific fitting. See imputation networks.

Zero-shot anomaly detection

Without any labelled anomalies, a model can score how unlikely a row is under its learned view of the table. Rows that the model finds hard to predict or reconstruct are flagged. This links to anomaly detection and to classic detectors such as Isolation Forest. Thresholds still need a small labelled sample or a business rule.

Sector and semantic classification

Zero-shot works well for tagging. A table or a column can be matched to a sector such as "healthcare" or a semantic type such as "date of birth" by comparing its content and names with text descriptions in a shared space. See semantic sector heads. This needs no examples for each new sector, only a label description.

Continual learning

A zero-shot capable model reduces the need for frequent retraining, so there is less risk of forgetting. When weights must be updated, methods such as experience replay and EWC help preserve earlier skills.

The boundary with few-shot

In tabular practice, zero-shot and few-shot blur. Giving the model the labelled rows of a new table moves it from zero-shot to few-shot at the row level. Teams often start with a model that has seen no data from their domain, check it on a small labelled sample, and then add examples. For measured behaviour, see /model/benchmarks.

Practical guidance and best practices

Write good label descriptions

• Make labels specific and mutually exclusive.

• Use full phrases rather than single words.

• Add a short definition when the term is ambiguous.

• Include a catch-all label for items that fit nothing.

Always keep a small labelled test set

• Even fifty labelled items give a useful signal.

• Sample them at random, not only from easy cases.

• Re-measure after any change to labels or prompts.

Calibrate and threshold

• Look at the score distribution for correct and incorrect predictions.

• Choose a threshold that balances precision and recall for your use.

• Route low-margin items to people.

Test prompt and label variants

• Try several phrasings and compare.

• Average over phrasings to reduce wording sensitivity.

• Fix the final wording and version it.

For generalised settings

• Apply a penalty or calibration to the scores of seen classes.

• Evaluate seen and unseen classes separately.

• Report the harmonic mean of the two accuracies if you need one number.

For tables

• Standardise obvious formats before prediction, such as dates and units.

• Do not remove missing values blindly. Let the model see the mask.

• Check that no identifier column leaks the target.

• Compare to a simple tree-based baseline such as XGBoost on a small labelled sample.

Move up the ladder when needed

1. Start with zero-shot.

2. Add a few examples and measure the change.

3. Add active learning to choose which rows to label next.

4. Fine-tune only if the gain justifies the cost.

Governance notes

• Log the exact label text and model version for each prediction.

• Review a sample of predictions regularly.

• Document known failure cases and share them with users.

Common mistakes

• Trusting zero-shot output without a test set. You cannot tell quality by looking at a few examples.

• Vague labels. "Other" and "misc" attract many items and hide errors.

• Overlapping labels. If two labels mean nearly the same, scores split randomly.

• Ignoring the generalised setting. Real data mixes seen and unseen cases.

• Assuming one prompt is final. Wording changes can move accuracy by a lot.

• Treating scores as probabilities. Similarity scores are not calibrated.

• Using zero-shot for high-stakes decisions alone. Add review or labelled evaluation.

• Forgetting data leakage in tables. An ID or timestamp column may reveal the target.

• Skipping error analysis. Group errors by label to see where the descriptions fail.

• Comparing with a weak baseline. Always compare to a simple supervised model when you have labels.

Frequently asked questions

What does zero-shot mean?

Zero-shot means the model sees zero labelled examples of the task or class at the moment it must solve it. It relies on a description, an instruction or general knowledge from pre-training. The word shot refers to an example, so zero-shot means no example was shown.

How can a model recognise something it was never trained on?

It uses shared knowledge. If the model knows what stripes and horses are, and you describe a zebra as a striped horse, it can combine those ideas. In modern systems, text and images or text and tables are mapped to a shared space where a new description lands near matching inputs.

Is zero-shot learning as accurate as supervised learning?

Usually not when plenty of good labels exist. Supervised models fit the exact task, so they tend to win on accuracy. Zero-shot wins on speed and flexibility, and it can be surprisingly strong when the task is close to what the model saw in pre-training.

What is generalised zero-shot learning?

It is the harder setting where test items come from both seen and unseen classes. Models often over-predict the seen classes because they were trained on them. Evaluating seen and unseen accuracy separately, and adjusting scores, helps correct this bias.

Can zero-shot learning work on tabular data?

Yes, in specific ways. A tabular foundation model pre-trained on many tables can predict on a new table with no per-table training. It still usually reads labelled rows as context, which makes it few-shot at the row level. Label-free tasks such as imputation and anomaly scoring are closer to pure zero-shot.

What is the difference between zero-shot and unsupervised learning?

Unsupervised learning finds structure such as clusters without any labels, but it does not name the structure. Zero-shot learning produces named answers by linking inputs to descriptions. One discovers groups, and the other assigns known names to inputs.

How should I write labels for zero-shot classification?

Use clear, specific phrases and avoid overlapping meanings. Full sentences often work better than single words because they carry more context. Add a catch-all label for items that do not fit, and test a few phrasings to see how stable the results are.

How do I evaluate a zero-shot system without labels?

You need at least a small labelled sample, even if the system itself uses none. Fifty to a few hundred randomly chosen items give a reasonable signal. Without this check, it is easy to believe a system works when it fails on important cases.

When should I switch from zero-shot to few-shot?

Switch when accuracy on your labelled sample is below what the task needs, or when errors cluster on specific labels. Adding a few well-chosen examples usually brings a large gain for little effort. Choosing those examples with active learning makes the gain larger.

Does zero-shot learning remove the need for pre-training?

No, it depends on pre-training even more than other methods. The model can only bridge to new tasks using what it learned before. Better, broader pre-training, including diverse synthetic data for tables, directly improves zero-shot performance.

Glossary

• Attribute: a named property used to describe classes.

• Cosine similarity: a measure of the angle between two vectors.

• Embedding: a vector that captures meaning.

• Foundation model: a large model pre-trained on broad data and reused for many tasks.

• Generalised zero-shot: a test set that mixes seen and unseen classes.

• Hubness: a high-dimensional effect where a few points are close to many others.

• In-context learning: adapting by reading examples or instructions in the input.

• Instruction tuning: training a language model to follow written instructions.

• Label description: text that defines a class.

• Prompt: the input text that frames a task.

• Seen class: a class present in training data.

• Semantic space: a vector space where distance reflects meaning.

• Shot: a labelled example given for a task.

• Unseen class: a class absent from training data.

• Verbalizer: a map from model outputs to labels.

• Zero-shot: solving a task with no task-specific examples.

Summary

Zero-shot learning lets a model handle new classes and tasks with no task examples by connecting a description to general knowledge. It works through attributes, shared embedding spaces, instruction following, and learned procedures over broad data. It is fast and flexible, but it is less accurate than supervised training when labels exist, it is sensitive to wording, and it needs a small labelled test set to be trusted.

For tables, zero-shot ideas appear in pre-trained models that predict on unseen tables with no per-table training, trained on diverse and often synthetic tables. They support work on raw tables with no column names, missing values, imputation, anomaly scoring, and sector or semantic tagging, and they blend into few-shot practice once labelled rows are given as context. Related reading includes few-shot learning, meta-learning, TabPFN, /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: