Meta-learning

Overview

Meta-learning, often called "learning to learn", trains a model across many different tasks so that it can pick up a new task from very little data. Instead of learning one function, the system learns a way of learning. After meta-training, a new task with only a handful of labelled examples can be solved in a few gradient steps, or even with no gradient steps at all.

The idea matters most when data per task is scarce but there are many related tasks. That is the usual shape of tabular work: thousands of small tables, each with its own columns and labels. Meta-learning is the theory behind few-shot learning and a direct ancestor of in-context learners such as TabPFN, which are trained on many synthetic tasks and then predict for a new table in a single forward pass.

What it is

In ordinary supervised learning, one dataset defines one task. The model fits the data and is judged on held-out rows from the same task. In meta-learning, the unit of data is a task. A task is a small dataset with its own support set, used to learn from, and query set, used to test what was learned. The meta-learner sees many such tasks during training and is judged on how well it handles tasks it has never seen.

Two loops make this concrete.

• Inner loop. Adapt to one task using its support set. This could be a few gradient steps, a nearest-prototype rule, or a forward pass through a sequence model.
• Outer loop. Update the meta-parameters so that the inner loop does better across tasks. The outer loss is computed on the query sets.

Think of a student preparing for a quiz show. Memorizing answers is ordinary learning. Practising how to read a new question, find the clue and answer quickly is meta-learning. The student is trained on the skill of picking up new topics fast, not on any single topic.

What gets meta-learned

Different methods meta-learn different things.

Terms you will see

• Episode: one training round built from one task, with a support and a query set.
• N-way K-shot: a classification task with N classes and K labelled examples per class. A 5-way 1-shot task has five classes with one example each.
• Task distribution: the population from which training and test tasks are drawn.
• Meta-train, meta-validation, meta-test: splits of tasks, not of rows.

Where it came from

The idea of improving the learning process itself is old. Early work in the late 1980s and 1990s explored "learning to learn" and "inductive bias transfer". Researchers asked how a system could discover which assumptions help on a family of tasks. Self-referential learners that modify their own rules were studied around the same time. In the early 2000s, work on learning from related tasks, called multi-task learning and lifelong learning, built the foundation.

Deep meta-learning took off in the mid 2010s. Memory-augmented networks showed that a recurrent model could learn to bind new information in a single exposure. "Matching Networks" (2016) introduced an episodic training scheme with an attention-based classifier. "Optimization as a Model for Few-Shot Learning" (2017) learned the update rule with a recurrent network. "Model-Agnostic Meta-Learning", known as MAML (2017), proposed learning an initialization that adapts in a few gradient steps with any differentiable model. "Prototypical Networks" (2017) showed that a very simple rule, classify by the nearest class mean in an embedding space, is hard to beat. "Reptile" (2018) offered a first-order alternative to MAML that avoids second derivatives.

Later work asked whether complex meta-learners were needed at all. Studies found that strong pre-training plus simple fine-tuning often matches or beats episodic methods on image benchmarks. At the same time, large sequence models showed in-context learning, which can be viewed as meta-learning that emerges from training on a broad data stream. "Transformers Can Do Bayesian Inference" (2021) and "TabPFN" (2022) applied this view to tabular data: train a transformer on a huge number of synthetic tasks so that it performs inference for a new task in one pass. See the research page for how this line of work fits the wider picture.

How it works

The core recipe

All meta-learning methods follow the same recipe, with different choices for the inner loop.

1. Define a task distribution. Decide what a task looks like. For tables, a task could be one dataset, or one random sub-table with a random choice of target column.
2. Sample a batch of tasks. Draw several tasks per step.
3. Split each task. Put a few rows in the support set and others in the query set.
4. Adapt on the support set. Run the inner loop to get task-specific behaviour.
5. Score on the query set. Compute the loss of the adapted model on the query rows.
6. Update meta-parameters. Use the average query loss over tasks to change the shared parameters.
7. Repeat until the meta-validation score stops improving.

An analogy: a driving instructor takes a learner through many different cars, one lesson each. After each lesson the learner takes a short test in that same car. The instructor adjusts the teaching method based on all test scores. After many cars, the learner can step into a car they have never driven and be competent within minutes.

Family 1: optimization-based methods

These methods learn a starting point or an update rule. MAML is the best known.

• The inner loop takes a few gradient steps from the shared initialization using the support loss.
• The outer loop differentiates through those steps to find an initialization that, after adaptation, gives low query loss.
• The cost is second-order derivatives, which are heavy for large models. First-order variants drop them with small loss in quality.

The result is a set of weights that sit in a spot where small moves in any task direction produce big gains. In landscape terms, the initialization is a base camp from which every task summit is close.

Family 2: metric-based methods

These methods learn an embedding so that a simple rule works in it.

• Prototypical networks compute one prototype per class, the mean embedding of its support examples. A query is assigned to the closest prototype.
• Matching networks use attention over the support set. A query's label is a weighted vote of support labels.
• Relation networks learn the similarity function itself.

No gradient step happens at test time. Adaptation is just computing means or attention. This makes the methods fast and robust with few examples. It is the same spirit as a librarian who sorts new books by checking which shelf has the most similar neighbours.

Family 3: model-based methods

A model with memory or attention reads the support set and produces predictions directly. Recurrent networks, memory-augmented networks and transformers all fit. The meta-learning is hidden in the weights, which have learned to implement a learning algorithm in the forward pass. This family leads to modern in-context learning.

Family 4: Bayesian and probabilistic methods

These methods treat the task-specific parameters as random variables with a prior learned across tasks. Adaptation becomes posterior inference. Neural processes and related models learn to map a context set to a distribution over functions. They give uncertainty estimates, which matter when support sets are tiny. See probability and Bayesian deep learning for background.

Family 5: pre-training plus fine-tuning

A strong baseline needs no episodes. Train a large model on a broad source with self-supervised learning or supervised learning, then fine-tune on the small target. This path is simple and often competitive. It overlaps with transfer learning.

Family 6: meta-learning of hyperparameters and search

Related ideas learn good starting configurations across datasets. A system can warm-start a search by looking at which settings worked on similar tables. This links to Bayesian optimization and to automated model selection.

Training tricks that matter

• Match train and test conditions. If test tasks have 5 shots, train with about 5 shots. Large mismatches hurt.
• Use a wide task distribution. Narrow tasks produce a meta-learner that cannot generalize to new kinds of tasks.
• Keep episodes diverse. Randomize class subsets, feature order and sizes.
• Regularize the outer loop. Meta-overfitting to the training tasks is common, since the number of tasks is far smaller than the number of rows.
• Normalize carefully. Batch statistics inside an episode can leak query information. Use methods that compute them from the support set only.
• Evaluate on held-out tasks. A random split of rows is not a meta-test.

Method deep dives

MAML step by step

MAML has a compact definition. Let θ be the shared weights. For task i, the inner step is:

θ_i' = θ - α × ∇ loss_support_i(θ)

The outer step then minimizes the query loss at the adapted weights:

θ ← θ - β × ∇ Σ_i loss_query_i(θ_i')

The gradient of the outer loss passes through the inner step, which is where second derivatives appear. In words: do not look for weights that are good on average, look for weights that become good after one small step.

Practical notes:

• Use one to five inner steps. More steps raise memory use, because every step must be kept for back-propagation.
• Learn per-layer or per-parameter inner learning rates if the model is large. This often helps stability.
• Adapt only the last layers if the network is big. This is cheaper and less prone to overfitting a tiny support set.
• Use first-order variants when memory is tight. They give most of the benefit in many settings.

Prototypical networks step by step

1. Embed every support row with a shared encoder.
2. For each class, average its support embeddings to get a prototype.
3. For a query, compute the distance to each prototype, usually squared Euclidean.
4. Turn negative distances into class probabilities with a softmax.
5. Train the encoder with cross-entropy on the query labels.

Because the classifier has no extra parameters, there is little to overfit. It also handles a different number of classes in each episode, which makes it flexible for tasks whose label sets change.

Model-based learners step by step

1. Write each support example as a token that holds features and, for labelled rows, the label.
2. Write each query example as a token with features only.
3. Let attention move information from support tokens to query tokens, and optionally among support tokens.
4. Read the prediction from the query tokens.
5. Train the whole network by gradient descent on the query loss, over a vast number of tasks.

There is no inner-loop optimizer in the code. The network has learned, in its weights, how to turn examples into predictions. That is why this family is called amortized.

Neural processes and probabilistic views

A neural process encodes the context set into a summary vector, then decodes a predictive distribution at new inputs. The summary is order-invariant, as a set should be. The model can output wide error bars when the context is small, and narrow ones when it is rich. This behaviour is the one you want from a few-shot predictor, because it says how much to trust the answer.

Designing a task distribution for tables

The quality of a meta-learner is capped by its task distribution. For tabular work, think about the following axes and make sure each varies widely during training.

A good habit is to sample the settings from broad priors, then inspect random draws by eye. If many tasks look alike, widen the priors.

How to evaluate a meta-learner

An honest evaluation protocol has these elements.

1. Task-level splits. Meta-train, meta-validation and meta-test tasks come from different tables, customers or sites.
2. Fixed shot counts. Report results at several support sizes, for example 5, 20 and 100 rows, to show the learning curve.
3. Strong baselines. Include tuned gradient boosted trees, a fine-tuned pre-trained model and a simple nearest-neighbour rule.
4. Many test tasks. Averages over a handful of tasks are noisy. Use enough tasks to give tight intervals.
5. Cost reporting. Include time per task for adaptation and prediction, not only accuracy.
6. Robustness checks. Test with shifted tasks, noisy labels and extra irrelevant columns.

See the benchmarks page for how comparisons across many datasets are framed.

When meta-learning is the wrong tool

• You have one large task with plenty of labels. Standard training is simpler.
• Your tasks are very unlike each other and share no structure. There is nothing to meta-learn.
• You cannot construct enough tasks, real or synthetic.
• You need a fully explainable model whose rules can be audited line by line. A simple model may serve better, perhaps with SHAP for explanation.

Key concepts and terms

A worked example

Consider a simple meta-learning problem: every task is a regression on a line y = a × x + b, with a different slope and intercept per task. The meta-learner should predict well after seeing only five points from a new line.

A numeric walk-through of the idea

Take two training tasks.

• Task 1: y = 2x + 1.
• Task 2: y = -x + 3.

A model trained on the pooled data of both tasks learns an average line, roughly y = 0.5x + 2. It is wrong on both. A meta-learner that has seen many lines learns something better: an initialization from which five points are enough to fit the line. After seeing five points from Task 1, a few gradient steps move the weights to a ≈ 2 and b ≈ 1. The shared knowledge is "the world is made of lines", so adaptation only has to find two numbers.

Code: a first-order meta-learner on sine-like tasks

The sketch below uses a Reptile-style update with a small network. It is meant to show the loops, not to be a tuned system.

import copy
import numpy as np
import torch
from torch import nn

def make_task(rng):
    amp = rng.uniform(0.5, 3.0)
    phase = rng.uniform(0, np.pi)
    def sample(n):
        x = rng.uniform(-5, 5, size=(n, 1)).astype("float32")
        y = (amp * np.sin(x + phase)).astype("float32")
        return torch.from_numpy(x), torch.from_numpy(y)
    return sample

model = nn.Sequential(nn.Linear(1, 40), nn.ReLU(),
                      nn.Linear(40, 40), nn.ReLU(),
                      nn.Linear(40, 1))
meta_lr, inner_lr, inner_steps = 0.1, 0.01, 5
rng = np.random.default_rng(0)

for step in range(2000):
    task = make_task(rng)
    fast = copy.deepcopy(model)                      # inner copy
    opt = torch.optim.SGD(fast.parameters(), lr=inner_lr)
    for _ in range(inner_steps):                     # inner loop on support
        x, y = task(10)
        loss = nn.functional.mse_loss(fast(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    # outer loop: move shared weights toward the adapted weights
    with torch.no_grad():
        for p, q in zip(model.parameters(), fast.parameters()):
            p += meta_lr * (q - p)

# Test: adapt to a new task with five points.
task = make_task(rng)
fast = copy.deepcopy(model)
opt = torch.optim.SGD(fast.parameters(), lr=inner_lr)
xs, ys = task(5)
for _ in range(10):
    loss = nn.functional.mse_loss(fast(xs), ys)
    opt.zero_grad(); loss.backward(); opt.step()
xq, yq = task(100)
print("query MSE:", nn.functional.mse_loss(fast(xq), yq).item())

Reading the code

• make_task defines the task distribution. Each call is a different sine wave.
• The inner loop copies the model and trains the copy on a few points. That is the adaptation.
• The outer update nudges the shared weights toward the adapted copy. This is Reptile. MAML would instead differentiate the query loss through the inner steps.
• At test time, a fresh task is adapted with five points and judged on 100 unseen points.

A tabular version of the same idea

For tables, a task might be built as follows.

def sample_table_task(tables, rng, n_support=32, n_query=64):
    df = tables[rng.integers(len(tables))]          # pick a table
    target = rng.choice(df.columns)                 # pick a target column
    rows = df.sample(n_support + n_query, random_state=int(rng.integers(1e9)))
    X = rows.drop(columns=[target])
    y = rows[target]
    return (X.iloc[:n_support], y.iloc[:n_support],
            X.iloc[n_support:], y.iloc[n_support:])

Random choice of table, target and rows makes the task distribution wide. A model meta-trained on such episodes learns general habits, such as "numeric columns with a skewed distribution are often log-like", that help on a new table.

A nearest-prototype example in numbers

Suppose an embedding maps rows to two numbers. A 2-way 3-shot task has these support embeddings.

The prototypes are the class means: A = (1.0, 1.0) and B = (3.0, 3.0). A query at (1.5, 1.4) is closer to A, since the squared distance to A is 0.25 + 0.16 = 0.41, against 2.25 + 2.56 = 4.81 for B. The prediction is A. The learning happened earlier, in building an embedding where such clusters are tight.

Strengths and limitations

Strengths

• Strong results with very few labelled examples per task.
• Fast adaptation, often a few steps or none.
• Uses the structure shared across many tasks, instead of treating each as new.
• Natural fit for settings with many small datasets, such as per-customer or per-site tables.
• Leads to amortized methods that replace per-task training with a single forward pass.
• Can carry uncertainty estimates when built in a Bayesian way.

Limitations

• Needs a meaningful task distribution. If test tasks differ from training tasks, gains shrink.
• Many tasks are required. Building them is a design problem in its own right.
• Second-order methods are costly, and training can be unstable.
• Meta-overfitting is common when the number of tasks is small.
• Simple baselines, such as strong pre-training with fine-tuning, are often as good.
• Evaluation is easy to get wrong, because splits must be at the task level.
• Interpretability is limited. It is hard to say what exactly the meta-learner has learned.

How it compares with alternatives

A useful way to see it: meta-learning improves the learner, domain adaptation improves the fit to a new distribution, and active learning improves which labels you ask for. They combine well.

Role in tabular foundation models

Tabular foundation models aim to learn from raw tables in general and then work on a new table with little or no training. This is a meta-learning problem by definition. The task distribution is the world of tables, and the new table is the new task.

Pre-training as meta-training

In a tabular foundation model, pre-training can be seen as meta-training. Each pre-training example is a whole dataset, split into context rows and query rows. The model must predict the query labels given the context. Over millions of such episodes, the network learns a general-purpose prediction procedure. When a real table arrives, the context is the labelled rows of that table, and the model predicts for the rest. Models such as TabPFN and TabICL follow this pattern, as described on the architecture page.

Synthetic tasks as the task distribution

A large, diverse task distribution is the hardest ingredient. Real tables are limited, licensed and uneven. A solution is to generate tasks from priors over data-generating processes: random structural causal graphs, random mixtures, random noise and random missingness. The resulting tasks differ in size, in column types, in relationships and in difficulty. This is a deliberate design of the task distribution, and it is why synthetic data generation is central to the field. Real tables can be mixed in to cover patterns that simulators miss.

In-context learning as amortized adaptation

In a model-based meta-learner, the inner loop is a forward pass. The labelled rows go in as tokens and the prediction comes out, with no weight change. This is the in-context learning that tabular foundation models use. Its main benefit is speed and simplicity: no tuning per table, no learning rate, no early stopping. Its main cost is that the context must fit in the model's window, so very large tables need sampling or retrieval of the most relevant rows.

Handling mixed types and missing values

A task in the wild has numeric, categorical, date and text columns, with gaps. A meta-learner for tables must put all of these into one representation. Common choices are per-column encoders that look at the value distribution, learned embeddings for categories that depend on frequency and not on spelling, and a mask token for missing cells. Training with random masks, as in masked cell modeling, makes missing values part of the task distribution, so the model sees them often and learns to cope.

No column names

New tables often lack useful names. A meta-learned model that relies on names fails there. Models built for tables use structure instead: attention across rows and across columns, which treats columns as an unordered set. Shuffling column order during meta-training makes the model invariant to it. Where names exist, they can be added as an extra signal through a text encoder, which links to semantic sector heads for classification of table meaning.

Imputation and anomaly detection as tasks

The same machinery solves other table tasks by redefining the query. For imputation, the query is a hidden cell and the context is the rest of the table. See imputation networks. For anomaly detection, the context describes normal behaviour, and rows that the model finds unlikely are flagged. Because adaptation uses the context, the same model gives site-specific baselines without retraining.

Continual learning

A foundation model is not static. New task types appear, and new data arrive. Meta-learning ideas help here: learning an initialization that is easy to adapt, and learning to adapt without forgetting. Methods such as elastic weight consolidation and learning without forgetting protect old skills while a model takes on new ones. Replay of earlier episodes keeps the task distribution broad.

Sector or semantic classification

When the goal is to say what a table is about, such as a sector or domain, each table or column is a small classification task with few labelled examples. Prototype-style heads on top of table embeddings are a natural fit, because a new sector can be added by supplying a few example tables. This is few-shot classification in the metric-based family.

Where the analogy ends

Not every foundation model is a pure meta-learner. Some are trained as large pre-trained encoders and fine-tuned, closer to transfer learning. The best way to judge a model is by held-out-task results, as discussed on the benchmarks page, not by the label used to describe its training.

Practical guidance and best practices

1. Start with a baseline. Compare against gradient boosted trees and a fine-tuned pre-trained model before adopting a meta-learner.
2. Design the task distribution first. Spend effort on how tasks are drawn. Cover sizes, column types, noise levels and missing patterns.
3. Split by task. Hold out whole tasks, such as whole tables or customers, for validation and testing.
4. Match shots. Train on episode sizes close to what you expect in production.
5. Prefer simple inner loops. Prototypes, attention and first-order updates are cheaper and more stable than full second-order MAML.
6. Watch for meta-overfitting. If training-task scores climb while held-out tasks stall, add task diversity or regularization.
7. Control context size. For in-context models, select the most informative context rows, using clustering or nearest-neighbour retrieval, when a table is larger than the window.
8. Test on shift. Include tasks that differ from the training distribution, and measure the drop. See domain adaptation.
9. Combine with labelling strategy. Use active learning to choose which few rows to label for the support set.
10. Keep experiments reproducible. Log task seeds, split definitions and versions of the generators.

A small configuration sketch

meta_training:
  task_distribution:
    source: synthetic_plus_real
    tables_per_batch: 16
    support_rows: [16, 256]
    query_rows: 128
    column_shuffle: true
    mask_rate: [0.0, 0.3]
  inner_loop: in_context
  outer_optimizer:
    name: adamw
    lr: 3.0e-4
  validation:
    split_by: table
    heldout_tables: 200
  logging:
    server: tracking.example.internal

Common mistakes

• Splitting by rows, not tasks. This leaks information and gives results that do not hold on new tasks.
• Too narrow a task distribution. The meta-learner then fits only a small corner of the problem.
• Huge mismatch between train and test shots. A model trained on 20-shot episodes may fail at 2 shots.
• Using batch statistics across the support and query sets. This lets the query leak into the adaptation.
• Ignoring simple baselines. A tuned tree model or fine-tuned encoder often wins, and you would not know without testing it.
• Over-tuning the inner loop on test tasks. Tune only on meta-validation tasks.
• Treating meta-learning as magic. It cannot create information that is not in the support set. With too few or noisy examples, it fails like any other method.
• Forgetting calibration. Few-shot predictions can be badly calibrated. Check and recalibrate.
• Letting identifiers act as shortcuts. Row ids or table ids in the features let the model cheat inside training tasks.
• Skipping the diversity audit. Check that sampled tasks really differ in size, type and difficulty, not just in seed.

Frequently asked questions

What is meta-learning in simple terms?

Meta-learning is training a model on many different tasks so that it becomes good at learning new tasks quickly. Instead of memorizing one problem, it learns a method or a starting point that works across problems. After this training, a new task with only a few labelled examples can often be solved in a few steps or with a single forward pass.

How is meta-learning different from transfer learning?

Transfer learning takes a model trained on one large source and adapts it to a target task, usually by fine-tuning. Meta-learning trains explicitly for adaptation, using many tasks and episodes that mimic the low-data situation. In practice the two overlap, and a strong pre-trained model with fine-tuning is a competitive baseline for many meta-learning problems.

What is MAML?

MAML stands for Model-Agnostic Meta-Learning. It learns an initialization of model weights such that a few gradient steps on a small support set give good performance on a new task. It works with any model trained by gradient descent. The full method needs second-order derivatives, and cheaper first-order variants exist.

What does N-way K-shot mean?

It describes a classification episode with N classes and K labelled examples for each class. A 5-way 1-shot task has five classes and one example per class. The query set then asks the model to classify further examples from those same classes.

How does meta-learning relate to in-context learning?

In-context learning can be seen as a form of meta-learning where the inner loop is a forward pass. A model trained on a broad stream of tasks learns to read examples in its input and infer the rule, without changing its weights. Tabular models that take labelled rows as context and predict for new rows work this way.

Why is meta-learning useful for tables?

Tabular work often involves many small datasets, each with its own columns and labels, and few rows per dataset. A meta-learner can use patterns shared across tables, such as how skewed numbers behave or how categories relate to targets, to do well on a new table with little data. That reduces tuning and labelling per table.

Does meta-learning need synthetic data?

Not necessarily, but a rich task distribution is essential and real tasks can be limited. Synthetic generators can create millions of varied tasks cheaply, with known structure. Many approaches mix synthetic and real tasks to cover both controlled variety and real-world quirks.

What is meta-overfitting?

Meta-overfitting happens when the meta-learner memorizes the training tasks instead of learning a general way to learn. Training episodes look good, but new tasks do poorly. Remedies include a wider task distribution, held-out task validation, regularization and simpler inner loops.

Is meta-learning better than gradient boosted trees on tabular data?

Not always. Gradient boosted trees are strong, fast baselines, particularly with plenty of rows. Meta-learned and in-context models tend to shine on small tables where data per task is limited. The right answer comes from testing both on held-out tasks that look like your real workload.

How many tasks do I need for meta-training?

There is no fixed number, but more diverse tasks are better, and the count is usually in the thousands to millions for large models. Diversity matters more than raw count. A million near-duplicate tasks teach less than a thousand truly different ones.

Glossary

• Amortized inference: Paying a large training cost once so that each new inference is a single cheap pass.
• Episode: One meta-training step built from one task.
• First-order method: An update that ignores second derivatives.
• In-context learning: Learning from examples placed in the input, with no weight updates.
• Inner loop: The fast adaptation to a single task.
• MAML: An algorithm that learns an initialization for fast gradient-based adaptation.
• Meta-overfitting: Fitting the training tasks too closely.
• Metric learning: Learning an embedding in which distance reflects similarity.
• N-way K-shot: A task with N classes and K examples per class.
• Outer loop: The slow update of shared parameters across tasks.
• Prototype: The mean embedding of a class in a support set.
• Query set: The held-out part of a task used for the meta-loss.
• Reptile: A first-order meta-learning algorithm.
• Support set: The labelled examples a task provides for adaptation.
• Task distribution: The population from which tasks are drawn.

Summary

Meta-learning trains a model across many tasks so that it can learn a new task quickly from little data. The methods fall into a few families: learn a good starting point, learn a good embedding, learn a procedure that reads examples directly, or learn a probabilistic prior. The key design question is the task distribution, and the key discipline is splitting and testing at the task level.

For tabular foundation models, the link is direct. Pre-training on a wide mix of synthetic and real tables is meta-training, and in-context prediction is the inner loop. Handling mixed types, missing values and unnamed columns becomes part of the task distribution, and imputation, anomaly detection and semantic classification become different queries on the same context. Read more on the architecture, research and benchmarks pages.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: