In-context learning

Overview

In-context learning (ICL) is when a trained model picks up a task from examples placed in its input and applies it to a new case, all in one forward pass, without any change to its weights. You show a language model three examples of English words with their French translations, then a fourth English word, and it writes the French. You give a tabular model a few thousand labelled rows and some unlabelled ones, and it predicts the missing labels. Nothing is trained at that moment. The "learning" happens inside the computation.

This is different from the usual meaning of learning in machine learning, where examples are used to update weights by gradient descent. In ICL, the weights were trained long before, on a broad mix of data, and what they learned is a general way to read examples and act on them. The examples are then just part of the input, like the question itself.

An everyday picture: a new temporary worker at a post office is shown three parcels being sorted into bins, with a word of explanation, and then sorts the fourth correctly. Nobody rewired their brain. They used years of general experience to infer the rule from a short demonstration. When the shift ends, the specific rule is gone, but the ability to pick up rules quickly remains. That is in-context learning.

What it is

The number of demonstrations names the setting. Zero-shot means an instruction but no examples. One-shot means a single example. Few-shot usually means a handful to a few dozen. Many-shot means hundreds or thousands, which long context windows now allow. These terms predate language models: few-shot learning and zero-shot learning were research areas in computer vision, where they usually involved some training on the new classes or on descriptions of them.

Where it came from

The idea of a model that learns how to learn is older than the name. Jürgen Schmidhuber wrote about meta-learning in 1987, and Hochreiter, Younger and Conwell showed in 2001 that a recurrent network could be trained so that its hidden state learned new functions from a stream of examples. That is in-context learning in a recurrent network. Zero-shot recognition of unseen classes from attributes was studied by Lampert, Nickisch and Harmeling in 2009, and one-shot learning of object categories by Fei-Fei, Fergus and Perona in 2006.

The term and the modern interest come from language models. Brown and colleagues' 2020 paper "Language Models are Few-Shot Learners" introduced GPT-3, a 175-billion-parameter model, and showed that it could perform many tasks in zero-, one- and few-shot settings with demonstrations given only as text and no gradient updates. Few-shot performance improved strongly with model size.

Explanations followed. Xie and colleagues (2021) framed ICL as implicit Bayesian inference over a latent concept. Garg and colleagues (2022) trained Transformers from scratch to learn simple function classes such as linear functions in context, with accuracy comparable to least squares. Akyürek and colleagues (2022) and von Oswald and colleagues (2023) showed that Transformers can implement learning algorithms such as gradient descent on linear regression inside their forward pass. Olsson and colleagues' 2022 paper "In-context Learning and Induction Heads" linked a specific attention circuit to ICL. Chan and colleagues (2022) showed that properties of the training data, such as bursty, clustered examples, help ICL emerge.

In tables, Müller and colleagues' 2022 prior-data fitted networks and TabPFN (2023) made in-context learning the main prediction mechanism, followed by larger successors and models such as TabICL (2025).

How it works

The basic mechanism

A model trained to predict the next token, or the missing label, learns to use everything in its input. When the input contains several input-output pairs following one pattern, the most useful thing to do is to infer that pattern and apply it to the last input. Training rewards this, so the model gets good at it.

In-context prediction:
    ŷ = argmax_y  p_θ( y | x₁, y₁, x₂, y₂, …, x_k, y_k, x_query )

Bayesian view (implicit inference over a latent task c):
    p(y | prompt) = Σ_c  p(y | x_query, c) · p(c | x₁, y₁, …, x_k, y_k)

One gradient step on in-context linear regression (what some attention layers can emulate):
    W ← W − η · Σ_i (W · x_i − y_i) · x_iᵀ

In the first line θ are the fixed weights, (x_i, y_i) are the k demonstrations, x_query is the new input and ŷ the predicted output. In the Bayesian line, c is an unseen task or concept (for example "translate English to French" or "the label depends on column 3"), p(c | …) is how strongly the examples point to each concept, and p(y | x_query, c) is the answer under that concept. More examples sharpen p(c | …) towards the right concept. In the last line W is a linear model's weight matrix, η a step size and x_iᵀ the transpose of x_i; the result shows that one step of least-squares gradient descent can be written as a sum over examples, which is the kind of computation linear attention performs.

The meta-learning view

Pretraining on a vast variety of text, or on millions of synthetic tables, is like training on millions of small tasks. Each document or table is its own little dataset. The model is therefore trained to learn quickly from whatever task appears in its input. This is meta-learning: learning a learning procedure. The procedure lives in the weights; each specific task lives only in the context.

The Bayesian inference view

Under the Bayesian view, pretraining teaches the model a prior over the kinds of tasks and patterns that occur. The demonstrations are evidence. The model's output approximates the posterior prediction: what is most likely, given everything seen, under that prior. Prior-data fitted networks for tables make this explicit by training on datasets sampled from a chosen prior, so the network's predictions approximate the Bayesian posterior predictive for data like that prior.

Induction heads, in plain words

An induction head is a pair of attention heads working together. One head notes, for each token, which token came just before it. The other head, when it sees the current token, looks back for earlier places where the same token appeared and copies what came next. In a sequence like "Mr Okafor … Mr" it predicts "Okafor". This is a simple pattern-completion skill: "after A came B before, so after A comes B again". Olsson and colleagues found that induction heads form during a sharp change early in training, at the same time as a jump in in-context learning ability, and argued they are an important mechanism for it, especially in smaller models. In larger models, more abstract versions of the same idea, matching on meaning rather than exact tokens, likely contribute.

What the demonstrations actually do

Experiments show ICL is not always learning the mapping from scratch. Min and colleagues (2022) found that replacing demonstration labels with random labels often hurt performance only a little, suggesting that demonstrations mainly tell the model the format, the label set and the input distribution, while the model relies on prior knowledge for the mapping. Wei and colleagues (2023) found that larger models can follow flipped or arbitrary labels in context more readily than smaller ones, so the balance between prior knowledge and in-context evidence shifts with scale. Zhao and colleagues (2021) showed biases towards the most frequent label in the prompt and towards the last example, and Lu and colleagues (2022) showed that the order of examples can swing results widely.

In tabular models

In a tabular foundation model, the "prompt" is the training table. Labelled rows and unlabelled rows go in together; attention across rows lets each unlabelled row draw on similar labelled rows and their targets. There are no tokens and no instruction; the task is defined by which column is the target. Because row order has no meaning, these models avoid positional information across rows. See in-context prediction for a step-by-step account.

Context length limits

Every model has a maximum input size. For language models it is a token count; long demonstrations or many of them use it up and raise cost, since each call reprocesses the context (caching, such as a KV cache, helps for repeated prefixes). Attention cost grows roughly with the square of input length. Models also do not use all positions equally: Liu and colleagues' "Lost in the Middle" (2023) found that information in the middle of a long context is used less reliably than at the start or end. Agarwal and colleagues (2024) showed that many-shot ICL with hundreds or thousands of examples can keep improving results when the window allows. For tabular models the limit is the number of rows and columns that fit in memory, which sets a practical cap on table size.

Key concepts and terms

A worked example

An illustrative online shop wants to tag incoming support messages as "billing", "delivery", "returns" or "other".

1. Zero-shot: an instruction listing the four labels and a one-line description of each. On 300 labelled test messages, accuracy is 78 per cent.
2. Few-shot: add eight demonstrations, two per label, chosen to be typical and clearly worded. Accuracy rises to 86 per cent.
3. Order check: shuffle the eight demonstrations five times. Accuracy ranges from 83 to 87 per cent, so order matters a little; the team fixes one balanced order with labels interleaved.
4. Retrieval of examples: for each new message, pick the eight most similar labelled messages from a pool of 2,000. Accuracy rises to 90 per cent.
5. Decision: the shop handles 50,000 messages a day. Long few-shot prompts on every call are costly, so they compare with a small fine-tuned classifier. That reaches 92 per cent at much lower cost per message, so they fine-tune for the high-volume task and keep ICL for new categories that appear before enough labelled data exists.

A prompt from step 2 might look like this:

Classify each message as billing, delivery, returns or other.

Message: I was charged twice for order 4471.
Label: billing

Message: The parcel says delivered but nothing arrived.
Label: delivery

Message: How do I send back shoes that are too small?
Label: returns

Message: My card payment failed but the money left my account.
Label:

The numbers above are illustrative.

Strengths and limitations

Strengths:

• No training run: a new task is ready as soon as you write the prompt or pass the table.
• Easy to change: edit the examples and behaviour changes immediately.
• Works with very little labelled data.
• One model serves many tasks at once.
• In tabular models, gives strong results on small tables in seconds.

Limitations:

• Every call reprocesses the examples, which costs compute and latency at high volume.
• Bounded by the context window; large training sets do not fit.
• Sensitive to the choice, format and order of examples.
• Biased towards frequent or recent labels in the prompt.
• Nothing is retained between calls; the model does not improve with use.
• Can underperform fine-tuning when plenty of labelled data is available.

How it compares with alternatives

Comparisons between ICL and fine-tuning depend on model size and evaluation details. Mosbach and colleagues (2023) found that, when compared fairly, few-shot fine-tuning generalises as well as or better than in-context learning in many settings. Liu and colleagues (2022) found that parameter-efficient fine-tuning could be both more accurate and cheaper than few-shot ICL. ICL's real advantage is speed of setup and flexibility.

Practical guidance and best practices

1. Start zero-shot with a clear instruction and list of allowed outputs; add examples only if needed.
2. Pick demonstrations that are representative, correct and cover every label.
3. Balance labels and interleave them to reduce frequency and recency bias.
4. Keep the format of examples and query identical, including spacing and label words.
5. Test several orders and example sets; report the spread, not the best run.
6. Retrieve the most similar examples for each query when you have a labelled pool.
7. Put key instructions at the start or end of long contexts, not buried in the middle.
8. Track cost per call; at high volume, compare against a fine-tuned model.
9. For tabular models, respect the documented row and column limits and evaluate on several splits.

Common mistakes

• Using examples that are all from one class, so the model leans towards that label.
• Testing on the same examples that appear in the prompt.
• Judging a prompt from one run and one order of examples.
• Filling the context with many weak examples instead of a few good ones.
• Assuming the model learned the rule when it may be relying on prior knowledge.
• Ignoring cost and latency until the prompt is in production.

Frequently asked questions

Does in-context learning change the model?

No. The weights stay exactly the same. The examples affect only the computation for that one input. As soon as the next request arrives without them, the behaviour they induced is gone. Persistent change requires training, such as fine-tuning.

How many examples should I use?

Often three to ten well-chosen examples capture most of the gain for simple classification or formatting tasks. Harder tasks and long-context models can benefit from many more. The best number is found by testing on held-out data, watching both accuracy and cost.

Why does in-context learning work at all?

Pretraining on enormous, varied data means the model has effectively practised inferring patterns from short sequences millions of times. Researchers explain the result as meta-learning, as implicit Bayesian inference over likely tasks, and through specific mechanisms such as induction heads and attention layers that behave like simple learning algorithms. These views are complementary rather than competing.

Is in-context learning the same as few-shot learning?

Few-shot learning is the broader goal of learning from few examples, and older methods achieved it with some training on the new classes. Few-shot in-context learning is one way to reach that goal: the examples go into the input and no training happens.

Why do tabular models use in-context learning?

Tables are often small, so training a large network per table would overfit. A model pretrained across many tables, synthetic or real, can instead treat the training rows as context and predict new rows in one pass. This gives strong accuracy on small and medium tables with no tuning.

What limits in-context learning?

The context window limits how many examples fit, cost grows with context length, models use the middle of long contexts less reliably, and results depend on example choice and order. When labelled data is plentiful and the task is stable, fine-tuning usually wins on accuracy and cost.

Glossary

• In-context learning: performing a task from examples in the input, without weight updates.
• Demonstration: one example input and output shown in the prompt.
• Zero-shot learning: handling a task or class with no examples of it.
• Few-shot learning: learning a task from a small number of examples.
• Induction head: attention heads that find an earlier match and copy what followed.
• Meta-learning: learning how to learn new tasks quickly.
• Context window: the maximum input length a model accepts.
• Posterior predictive: the Bayesian prediction after seeing the examples.
• Recency bias: a lean towards the label of the last example.
• Many-shot prompting: using a very large number of examples in a long context.

Summary

In-context learning is a trained model's ability to take up a task from examples in its input and apply it to a new case, with no change to its weights. In language models it appears as zero-, one-, few- and many-shot prompting, made famous by GPT-3 in 2020. In tabular foundation models it is the main prediction mechanism, with the training rows as context. It works because pretraining across countless small tasks teaches a general learning procedure, which can be read as meta-learning or as implicit Bayesian inference and is supported by mechanisms such as induction heads. Its limits are the context window, cost per call, and sensitivity to the choice and order of examples. When data is plentiful and the task stable, fine-tuning is often more accurate and cheaper; when tasks change fast or data is scarce, in-context learning is the quickest way to a working system. For the architecture underneath, see Transformers.

Related

It belongs to Large tabular foundation models in my stack, under Deep learning. Others in the same group: