What is In-context learning? Definition and meaning
The ability of a trained model to perform a new task from examples or instructions in its input, without changing its weights.
Definition
In-context learning: The ability of a trained model to perform a new task from examples or instructions in its input, without changing its weights.
In short: Learning from the prompt alone.
When it is used
Few-shot prompting of language models, and prediction on new tables by passing training rows as context.
Where it fits
In-context learning belongs to Large tabular foundation models in the Deep learning topic. These are the other terms in the same group:
- Tabular foundation models: A large neural network pretrained across many tables so it can predict, impute or score new tables with little or no task-specific training.
- TabPFN: A transformer pre-trained on synthetic datasets that predicts in a single forward pass with in-context learning.
- TabICL: A tabular foundation model that adds column-then-row attention to scale in-context learning to bigger datasets.
- TabSTAR: A tabular model that reads textual features through a pretrained language encoder together with numbers.
- XTab: A framework that pre-trains shared transformer layers across many tables with table-specific embeddings.
- SAINT: A transformer that attends across features and across rows, with contrastive pre-training.
- FT-Transformer: Turns each feature into an embedding and processes them with a transformer.
- TabNet: A deep model that selects features step by step with sequential attention.
- Masked cell modeling: The tabular counterpart of masked language modelling: random cells are masked and the model reconstructs them.
- Imputation networks: Models trained to predict missing cells from the observed cells of the same row and table.
Learn more
The full guide to In-context learning explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Imputation networks · Next term: Industry 4.0