Active learning

Overview

Active learning is a way of training a model in which the model helps choose which unlabelled examples a human should label next. Instead of labelling a random sample, you label the examples that are expected to teach the model the most. The aim is to reach a target quality with far fewer labels, which saves time and money when annotation is slow, expensive or needs an expert.

The loop is simple: train a model on what is labelled, score the unlabelled pool, pick the most useful items, ask a person for their labels, add them and repeat. It fits into the family of methods that stretch scarce labels, along with semi-supervised learning, transfer learning and few-shot learning. In tabular work, it helps decide which rows, columns or records deserve a person's attention, and it pairs naturally with human in the loop workflows.

What it is

In ordinary supervised learning the labelled set is fixed before training. In active learning the labelled set grows during training, and the learner takes part in choosing it. The learner is sometimes called the student, and the labelling person or system the oracle or teacher.

An everyday analogy is a student preparing for an exam with a tutor who has limited time. A passive student asks the tutor about random pages of the textbook. An active student first tries the exercises, notices which ones are confusing, and asks the tutor only about those. The tutor's time goes much further.

Scenarios

Active learning has three classic scenarios, which differ in how candidate items arrive.

Pool-based sampling is the focus of this article, because it matches how most tables are stored.

The cost model

Active learning only pays off when labels are costly relative to compute. If a label takes seconds and is cheap, labelling more data at random may be simpler. If a label needs a domain expert, a lab test or a legal review, then each saved label is valuable. A realistic cost model includes the time to read an item, the expertise needed, the chance of labeller error and the delay before a label comes back.

Labels are not always classes

The "label" can be many things in a tabular setting:

• A class for a row, such as fraud or not fraud.
• A numeric target, such as a measured value.
• A correction, such as the right value for a flagged cell.
• A link between records, such as whether two rows describe the same entity.
• A column meaning, such as the semantic type of a column.

All of these can be queried actively.

Where it came from

The idea that a learner might ask questions goes back to early work on learning theory, where a learner could query an oracle about membership of an example in a concept. In the 1990s, researchers showed that choosing queries well could make learning exponentially more efficient in some simple settings. "Query by committee" and "uncertainty sampling" were introduced in that period. Text classification was the first large application area, because documents are plentiful and labelling them is slow.

Through the 2000s, active learning grew along with kernel methods, with strategies built for support vector machines, such as choosing points close to the decision margin. Surveys from that decade organized the field into the scenarios and strategy types still used today. Practical issues also came into focus: batch queries, noisy labellers, class imbalance and the "cold start" problem.

Deep learning brought new strategies. Bayesian approaches used dropout or ensembles to estimate uncertainty. "Bayesian Active Learning by Disagreement", known as BALD, scored items by the mutual information between predictions and model parameters. Geometric approaches such as the core-set method (2018) chose batches that cover the data well. Later work combined active learning with self-supervised learning, where pre-trained embeddings give better starting points for selection and fix much of the cold start problem. Recent studies also show that active learning does not always beat random sampling, especially when the model is weak early on, which led to more careful evaluation.

How it works

The basic loop

1. Start. Begin with a small labelled seed set and a large unlabelled pool. If there is no seed, pick a few items at random or by clustering.
2. Train. Fit a model on the labelled set.
3. Score. Compute an informativeness score for every item in the pool.
4. Select. Choose the top items, one at a time or in a batch.
5. Label. Send them to the oracle and receive labels.
6. Update. Add them to the labelled set.
7. Stop or repeat. Stop when the budget runs out, when accuracy on a validation set stops improving, or when the model is confident on nearly all of the pool.

The loop is a conversation: the model says what it does not understand, the human explains, and the model improves.

Query strategies

Uncertainty sampling. Pick items where the model is least sure. There are three common scores for classification.

• Least confidence: one minus the probability of the top class.
• Margin: the gap between the top two class probabilities. Small gap means high uncertainty.
• Entropy: the spread of the whole probability vector. High entropy means high uncertainty.

For regression, use the predictive variance, the width of a prediction interval, or the disagreement between models. This is the simplest strategy and often a good first choice. The analogy is a student focusing on the exercises they are most unsure about.

Query by committee. Train several models, a committee, on the labelled data. Ask about items on which they disagree most. Disagreement can be measured by vote entropy or by the average divergence from the committee mean. Random forests and ensembles of neural networks give a committee for free. See random forest.

Expected model change. Pick items that would change the model most if their labels were known. For gradient-based models, this means the expected gradient length. It is more costly to compute but captures impact and not only doubt.

Expected error reduction. For each candidate and each possible label, imagine retraining, and estimate how much the future error falls. It is principled and very expensive, so it is used only on small pools.

Density-weighted methods. Pure uncertainty may pick outliers. Multiply uncertainty by how typical the item is, so that picked items are both unclear and representative. This avoids spending labels on weird rows that say little about the bulk of the data.

Diversity and core-set methods. When selecting a batch, make sure items are different from each other. Clustering, k-center greedy selection and determinantal sampling all enforce diversity. Without it, a batch may hold ten near-duplicates of the same confusing case.

Hybrid methods. Combine uncertainty and diversity, for example by clustering the most uncertain items and taking one per cluster. Many strong practical systems use this idea.

Bayesian methods. BALD and related scores favour items on which individual plausible models disagree confidently. They separate uncertainty due to lack of knowledge from uncertainty due to inherent noise. Only the first kind can be reduced by labelling.

Exploration-based methods. Early in the loop, the model is poor, so its uncertainty is unreliable. Mixing in random or cluster-based picks guards against a model that confidently ignores whole regions.

Batch mode

Real labelling happens in batches, since people work in sessions and retraining takes time. Batch selection should balance informativeness and diversity. A common recipe is to take the top few hundred items by uncertainty, cluster them, and pick a representative from each cluster.

Stopping rules

When should the loop end? Options include:

• A fixed labelling budget.
• A plateau in validation performance over several rounds.
• A drop in the average uncertainty of the pool below a threshold.
• Agreement between the last two models on the pool above a threshold.

Handling label noise

Human labels contain errors, especially for borderline items, which are exactly the ones that active learning chooses. Mitigations include asking two labellers for hard items, tracking labeller agreement, using noise-robust losses and re-querying items that the model strongly disagrees with after training.

Choosing a strategy

Designing the labelling workflow

The model is only part of the system. The human side decides whether the savings are real.

• Show enough context. A reviewer who sees one row in isolation will guess. Show related rows, history and the model's own suggestion if bias is acceptable.
• Offer an "unsure" option. Forcing a label on a truly unclear case adds noise. Keep unsure items aside for an expert.
• Mix in some easy items. This keeps reviewers calibrated and gives a free quality check.
• Randomize order. Showing items sorted by model score can bias the reviewer.
• Track time per item. Time is the real cost, and it varies by item. Time can itself become part of the selection score.
• Close the loop quickly. Retrain soon after labels arrive, so the next batch reflects the latest knowledge.
• Write labelling guidelines. Define edge cases with examples, so labels stay consistent over weeks.

Evaluating honestly

A sound evaluation has four parts.

1. A fixed random test set, labelled once, that never takes part in selection.
2. A learning curve with the number of labels on the x axis, repeated over several random seeds.
3. A random-sampling baseline with the same model and the same budget.
4. A cost view, such as performance against reviewer hours.

Report the area under the learning curve or the number of labels needed to reach a fixed target. A statement such as "reached the target with fewer labels" is clear and easy to check.

Query types specific to tables

Each query type has its own uncertainty measure, but the loop is the same. Because a table has both rows and columns, the choice of whom to ask also involves the choice between asking about a row, a column or a cell. A tabular system can estimate the expected value of each and pick the cheapest high-value query.

Combining with other label-saving ideas

• Active learning plus pseudo-labels. After each round, add the model's most confident predictions as pseudo-labels, and keep the uncertain ones for people. This is a mix of active and semi-supervised learning.
• Active learning plus pre-training. Start from a pre-trained encoder, so uncertainty estimates are useful from round one.
• Active learning plus weak rules. Let simple rules label obvious cases, and ask people about the rest.
• Active learning plus drift monitoring. Trigger a labelling round when input drift or confidence drops, not on a fixed calendar.

Key concepts and terms

A worked example

Suppose you have 20,000 transaction rows and need a fraud classifier. A reviewer can label about 300 rows. You will run five rounds of 60 labels. The pool is unlabelled, apart from a seed set of 40 rows.

Code

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.cluster import KMeans

pool = pd.read_parquet("example.internal/data/transactions.parquet")
feature_cols = [c for c in pool.columns if c not in ("row_id", "label")]
X = pool[feature_cols].to_numpy()

rng = np.random.default_rng(7)
labelled = list(rng.choice(len(pool), size=40, replace=False))
oracle_labels = {}           # filled when the reviewer answers

def ask_oracle(indices):
    # Placeholder: in practice send rows to a review tool at review.example.internal
    return {i: int(pool.loc[i, "label"]) for i in indices}

oracle_labels.update(ask_oracle(labelled))

for round_id in range(5):
    idx = np.array(sorted(oracle_labels))
    y = np.array([oracle_labels[i] for i in idx])
    model = RandomForestClassifier(n_estimators=300, class_weight="balanced",
                                   random_state=0).fit(X[idx], y)

    unlabelled = np.setdiff1d(np.arange(len(pool)), idx)
    proba = model.predict_proba(X[unlabelled])
    top2 = np.sort(proba, axis=1)[:, -2:]
    margin = top2[:, 1] - top2[:, 0]                  # small = uncertain

    # Take the 300 most uncertain rows, cluster them, pick one per cluster.
    cand = unlabelled[np.argsort(margin)[:300]]
    km = KMeans(n_clusters=60, n_init=5, random_state=0).fit(X[cand])
    chosen = []
    for k in range(60):
        members = cand[km.labels_ == k]
        centre = km.cluster_centers_[k]
        chosen.append(members[np.argmin(((X[members] - centre) ** 2).sum(axis=1))])

    oracle_labels.update(ask_oracle(chosen))
    print(f"round {round_id}: labels = {len(oracle_labels)}")

What the code does

1. Trains a random forest on the labels collected so far.
2. Scores the unlabelled pool with the margin, the gap between the top two class probabilities.
3. Keeps the 300 most uncertain rows.
4. Clusters these into 60 groups and picks the row nearest each cluster centre, which adds diversity.
5. Sends the 60 rows to the reviewer and adds the answers.

A numeric example of the three uncertainty scores

Suppose a three-class model outputs these probabilities for two items.

• Least confidence: X = 1 - 0.50 = 0.50. Y = 1 - 0.40 = 0.60. Y is more uncertain.
• Margin: X = 0.50 - 0.45 = 0.05. Y = 0.40 - 0.30 = 0.10. X is more uncertain, since the top two are nearly tied.
• Entropy: X is about 0.94 bits. Y is about 1.57 bits. Y is more uncertain.

The scores disagree about which item is more uncertain. Margin cares about the closest competitor, entropy about the whole spread. Pick the score that matches your decision: if you only care about the top-versus-second call, margin is natural. If errors across many classes matter, entropy is a better guide.

Comparing with random sampling

A learning curve makes the benefit visible. Record validation performance after each round for both strategies, using the same seed set and model.

def run(strategy, rounds=5, batch=60):
    scores = []
    # ... loop as above, with strategy in {"random", "margin_cluster"}
    return scores

curve_random = run("random")
curve_active = run("margin_cluster")
# Plot both curves against the number of labels and compare area under each.

If the active curve sits above the random one at the same label count, active learning is paying off. If the curves overlap, random sampling is enough for this problem, and you save the engineering effort.

Practical result patterns

• With a skewed class balance, active learning often finds many more rare-class examples than random sampling does, because uncertain items concentrate near the rare class.
• With very noisy labels, gains shrink, since uncertain items are also the ones people disagree on.
• With weak early models, a random or cluster-based warm start protects against bad picks.

Why it can work: a short theory note

A simple case shows why choosing labels can beat random labelling. Imagine a single threshold that splits two classes along a line, and you want to find where it sits. With random labels, the error shrinks slowly, roughly in proportion to one over the number of labels. With active labelling you can do a binary search: label the middle point, then the middle of the uncertain half, and so on. The error then shrinks exponentially fast in the number of labels. Real problems are messier, because of noise and many dimensions, so the gain is smaller. Still, the idea holds: labels near the boundary carry more information than labels far from it.

The same story explains the main risk. If you only ever look near the current boundary, and the model's boundary is wrong, you may never see the regions where the model is confidently mistaken. That is why exploration and random validation are not optional extras.

Typical settings

• Quality inspection. Millions of unlabelled records, and each label needs an engineer. Active learning routes the borderline cases to the engineer.
• Data cleaning. A model flags cells that look wrong, and a steward confirms a ranked subset. See data catalog practice for where such feedback is stored.
• Document and record classification. Labels come from trained staff, whose time is the main cost.
• Scientific measurement. Each label is an experiment. Active learning chooses the experiments, close to the idea of Bayesian optimization.
• Monitoring and alert triage. Analysts review a small share of alerts. Their verdicts retrain the detector.
• Onboarding a new customer table. A handful of well-chosen labels from the customer stand in for a full labelled history.
• Cost-sensitive review. When some items cost more to review than others, divide the informativeness score by the expected review cost.
• Rare event discovery. When positives are one in a thousand, ranking by model score and reviewing the top finds them far faster than scanning at random.

Strengths and limitations

Strengths

• Reduces labelling cost, sometimes by large factors on skewed or well-structured problems.
• Focuses human attention on cases that matter, which improves reviewer motivation.
• Finds rare classes faster than random sampling.
• Works with almost any model that can score uncertainty.
• Pairs with semi-supervised learning: label the hard items by hand and let the model pseudo-label the easy ones.
• Provides a natural audit trail of why each item was reviewed.

Limitations

• The cold-start problem: an early weak model picks poor items.
• Sampling bias: the labelled set drifts away from the true distribution, so naive evaluation on it misleads. A separate random validation sample is needed.
• Gains are not guaranteed, and random sampling can win on some tasks.
• Batch effects: ten similar uncertain items waste labels without diversity.
• Dependence on the model: items chosen for one model may not help another.
• Labeller noise concentrates on the hard items that are chosen.
• Engineering cost: you need a loop, a labelling interface, versioning and monitoring.
• Poorly calibrated models give unreliable uncertainty.

How it compares with alternatives

Active learning is close in spirit to Bayesian optimization. Both choose the next query using a model of what is known. The difference is the goal: optimization seeks the best value of a function, active learning seeks the best model.

Role in tabular foundation models

Tabular foundation models learn from raw tables in general, then handle a new table with little or no training. Labels are still the bottleneck on the new table, and active learning decides where to spend them.

Choosing the context for in-context learning

An in-context model predicts from labelled rows placed in its input. The window is limited, and the quality of those rows matters more than their number. Active learning ideas select which rows to label and include. Pick rows that are diverse, to cover the table, and uncertain, to settle the hard boundaries. This turns a labelling budget of a few dozen rows into a strong context for models such as TabPFN and TabICL.

Uncertainty from foundation models

A good query strategy needs calibrated uncertainty. In-context tabular models can produce predictive distributions, and ensembles over different context orderings or subsamples provide committee-style disagreement. Test-time variation of the context, such as permuting rows or dropping some, gives cheap committees without training more models. The architecture page describes the row and column attention that makes such permutations meaningful.

Embeddings help the cold start

The cold-start problem comes from having no useful model at the beginning. A pre-trained table encoder gives embeddings for rows from the first moment. Clustering these embeddings picks a diverse seed set, and distance in the embedding space supports core-set selection. This is how self-supervised learning and active learning combine in practice.

Mixed types, missing values and no column names

Active learning on raw tables must cope with numbers, categories, dates and text, with gaps and unclear headers. A foundation model that encodes each cell regardless of type, and handles masks natively as in masked cell modeling, supplies a uniform way to measure similarity and uncertainty between rows. That makes diversity measures meaningful on messy data without hand-built distance functions.

Imputation and cell-level queries

Active learning does not need to ask only about row labels. For imputation networks, the model can rank missing cells by how uncertain the imputed value is, and ask a person to fill the most uncertain, high-impact ones. This focuses data collection where it improves downstream results most.

Anomaly detection

For anomaly detection there are few labelled anomalies. Active learning here asks an analyst to review the highest-scoring and the most ambiguous rows. Their verdicts become labels that tune thresholds and turn an unsupervised detector into a semi-supervised one. A common pattern is to show the analyst a ranked list, collect feedback, and update the model each day.

Sector and semantic classification

When a model must tell what sector or meaning a table or column has, labels from experts are costly. Active learning picks the tables and columns whose predicted category is least clear. A few corrections often fix a whole family of look-alike columns. This ties to semantic sector heads.

Continual learning and shift

Active learning also manages change. When data drift or a new table type appears, the model's uncertainty rises in that region, and the query strategy sends those rows to review. The corrected labels feed continual updates, protected against forgetting by methods such as elastic weight consolidation. This closes the loop between monitoring, labelling and retraining. See also domain adaptation.

Evaluation

Evaluate active learning with learning curves that count labels, not rounds. See the benchmarks and research pages for how label efficiency is compared across models.

Practical guidance and best practices

1. Check that labels are the bottleneck. If labelling is cheap, random sampling at scale may be easier.
2. Hold out a random validation set. Label it once at random and never use it for selection. It is your unbiased yardstick.
3. Begin with a diverse seed. Use clustering on embeddings or a stratified random sample to avoid the cold-start trap.
4. Choose a simple strategy first. Margin sampling with a diversity step is a strong default.
5. Calibrate the model. Poor probabilities lead to poor uncertainty. Use temperature scaling or isotonic regression.
6. Select in batches with diversity. Avoid sending duplicates to the reviewer.
7. Mix in exploration. Reserve 10 to 20 percent of each batch for random picks, to catch regions the model wrongly trusts.
8. Design the labelling tool well. Show context, allow "unsure" and "skip", and measure time per item.
9. Measure labeller agreement. Have a sample labelled twice. Low agreement caps what any model can learn.
10. Track cost, not only accuracy. Plot performance against total labelling hours.
11. Version everything. Log the pool snapshot, model version, strategy and selected ids for each round.
12. Define a stop rule early. Decide the budget and the plateau criterion before you begin.

A small configuration sketch

active_learning:
  pool: warehouse.example.internal/transactions_unlabelled
  seed:
    method: cluster_embeddings
    size: 50
  rounds: 10
  batch_size: 60
  strategy:
    score: margin
    candidate_multiplier: 5
    diversity: kmeans_one_per_cluster
    random_fraction: 0.15
  validation:
    random_labelled_rows: 500
  stop:
    max_labels: 700
    plateau_rounds: 3
  review_tool: review.example.internal

Common mistakes

• Evaluating on the actively chosen set. The labelled set is biased toward hard items. Evaluate on a separate random sample.
• Ignoring the cold start. A weak first model picks nonsense. Warm up with diversity.
• Skipping diversity in batches. You pay for ten labels and learn the lesson of one.
• Trusting uncalibrated scores. Uncertainty from a badly calibrated model is a poor guide.
• Chasing outliers. Pure uncertainty favours strange rows. Weight by density.
• Forgetting the class imbalance. The rare class needs enough examples, and the strategy should make sure it gets them.
• Letting the model and the strategy drift apart. If you change the model, the old selections may be less useful. Recompute scores each round.
• Ignoring labeller fatigue. A reviewer given only hard cases tires and errs more. Mix in some easy items and rotate staff.
• Having no stop rule. Without one, labelling goes on past the point of benefit.
• Never comparing with random. You cannot claim savings without a baseline run on the same budget.
• Mixing up cost units. Ten hard labels may cost more than fifty easy ones. Count time and expertise, not only counts.
• Leaking labels through features. If a column such as a review status is a feature, the model cheats. Remove it.

Frequently asked questions

What is active learning in simple terms?

Active learning is training in which the model chooses which unlabelled examples a person should label next. It picks the ones it expects to learn the most from, such as items it is unsure about. This means a model can reach good accuracy with fewer labelled examples than random labelling would need.

What is the difference between active learning and semi-supervised learning?

Active learning asks a human for labels on carefully chosen items. Semi-supervised learning uses the unlabelled items directly, for example through pseudo-labels or consistency training, with no new human labels. They combine well: use semi-supervised methods on the easy items and active learning for the hard ones.

What is uncertainty sampling?

Uncertainty sampling picks the unlabelled items for which the current model is least certain about its prediction. Common scores are least confidence, margin and entropy. It is the simplest and most widely used query strategy in active learning.

What is the cold-start problem in active learning?

Cold start refers to the early rounds when the model has too few labels to be a good guide. Its uncertainty estimates are unreliable, so it may select unhelpful items. Remedies include a diverse seed set from clustering, random exploration, and starting from pre-trained embeddings.

Does active learning always beat random sampling?

No. It often helps when labels are costly, classes are imbalanced or the model is well calibrated, but on some problems random sampling is just as good or better. Always compare against a random baseline with the same label budget before claiming a benefit.

How many labels do I need to start?

A few dozen are usually enough to train a first model on a simple problem, and fewer if you start from strong pre-trained embeddings. What matters more is that the seed set is diverse and includes at least a few examples of each class. Very rare classes may need targeted search to find the first examples.

What is query by committee?

Query by committee trains several models on the same labelled data and asks for labels on items where they disagree most. Disagreement signals that the available data does not yet settle the answer for that item. Ensembles such as random forests give a committee without extra effort.

How does active learning handle noisy labels?

It does not remove noise, and may even raise it, since chosen items are often borderline. Good practice is to have a share of items labelled by two reviewers, measure agreement, use losses that tolerate noise and re-check items where the model strongly disagrees with the label.

Can active learning be used for regression and for cells in a table?

Yes. For regression, uncertainty can be measured with predictive variance, interval width or ensemble disagreement. For tables, the query can be a missing cell, a record link or a column meaning, as well as a row label. The same loop applies: score, select, ask, update.

How do I know when to stop?

Stop when the budget is spent, when validation performance has stopped improving for several rounds, or when the model is confident on almost all of the pool. Decide the rule before starting, and judge it on a random validation sample, not on the items chosen by the strategy.

Glossary

• Annotation: The act of attaching a label to an item.
• Batch mode: Choosing many items per labelling round.
• BALD: A Bayesian score that favours items where plausible models disagree.
• Calibration: Agreement between predicted probabilities and real frequencies.
• Cold start: The early period when the model is too weak to guide selection.
• Committee: A set of models used to measure disagreement.
• Core-set: A small set that represents a larger one well.
• Entropy: A measure of the spread of a probability distribution.
• Learning curve: Performance plotted against the number of labels.
• Margin: The gap between the two highest predicted probabilities.
• Oracle: The labelling source.
• Pool-based sampling: Choosing from a fixed pool of unlabelled items.
• Query strategy: The rule that picks the next items to label.
• Sampling bias: A labelled set that no longer reflects the data distribution.
• Seed set: The initial labelled data.
• Stream-based sampling: Deciding to label or skip each item as it arrives.

Summary

Active learning lets a model steer its own labelling. By scoring the unlabelled pool for uncertainty, disagreement or impact, and by adding diversity to each batch, a team can reach a target quality with a smaller labelling bill. The method works best when labels are costly, classes are skewed and the model is reasonably calibrated, and it must always be checked against a random baseline on a clean validation set.

For tabular foundation models, active learning decides which few rows to label for the context, which cells to repair, which anomalies to review and which columns to classify by hand. Pre-trained table embeddings fix the cold start, calibrated in-context predictions give the uncertainty, and continual updates keep the system current. Explore the architecture, research and benchmarks pages for more.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: