Semi-supervised learning

Overview

Semi-supervised learning trains a model with a small set of labeled rows and a large set of unlabeled rows. The labeled rows say what the right answer looks like. The unlabeled rows show how the data is spread. By using both, the model often does better than one trained on the labeled rows alone.

This matters because labels are costly and unlabeled rows are cheap. A hospital may hold millions of records but only a few thousand with a confirmed diagnosis. A factory may log every sensor reading but label only a handful of failures. Semi-supervised learning sits between supervised learning and unsupervised learning, and it is closely related to how tabular foundation models make use of vast unlabeled tables.

What it is

Semi-supervised learning is a family of methods that learn from partially labeled data. The target is the same as in supervised learning: predict a label for new rows. The difference is the training data. Most rows have no label.

Think of a language student with a small phrasebook, which is the labeled data, and a large pile of newspapers, which is the unlabeled data. The phrasebook gives exact meanings. The newspapers show how words are used in context. A student who reads both learns faster than one who reads only the phrasebook.

The core assumptions

Semi-supervised learning works only when the unlabeled data tells you something about the labels. Several assumptions describe when that is true.

• Smoothness: rows that are close in feature space tend to share a label.
• Cluster: rows in the same cluster tend to share a label.
• Low density separation: the boundary between classes should pass through sparse regions, not through dense ones.
• Manifold: the data lies on a low dimensional shape, and labels vary smoothly along it.

If none of these hold, unlabeled data adds noise and can hurt.

Where it sits among related ideas

• Supervised learning uses only labeled rows.
• Unsupervised learning uses no labels at all.
• Self-supervised learning builds its own labels from data. It is often used as a first stage before a few labels are added.
• Active learning chooses which rows to label next. It can be paired with semi-supervised methods.
• Transfer learning reuses a model trained elsewhere.
• Few-shot learning aims to work from a handful of labeled examples.

Where it came from

The idea of using unlabeled data is old. In the 1960s and 1970s, researchers studied self-training, where a model labels its own data and learns from those labels. Mixture models fitted with expectation maximization gave an early principled way to use unlabeled rows, treating the missing labels as hidden variables.

In the 1990s, co-training was proposed. It uses two different views of the same data, for example the text of a web page and the text of links pointing to it. Each view trains a model that labels data for the other. Transductive support vector machines appeared at about the same time and tried to place the class boundary away from unlabeled rows.

Graph based methods followed in the early 2000s. Label propagation spreads labels along a graph where nearby rows are linked. A widely read survey and a book on the topic helped define the field in the mid 2000s.

The deep learning era added consistency regularization, which asks a model to give the same answer for slightly changed versions of a row, and pseudo-labeling, which keeps confident predictions as new labels. Methods such as Mean Teacher, MixMatch, and FixMatch combined these ideas for images, and the principles carry to tables with care. Large scale self-supervised pre-training then changed the picture again. See the research overview for the link to table models.

How it works

The methods below differ in details, but all try to push the model to respect the structure of the unlabeled data.

Self-training and pseudo-labeling

This is the simplest recipe. It is like a student who grades their own practice test and then studies from those grades.

1. Train a model on the labeled rows.
2. Predict labels for the unlabeled rows.
3. Keep only predictions where the model is very confident.
4. Add those rows with their predicted labels to the training set.
5. Retrain and repeat.

The risk is confirmation bias. If early predictions are wrong, the model learns its own mistakes and grows more sure of them.

Co-training

Co-training needs two different views of each row. Imagine two detectives who look at different clues. Each one labels cases it is sure about and passes them to the other. For tables, views can be two disjoint sets of columns. The views should each be enough to predict the label and should not be fully dependent.

Label propagation and graph methods

Build a graph in which each row is linked to its nearest neighbors. Labeled rows start with their label. Labels then flow along links, like dye spreading through connected pipes. After several rounds, each unlabeled row takes the label that reached it most.

This fits the smoothness and cluster assumptions directly. It works well when the neighbor graph reflects real similarity.

Consistency regularization

The model is trained so that its output stays stable when the input is perturbed. The perturbation can be noise, dropout, or a small change to numeric columns. On unlabeled rows, the loss is the gap between predictions on two perturbed copies. Think of asking the same question in two slightly different ways and expecting the same answer.

Mean Teacher and ensembling

A teacher model is a slow moving average of the student model. The student is pushed to match the teacher on unlabeled rows. Because the teacher averages many past versions, its targets are more stable than the latest student output.

Entropy minimization

This method adds a term that rewards confident predictions on unlabeled rows. It nudges the class boundary into sparse regions. It must be used with care, since too much pushes the model to be falsely sure.

Generative approaches

A generative model learns how rows are produced, then uses the labeled rows to attach labels to the learned structure. Mixture models and variational autoencoders fit here. A variational autoencoder can learn a compact code from all rows and then train a small classifier on the codes of labeled rows.

Pre-train then fine-tune

This two stage recipe is now very common.

1. Pre-train on all rows, labeled and unlabeled, with a self-supervised objective.
2. Fine-tune or fit a small head on the labeled rows.

It is simple, stable, and often the strongest choice when data is plentiful.

Variants of the pseudo-labeling loop

• Hard labels keep only the top class and treat it as truth.
• Soft labels keep the full probability vector, which carries more nuance.
• Curriculum pseudo-labeling starts with the easiest rows and adds harder ones later.
• Class-wise thresholds use a lower bar for rare classes so they are not lost.
• Ensemble labeling uses the vote of several models to reduce single model bias.
• Noisy student training makes the student larger or more regularized than the teacher.

Perturbations that make sense on tables

Images are easy to flip or crop. Tables are less obvious. Options that tend to keep meaning:

• Add small Gaussian noise to scaled numeric columns.
• Randomly drop a column value, as if it were missing.
• Swap a value with one from another row in the same column.
• Mix two rows with a convex blend, for numeric columns.
• Apply dropout inside the network.

Avoid changes that break meaning, such as shifting a date past its valid range or altering an identifier.

How the loss is put together

Most methods use a sum of two parts.

• A supervised loss on labeled rows, such as cross entropy.
• An unsupervised loss on unlabeled rows, such as consistency or entropy.

A weight balances them. It usually starts near zero, so the model first learns from labels, and then rises as training goes on. This is called a ramp-up schedule.

Key concepts and terms

Transductive versus inductive

A transductive method predicts labels only for the unlabeled rows given. Label propagation is a typical example. An inductive method builds a model that can score any future row. Self-training with a classifier is inductive. Pick based on whether you need a reusable model.

A worked example

This example uses scikit-learn on a synthetic table. It hides most labels, trains a baseline on the few that remain, then applies self-training and compares the scores.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.metrics import accuracy_score

X, y = make_classification(
    n_samples=3000, n_features=12, n_informative=6,
    n_classes=2, class_sep=1.0, random_state=0
)
X_pool, X_test, y_pool, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=0
)

# Keep only 5 percent of labels. -1 means unlabeled.
rng = np.random.default_rng(0)
y_partial = y_pool.copy()
mask = rng.random(len(y_pool)) > 0.05
y_partial[mask] = -1
labeled = y_partial != -1
print("labeled rows:", labeled.sum(), "of", len(y_pool))

# Baseline: only the labeled rows.
base = RandomForestClassifier(n_estimators=200, random_state=0)
base.fit(X_pool[labeled], y_partial[labeled])
print("baseline accuracy:", round(accuracy_score(y_test, base.predict(X_test)), 3))

# Self-training with all rows.
st = SelfTrainingClassifier(
    RandomForestClassifier(n_estimators=200, random_state=0),
    threshold=0.9, max_iter=10
)
st.fit(X_pool, y_partial)
print("self-training accuracy:", round(accuracy_score(y_test, st.predict(X_test)), 3))

Reading the example

• Only about 5 percent of the pool keeps a label. The rest are marked as -1.
• The baseline sees only those few rows.
• Self-training starts from the same few rows, then adds confident pseudo-labels over up to ten rounds.
• The test set is fully labeled and never used for training, so the comparison is fair.
• Gains are common but not guaranteed. If the gap is small or negative, the unlabeled data is not helping.

A numeric walk-through of pseudo-labeling

Suppose you have 100 labeled rows and 10,000 unlabeled rows. A first model reaches 80 percent accuracy. You predict on the unlabeled rows and keep those with confidence of 0.95 or more. Say that is 3,000 rows. If the model is well calibrated, about 95 percent of them, or 2,850, are correct, and 150 are wrong.

You now train on 3,100 rows, of which 150 carry wrong labels. That is about 5 percent noise, which many models tolerate. The second model sees far more examples of the data shape and may reach 85 percent. If confidence were poorly calibrated, the kept set might hold 20 percent errors, and the second model could get worse. This is why calibration and threshold choice matter.

A graph example in numbers

Imagine five rows on a line with positions 1, 2, 3, 8, and 9. Row 1 is labeled A and row 9 is labeled B. The others are unlabeled. Link each row to its nearest neighbors.

• Row 2 is linked to rows 1 and 3. Label A flows in from row 1.
• Row 3 is linked to row 2 and, more weakly, to row 8. It leans toward A.
• Row 8 is linked to row 9 and row 3. It leans toward B.
• After a few rounds, rows 2 and 3 settle on A, and row 8 settles on B.

A supervised model with two labels would have to guess the boundary. The graph shows a gap between 3 and 8, so the boundary falls there. This is the low density separation idea in action.

Measuring the gain

Repeat with several random choices of which rows are labeled. A gain that appears in only one subset is not trustworthy.

Strengths and limitations

Strengths

• It cuts labeling cost by making use of unlabeled rows.
• It can improve accuracy a lot when labels are very few.
• It works with many model types, from trees to neural networks.
• It fits real settings where unlabeled data is abundant.
• It can improve robustness by teaching the model about the data shape.

Limitations

• It can hurt. If the assumptions fail, unlabeled data adds noise.
• Confirmation bias can lock in errors.
• It is sensitive to class imbalance. Pseudo-labels often favor the majority class.
• Distribution mismatch between labeled and unlabeled rows breaks it.
• Evaluation needs a clean labeled validation set, which is itself costly.
• Hyperparameters such as thresholds are hard to tune with few labels.
• On tables, perturbations for consistency are less natural than on images.

When it tends to help

How it compares with alternatives

Pseudo-labeling versus label propagation

Role in tabular foundation models

Tabular foundation models are pre-trained to read raw tables. Semi-supervised ideas appear in their design, their training, and their use.

The setting matches real tables

A typical business table has a target column that is only partly filled. Some rows have outcomes, others do not yet. A model that treats the table as a whole, labeled and unlabeled rows together, is a natural fit. In-context learning does this: labeled rows give examples, and unlabeled rows are the ones to predict. Some table models can also use the unlabeled rows as additional context, which is a transductive view.

Self-supervised pre-training is the first stage

The pre-train then fine-tune recipe maps directly to table models. Stage one learns from many tables without labels, by hiding cells and predicting them. This teaches how columns relate, how values are spread, and what missing patterns look like. See self-supervised learning. Stage two uses a few labels in context or in a light fit.

Row and column structure

Label propagation works because similar rows should share labels. A table model builds that similarity inside attention across rows. Attention across columns lets the model weigh which fields matter for similarity. Both are described on the architecture page and use transformers.

Mixed types and missing values

Neighbor graphs and consistency perturbations are hard to define on mixed columns. A pre-trained model learns a shared representation for numbers, categories, and dates, so similarity is better defined. Missing cells are handled by the same learned machinery and can be predicted too. See imputation networks.

No column names

When headers are absent, a model must infer meaning from values. Unlabeled rows help here, since the distribution of a column reveals its nature. Pre-training on large numbers of tables, including those from synthetic data generation, prepares the model for tables it has never seen.

Pseudo-labels and continual learning

In a long lived system, new unlabeled data arrives all the time. Confident predictions can be fed back as pseudo-labels, and true labels, when they arrive, can correct them. This is a loop that resembles continual learning. The guard rails are the same: confidence thresholds, calibration, monitoring of drift, and periodic checks against real labels.

Anomaly detection and semantic classification

Unlabeled rows teach a model what normal looks like, which supports anomaly detection. A few labeled examples of a table type, such as retail or logistics, plus many unlabeled tables, can support sector and semantic classification of whole datasets or columns.

Illustrating the role of context size

A table model can only attend to so many rows at once. If the labeled set is small, all of it fits in context, and the unlabeled rows can follow. If the labeled set is very large, a subset is chosen, often the rows nearest to the row being predicted. This retrieval step is itself a neighbor method, close in spirit to label propagation.

Benchmarks

Gains from using unlabeled rows are measured by comparing against the same model with labeled rows only. Methods for scoring table models are discussed on the benchmarks page.

Practical guidance and best practices

Before you start

• Confirm that unlabeled rows come from the same source as labeled ones.
• Keep a clean labeled validation set. Do not pseudo-label it.
• Start with a supervised baseline on the labeled rows. You need it as a floor.
• Check class balance, since pseudo-labels tend to favor common classes.

Choose the method by situation

Tune with care

• Use a high confidence threshold at first, then lower it slowly.
• Cap the number of rounds.
• Add pseudo-labeled rows in small batches.
• Balance classes among the pseudo-labels.
• Give pseudo-labeled rows a lower weight than true labels.
• Check calibration. Overconfident models make poor teachers.

Quality checks on the unlabeled pool

Before using unlabeled rows, look at them.

• Compare column ranges with the labeled set.
• Compare category frequencies.
• Train a simple classifier to tell labeled rows from unlabeled rows. If it can do so easily, the two sets differ and gains are at risk.
• Look for duplicates between the pool and the test set, which would leak information.
• Remove rows that are clearly corrupt.

How labeled data size changes the plan

Monitor and verify

• Track validation accuracy each round. Stop when it stops improving.
• Spot check pseudo-labels with a human.
• Compare against the supervised baseline at every stage.
• Watch for drift between labeled and unlabeled sets.

Estimating how much to trust a pseudo-label

A raw model probability is not always a good guide. Several signals can be combined.

• Agreement among models in an ensemble.
• Stability of the prediction under small perturbations.
• Distance to the nearest labeled row of the same class.
• Margin between the top two class probabilities.

A row that passes all four checks is a safer candidate than one that passes only the first.

A minimal recipe to try first

1. Split off a validation set and never touch it.
2. Fit a supervised baseline on the labeled rows.
3. Run self-training with a threshold of 0.95 and at most five rounds.
4. Compare to the baseline on validation.
5. If the gain is clear, lower the threshold a little and test again.
6. If not, try a pre-trained table model before spending more effort.

A short checklist

• Is the unlabeled data from the same distribution?
• Is there a clean validation set?
• Does the semi-supervised model beat the baseline?
• Are thresholds and weights justified?
• Is there a plan to replace pseudo-labels with true labels over time?

Where the technique is used

• Medical records, where expert review is costly and most records are unreviewed.
• Industrial monitoring, where failures are rare and sensor logs are vast.
• Document and product classification, where taxonomy labels are partial.
• Fraud and risk, where confirmed cases lag behind new transactions.
• Customer data, where outcomes arrive weeks or months later.
• Scientific measurements, where each label needs a lab test.

Safety nets for pseudo-labeling in production

• Log which rows carry pseudo-labels and the round in which they were added.
• Never mix pseudo-labels into the gold evaluation set.
• Replace pseudo-labels with true labels as soon as they exist.
• Alert when the share of confident predictions changes sharply.
• Keep a frozen copy of the supervised baseline for comparison.

Common mistakes

• Pseudo-labeling the validation or test set, which hides real error.
• Using a low confidence threshold, which floods training with wrong labels.
• Ignoring class imbalance, so the rare class vanishes.
• Mixing unlabeled data from a different source and expecting gains.
• Skipping the supervised baseline, so you cannot tell if the method helped.
• Retraining for too many rounds, which amplifies early mistakes.
• Using uncalibrated confidence scores as if they were probabilities.
• Perturbing rows in ways that break their meaning, such as shuffling a date.
• Assuming more unlabeled data always helps. Past a point it adds little.
• Forgetting to document which labels are true and which are guessed.

A short story of how it can go wrong

A team has 200 labeled loan outcomes and 50,000 unlabeled applications. They train a model, label the confident cases, and retrain for twelve rounds. Accuracy on the validation set climbs, then falls. Looking closer, they find that the unlabeled pool includes applications from a new region with different patterns. The model labeled these confidently by analogy to the old region and was wrong. Each round made the error larger. The fix was to filter the pool to match the labeled source, lower the number of rounds, and add a small set of true labels from the new region.

Frequently asked questions

What is semi-supervised learning in simple words?

It is learning from a mix of a few labeled rows and many unlabeled rows. The labels say what the right answers are, and the unlabeled rows show how the data is spread. Together they usually give a better model than the labels alone.

How is it different from supervised and unsupervised learning?

Supervised learning uses only labeled rows, and unsupervised learning uses none. Semi-supervised learning sits in between and uses both at once. The goal is still to predict a label, but the training signal comes partly from the shape of the unlabeled data.

What is pseudo-labeling?

Pseudo-labeling means letting a trained model guess labels for unlabeled rows and then training on the confident guesses as if they were real. It is simple and effective, but wrong guesses can reinforce themselves. A high confidence threshold and a clean validation set help control that risk.

Can semi-supervised learning make results worse?

Yes. If the unlabeled rows come from a different source, or if classes overlap so much that structure does not match labels, the extra data can mislead the model. This is why a supervised baseline and a clean validation set are essential.

How many labels do I need?

There is no fixed number. The method helps most when labels are very few compared with the unlabeled pool, yet there must be enough labeled rows of each class to anchor the model. A handful per class is a bare minimum, and a few dozen to a few hundred per class is more comfortable.

Does it work on tabular data?

It does, though the best methods differ from those used on images. Self-training with tree models, label propagation on a neighbor graph, and pre-training with cell masking all work on tables. Consistency tricks that rely on image augmentations are harder to apply.

How does it relate to active learning?

Both tackle costly labels. Semi-supervised learning uses unlabeled rows without asking for more labels. Active learning asks a person to label the most useful rows. They combine well: the model can pseudo-label easy rows and request human labels for hard ones.

How does it relate to tabular foundation models?

Tabular foundation models are pre-trained on large numbers of tables, often with self-supervised objectives that need no labels. At use time they take a few labeled rows as context and predict for the rest. That is a close cousin of semi-supervised learning, with the heavy work moved into pre-training.

What is the difference between transductive and inductive methods?

A transductive method gives labels only for the specific unlabeled rows it was given. An inductive method learns a model that can label any future row. Label propagation is usually transductive, and self-training with a classifier is inductive.

How do I evaluate a semi-supervised model?

Hold out a labeled validation set that the method never touches. Compare the model with a supervised baseline trained on the same labeled rows. Report the gain, and check results by class and over time. A method that does not beat the baseline is not worth the extra complexity.

Glossary

• Cluster assumption: The idea that rows in one cluster share a label.
• Co-training: Training two models on different views, each labeling data for the other.
• Confidence threshold: The probability above which a pseudo-label is kept.
• Confirmation bias: A model growing more sure of its own errors.
• Consistency regularization: A penalty for giving different outputs to slightly changed inputs.
• Entropy minimization: Encouraging confident predictions on unlabeled rows.
• Graph: A set of rows linked by similarity.
• Inductive learning: Learning a model that applies to any new row.
• Label propagation: Spreading known labels along a similarity graph.
• Manifold: A low dimensional shape on which data lies.
• Mean Teacher: A method in which a slow average of the model guides the model.
• Pseudo-label: A model-made label used as if it were true.
• Self-training: Repeatedly training on the model's own confident predictions.
• Transductive learning: Predicting labels only for the given unlabeled rows.
• Unlabeled data: Rows with features but no known target.
• Weak supervision: Using noisy rules or sources in place of exact labels.

Summary

Semi-supervised learning blends a few labeled rows with many unlabeled rows to train better models at lower labeling cost. Its main tools are self-training, co-training, label propagation, consistency regularization, and pre-train then fine-tune. It works when unlabeled data shares structure with the labels, and it can fail when that is not true.

Good practice needs a clean validation set, a supervised baseline, a high confidence threshold, and checks for calibration and drift. Most problems come from confirmation bias and mismatched data sources.

For tabular foundation models, the semi-supervised view is natural. Pre-training on unlabeled tables teaches row and column structure, mixed types, and missing value patterns. At use time, a few labeled rows in context guide predictions on the rest. The same backbone also supports imputation, anomaly detection, and semantic classification. See the research, architecture, and benchmarks pages for more.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: