Supervised learning

Overview

Supervised learning is the branch of machine learning where a model learns from examples that come with the right answer attached. Each example has inputs, called features, and a target, called the label. The model studies many such pairs and learns a rule that maps inputs to targets. Later it applies that rule to new rows where the answer is not known.

On tables, this is the everyday workhorse. Predicting whether a customer will leave, estimating the price of a house, flagging a payment as fraud, and scoring a lab result are all supervised tasks. The two main flavors are classification, where the target is a category, and regression, where the target is a number. Modern tabular foundation models such as TabPFN are also supervised at heart, but they learn the "rule finding" step ahead of time so a new table needs no long training run.

What it is

Supervised learning means learning a function from labeled data. Think of a student with an answer key. The student sees a question, gives an answer, checks the key, and adjusts. After enough practice the student can answer questions that were not in the practice set.

In a table, one row is one example. The columns are features. One special column is the target. The learner sees rows with the target filled in and tries to predict it from the other columns.

The two main task types

• Classification predicts a class. Examples are spam or not spam, churn or stay, and which product category a row belongs to.
• Regression predicts a number. Examples are demand next week, a sale price, or the time until a machine fails.

Some tasks sit between these. Ranking orders items. Multi-label classification assigns several tags at once. Probabilistic prediction returns a full distribution instead of one number. All of them still rely on labeled examples.

What makes it "supervised"

The supervision is the label. A human, a sensor, a business process, or a later event provides it. The word does not mean a person watches the training. It means the training signal is a known answer, and the error between prediction and answer drives learning.

How it differs from nearby ideas

• Unsupervised learning has no labels and looks for structure such as groups or unusual rows.
• Semi-supervised learning mixes a few labels with many unlabeled rows.
• Self-supervised learning invents labels from the data itself, for example by hiding a cell and predicting it.
• Reinforcement learning learns from rewards after actions, not from an answer key.

Where it came from

The roots go back to statistics. Least squares fitting for regression was developed in the early nineteenth century. Linear discriminant analysis, introduced in the 1930s, gave an early method for classification. Logistic regression became a standard tool for binary outcomes in the mid twentieth century.

In the 1950s and 1960s the perceptron showed that a simple unit could learn a linear decision rule from labeled examples. Interest cooled when its limits were shown, then returned with multi-layer networks trained by backpropagation in the 1980s.

Decision trees such as CART and ID3 appeared in the 1980s. They made supervised learning easy to read and handle mixed data types. In the 1990s support vector machines gave strong results with a clean mathematical base. Ensemble methods followed. Bagging and random forest averaged many trees. Boosting built trees one after another to fix earlier errors, and later systems such as XGBoost made this fast and practical.

From 2012 onward, deep networks won image and speech tasks, all trained in a supervised way on large labeled sets. On tables, tree ensembles stayed hard to beat for years. Recent work on pre-trained transformers for tables, including TabPFN, changed the picture for small and medium data. See the research overview for how this line of work fits together.

How it works

Supervised learning follows a loop. The steps below use a plain analogy: learning to guess the price of used cars.

Step 1: Collect labeled examples

You gather rows of cars with known sale prices. Each row has features such as age, mileage, and brand. The price is the label. The quality of the labels sets a ceiling on how good the model can be.

Step 2: Choose a model family

You pick a family of possible rules. A linear model draws a straight relation. A decision tree asks yes or no questions. A neural network stacks many small functions. Each family has its own knobs, called parameters.

Step 3: Define a loss

The loss measures how wrong a prediction is. For prices, a common loss is squared error. For classes, a common loss is cross entropy. Think of the loss as a score from a strict teacher: lower is better.

Step 4: Fit the parameters

The training algorithm adjusts the knobs to reduce the average loss on the training rows. Neural networks use gradient descent: they check which way each knob should turn and take a small step. Trees use greedy splitting: at each node they choose the question that best separates high prices from low ones.

Step 5: Hold out data to check

If you grade the student on the same questions used for practice, you learn nothing about real skill. So you keep some rows aside. A validation set guides choices such as model size. A test set gives the final honest score.

Step 6: Tune and regularize

Models can memorize noise. Regularization limits that. Examples include shallower trees, weight penalties, early stopping, and dropout. You tune these settings on validation data.

Step 7: Predict on new rows

For a new car, the model reads the features and outputs a price. A good system also reports how unsure it is, for instance a prediction interval.

Step 8: Monitor in use

The world changes. Prices shift, buyers change habits. A model trained last year may drift. You track error on fresh labeled data and retrain when needed.

Variations on the loop

The eight steps above are the common core. Real projects add a few twists.

• Online learning updates the model one row or one small batch at a time, so it can follow a stream.
• Batch learning fits on the full set at once and is retrained on a schedule.
• Multi-task learning trains one model for several related targets at the same time.
• Cost-sensitive learning gives different penalties to different mistakes.
• Weighted learning gives some rows more influence, for example newer rows or rarer classes.
• Ensembling blends several models, which often lowers variance.

Handling common data issues

Tables rarely arrive clean. The model family you pick changes how much cleaning you need.

Missing values

• Trees can split on "is missing" or route missing cells down a default branch.
• Linear models need a fill value, often the median plus a flag column.
• A flag column lets the model learn that missingness itself carries meaning.
• Pre-trained table models usually accept missing cells directly.

Categorical columns

• One-hot encoding makes a column per category. It works well for few categories.
• Target encoding replaces a category with the average label. It must be computed inside cross-validation, or it leaks.
• Embeddings learn a small vector per category. They suit neural networks.
• High-cardinality columns, such as postal codes, need special care.

Numeric columns

• Scaling matters for linear models and neural networks, but not for trees.
• Heavy tails may need a log transform.
• Outliers can dominate squared error. Robust losses help.

Dates and times

• Break a date into parts such as weekday, month, and age.
• Add lag features for time series, using only the past.
• Always split by time when the model will predict the future.

Key concepts and terms

Metrics worth knowing

• Accuracy counts the share of correct class predictions. It misleads when classes are imbalanced.
• Precision asks how many flagged rows were truly positive.
• Recall asks how many true positives were found.
• F1 combines precision and recall.
• AUC measures how well scores rank positives above negatives.
• Mean absolute error and root mean squared error summarize regression mistakes in the units of the target.
• Log loss rewards well calibrated probabilities.

Choosing a metric by situation

Bias and variance in plain words

Imagine throwing darts. High bias means the darts cluster far from the bullseye, always in the same wrong spot. High variance means the darts scatter widely, even if their center is right.

• A model that is too simple has high bias. It misses real structure.
• A model that is too flexible has high variance. It reacts to noise.
• The goal is a balance, found by watching validation error as capacity grows.
• More data usually lowers variance. Better features usually lowers bias.

Learning curves

A learning curve plots score against the number of training rows.

• If training and validation scores are both low and close, the model is too simple. Try richer features or a stronger model.
• If training score is high and validation score is low, the model overfits. Try more data or stronger regularization.
• If both curves still climb at the right edge, more data will help.
• If they have flattened, more data will not help. Change the model or the features.

A worked example

Here is a short, complete example with scikit-learn on a synthetic table. It trains a classifier, scores it on held-out rows, and prints a metric.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 2000
df = pd.DataFrame({
    "age": rng.integers(18, 80, n),
    "income": rng.normal(50_000, 15_000, n).round(0),
    "tenure_months": rng.integers(1, 120, n),
    "plan": rng.choice(["basic", "plus", "pro"], n),
})

# Hidden rule that generates the label, plus noise.
risk = (
    0.02 * (60 - df["tenure_months"])
    + 0.00001 * (50_000 - df["income"])
    + (df["plan"] == "basic") * 0.8
    + rng.normal(0, 0.5, n)
)
df["churned"] = (risk > 0.9).astype(int)

X = pd.get_dummies(df.drop(columns="churned"), columns=["plan"])
y = df["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

model = RandomForestClassifier(n_estimators=300, random_state=0)
model.fit(X_train, y_train)

proba = model.predict_proba(X_test)[:, 1]
print("AUC:", round(roc_auc_score(y_test, proba), 3))

Reading the example

• The label churned is known for every row, so this is supervised.
• The features mix numbers and a category. The category is turned into indicator columns.
• Stratified splitting keeps the class ratio equal in train and test.
• The model never sees the test rows while fitting.
• AUC is computed on the held-out rows only.

A numeric walk-through of the loss

Suppose three cars have true prices of 10, 12, and 20 (in thousands). A model predicts 11, 12, and 17. The errors are 1, 0, and 3. Squared errors are 1, 0, and 9. The mean squared error is 10 divided by 3, about 3.33. The root of that is about 1.83. Training tries to push this number down on training rows, while you watch the same number on validation rows. If training error keeps falling while validation error rises, the model has started to overfit.

Strengths and limitations

Strengths

• It has a clear goal. You know what you are optimizing and can measure it.
• It is well understood. There are mature tools, metrics, and checks.
• It scales from tiny tables to huge ones, with models from linear to deep.
• Results are easy to explain to non-experts, especially with trees and linear models.
• It supports a direct link to business value, since the label is often the outcome you care about.

Limitations

• It needs labels, and labels cost time and money. Some labels are slow, such as "did the loan default in two years".
• Labels can be noisy, biased, or delayed. The model learns those flaws.
• It only learns patterns present in training data. Shifts in the world hurt it.
• It can pick up shortcuts, such as an ID column that happens to correlate with the label.
• Rare events give few positive examples, which makes learning hard.
• A model trained for one task does not transfer to another by default.

How it compares with alternatives

Choosing among model families for tables

Role in tabular foundation models

This section matters most for readers of this site. A tabular foundation model is a model pre-trained to work on raw tables, not on one fixed dataset. Supervised learning is the task it is built to solve, and it also shapes how it is trained.

The task a table model serves

Most real table problems are supervised. A user has a table with a target column, some rows labeled and some not. They want predictions for the unlabeled rows. A foundation model must therefore accept labeled rows and produce predictions for new rows in a single pass.

Pre-training is supervised too

The pre-training of models such as TabPFN is supervised, but on a huge number of tables made by a generator. Each synthetic table has features and a target. The model sees a labeled part and must predict the target on a held-out part. Millions of such tables teach it a general prediction procedure. See synthetic data generation for how generators create varied tables. Real tables can also be used, often with self-supervised objectives that hide cells.

In-context learning replaces the training loop

In classic supervised learning, you fit a new model per table. In a foundation model, the labeled rows go into the input as context, and the model predicts for new rows directly. This is in-context learning. There are no weight updates at prediction time. The "training" has been moved into pre-training. This links supervised learning to few-shot learning, because the model can work with only a handful of labeled rows.

Row and column structure

Tables are not text and not images. Row order usually does not matter. Column order usually does not matter either. A good model respects both symmetries. Designs based on transformers use attention across rows and across columns. This is discussed on the architecture page.

Mixed types and missing values

Real tables mix numbers, categories, dates, and text. Many cells are missing. Supervised models must handle all of this without hand-made rules. Foundation models learn from pre-training to treat missing values as a signal, and they can also fill them. This ties to imputation networks.

No column names, or unhelpful ones

A table may have headers like "col_7". A model that depends on names would fail. Pre-trained table models can work from values and their relations alone. When names are present, they can add meaning, for example for sector or semantic classification of columns or of whole datasets.

One backbone, many supervised and related tasks

Once a model can read a table well, the same backbone can support several jobs:

• Classification and regression, the core supervised tasks.
• Imputation, which is supervised prediction of a hidden cell from the others.
• Anomaly detection, by scoring how unlikely a row is given the rest.
• Semantic or sector classification of a table or column.
• Continual learning, where new tables or new labeled batches update the system over time without forgetting older skills.

Benchmarks for such models appear on the benchmarks page.

A concrete picture of the foundation model workflow

Consider a small table with 300 labeled rows and 50 unlabeled rows.

1. The labeled rows and the unlabeled rows are placed together in the model input.
2. The model reads the table with attention across rows and across columns.
3. It compares each unlabeled row with the labeled rows it resembles.
4. It outputs a class probability or a number for each unlabeled row.
5. No gradient step happens. The same weights serve every table.

With a classic pipeline, the same job needs encoding, a split, a search over settings, and a final fit. The result may be similar, but the foundation model route is faster to start and has fewer ways to go wrong.

Where supervised signals still come from

• Human annotation by domain experts.
• Business outcomes recorded later, such as a payment that was or was not made.
• Sensors and logs, such as a failure code recorded by a machine.
• Weak labels from rules, which are noisy but cheap.
• Labels produced by a larger model, used to train a smaller one.

Limits that remain for table models

• Context length caps how many labeled rows can be used at once.
• Very wide tables with thousands of columns cost more to read.
• Huge tables may still favor boosted trees for speed.
• Labels with strong shift between training and use still hurt any model.

Why classic supervised models still matter

Tree ensembles remain strong baselines on large tables. Tuned XGBoost is a standard comparison. A foundation model proves its worth by matching or beating it with far less tuning on small and medium data, and by giving a single interface across many tasks.

Practical guidance and best practices

Before modeling

• Define the target precisely. State when it is measured and how.
• Check how labels were produced. Look for delay, bias, and noise.
• Split by time if the future will differ from the past. Random splits can flatter a model.
• Split by group if rows from the same entity appear many times, such as a customer with many orders.
• Remove or isolate identifiers that act as shortcuts.

During modeling

• Start with a simple baseline, such as a majority class or a linear model. You need a floor to beat.
• Try a tree ensemble next. It is a strong default on tables.
• Try a pre-trained table model when data is small or when tuning time is short.
• Use cross-validation for small data.
• Pick a metric that matches the real cost of errors.
• Check calibration if you use the probabilities for decisions.

After modeling

• Inspect errors by segment. An average score can hide a failing group.
• Explain predictions with feature importance or partial dependence.
• Track drift in inputs and in error.
• Plan retraining and keep the old model until the new one is verified.

Quick decision guide

• Under a few thousand rows, try a pre-trained table model and a tuned linear model.
• Thousands to millions of rows, try boosted trees first.
• Need full transparency, start with a linear model or a shallow tree.
• Need a probability that is trusted, check calibration and recalibrate if needed.
• Labels are scarce, look at semi-supervised, active, or transfer learning.
• Rewards instead of labels, look at reinforcement learning.
• Data cannot leave its owners, look at federated learning.

A short checklist

• Is the target well defined?
• Are train, validation, and test split in a way that matches use?
• Is any feature built from the future?
• Does the metric match the decision?
• Is there a simple baseline?
• Has the model been checked on subgroups?

Interpreting a trained model

People often need to know why a model made a call. Several tools help.

• Coefficients show direction and size in linear models.
• Tree paths show the exact questions that led to a prediction.
• Permutation importance shuffles one column and measures the drop in score.
• Partial dependence plots show the average effect of one feature.
• Shapley based explanations split one prediction into feature contributions.
• Counterfactual examples show the smallest change that would flip a decision.

Explanations describe the model, not the world. A feature can matter to the model without being a cause.

Fairness and safety

• Check error rates by group, not only the average.
• Remove direct sensitive columns, but remember proxies can remain.
• Document what the model should and should not be used for.
• Keep a human in the loop for high stakes decisions.

Common mistakes

• Evaluating on training data. The score looks great and means little.
• Tuning on the test set. The test score then stops being honest.
• Data leakage through a feature that is only known after the outcome. A "days since cancellation" column in a churn model is a classic case.
• Scaling or encoding on the full dataset before splitting. Statistics from test rows leak into training.
• Using accuracy on imbalanced data. A model that always says "no fraud" can score 99 percent.
• Ignoring time order. Shuffled splits on time series data give inflated results.
• Treating label noise as truth. If labels disagree, the ceiling is low.
• Over-tuning on a small validation set. Results then fail to repeat.
• Ignoring calibration when probabilities drive thresholds or costs.
• Forgetting that models inherit bias from labels. Check fairness across groups.
• Deploying without monitoring. Silent drift is common.

Frequently asked questions

What is supervised learning in simple words?

Supervised learning is teaching a model with examples that include the correct answer. The model compares its guess to the answer, measures the error, and adjusts. After many examples it can guess answers for new cases it has not seen.

What is the difference between classification and regression?

Classification predicts a category, such as yes or no, or one of several types. Regression predicts a number, such as a price or a duration. Both use labeled examples and the same overall training loop, but they use different losses and metrics.

How much labeled data do I need?

It depends on how hard the pattern is and how noisy the labels are. Simple patterns can be learned from a few hundred rows, while complex ones may need many thousands or more. Pre-trained table models and few-shot methods can work with far less, because they reuse knowledge from pre-training.

Is supervised learning the same as deep learning?

No. Deep learning is a family of model types, while supervised learning is a way of learning from labels. You can do supervised learning with linear models, trees, or neural networks. You can also use deep networks in unsupervised or self-supervised ways.

Why do tree models often beat neural networks on tables?

Tables have mixed types, uneven feature scales, and many weak or irrelevant columns. Trees handle these well and need little preprocessing. Plain neural networks need careful encoding and tuning to match them. Pre-trained transformer models for tables narrow this gap, especially on small data.

What is overfitting and how do I spot it?

Overfitting happens when a model learns noise in the training rows and then does worse on new rows. You spot it when training error keeps dropping while validation error stalls or rises. Remedies include simpler models, more data, regularization, and early stopping.

How does supervised learning relate to tabular foundation models?

Tabular foundation models are pre-trained on many tables to solve supervised prediction tasks in one pass. Given labeled rows as context, they predict for new rows without training a new model. So the supervised task stays the same, but the cost of fitting each new table drops sharply.

Can supervised learning handle missing values?

Yes, with the right method. Some models, such as certain gradient boosting systems, handle missing cells natively. Others need imputation first. Foundation models for tables learn to treat missingness as part of the data, and they can predict the missing cell itself.

When should I use semi-supervised or active learning instead?

Use them when labels are expensive but unlabeled rows are plentiful. Semi-supervised learning uses the unlabeled rows to improve the model. Active learning chooses which rows are worth labeling next, so each label adds the most value.

How do I know my model is good enough?

Compare it with a simple baseline on held-out data using a metric tied to the real decision. Check results by subgroup, check calibration, and test on data from a later period if the setting changes over time. If it beats the baseline by a meaningful margin and holds up in those checks, it is a strong candidate.

Glossary

• Backpropagation: A method to compute how each weight of a neural network affects the loss.
• Baseline: A simple model used as a point of comparison.
• Bias: Systematic error from wrong assumptions in the model or the data.
• Calibration: Agreement between predicted probabilities and observed frequencies.
• Classification: Predicting a category.
• Cross-validation: Estimating performance by training and testing on several splits.
• Data leakage: Use of information that would not be available at prediction time.
• Drift: A change over time in the data or in the label relation.
• Ensemble: A model made by combining several models.
• Feature: An input variable.
• Gradient descent: An optimization method that follows the slope of the loss downhill.
• Hyperparameter: A setting chosen before training.
• In-context learning: Using labeled examples in the input to predict, without updating weights.
• Label: The known answer for a training example.
• Loss: A measure of prediction error that training reduces.
• Overfitting: Fitting noise so that performance on new data suffers.
• Regression: Predicting a number.
• Regularization: Techniques that limit model complexity.
• Test set: Held-out data for the final score.
• Validation set: Held-out data used to tune settings.
• Variance: Sensitivity of a model to the particular training sample.

Summary

Supervised learning trains a model on rows with known answers so it can predict answers for new rows. It covers classification and regression, and it is the base of most business use of machine learning on tables. The loop is simple: choose a model, define a loss, fit on training data, tune on validation data, and report on a test set.

Most failures come from the data, not the algorithm. Leakage, bad splits, noisy labels, and drift cause more harm than a poor choice of model. Careful setup and honest evaluation matter most.

Tabular foundation models keep the supervised goal but move the learning step into pre-training on many tables. At use time, labeled rows act as context and predictions come in one pass. This links supervised learning to in-context and few-shot learning, to imputation, to anomaly detection, and to continual updates. Read more on the research, architecture, and benchmarks pages.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: