SVM

Overview

A support vector machine, or SVM, is a model that separates classes with the widest possible boundary. Given labeled points, it looks for the line, plane or higher-dimensional surface that keeps the two classes as far apart as it can. Only the points closest to the boundary, called support vectors, decide where it goes. All other points could be removed without changing the result.

SVMs became popular because of two ideas. The first is the maximum margin, which gives good generalization. The second is the kernel trick, which lets a linear method draw curved boundaries by working in a transformed feature space without computing that space directly. SVMs are best on small to medium datasets with a clear structure and a moderate number of features. They are less suited to very large tables, where training time grows quickly. They remain a core method in classical machine learning and a useful baseline for tabular work. For the toolkit that implements them see scikit-learn.

What it is

An SVM is a supervised learning method used for classification, regression and, in a variant, anomaly detection. In its simplest form it handles two classes. It learns a decision function and predicts the class from its sign.

The core picture

Picture red and blue marbles scattered on a table. You want to lay a straight stick between them so that red is on one side and blue on the other. Many placements work. The SVM chooses the placement that leaves the biggest empty strip, the margin, between the stick and the nearest marbles on each side. The marbles touching the edge of that strip are the support vectors.

Why this choice? A boundary that sits close to some points is fragile. A small shift in the data could move a point to the wrong side. A wide margin gives room for error, so new points are more likely to land on the correct side.

Linear SVM

A linear SVM uses a boundary of the form w · x + b = 0. Here w is a vector of weights and b is an offset. The prediction for a point is the sign of w · x + b. The margin width is 2 / ||w||, so maximizing the margin means minimizing the length of w.

Soft margin

Real data is rarely perfectly separable. A soft-margin SVM allows some points to be inside the margin or on the wrong side, at a cost. A setting called C controls the trade-off:

• Large C: penalize mistakes heavily, narrow margin, risk of overfitting.

• Small C: tolerate mistakes, wider margin, simpler boundary.

Analogy: C is how strict a teacher is. A strict teacher punishes every error and may force the class into odd habits. A relaxed teacher accepts a few mistakes and focuses on the main lesson.

Kernel SVM

A kernel SVM uses a kernel function to measure similarity between points. This lets it draw curved boundaries. Common kernels include:

Regression and other tasks

Support vector regression, or SVR, fits a tube around the data. Errors inside the tube cost nothing. Errors outside cost in proportion to their size. A one-class variant learns the boundary of normal data for anomaly detection. See One-class SVM.

Where it came from

The roots go back to the 1960s, when researchers proposed methods for finding optimal separating hyperplanes. The modern form came in the 1990s. A paper from the early 1990s introduced the kernel idea into margin classifiers, and a widely cited paper in the mid 1990s introduced the soft-margin version that handles non-separable data. Through the late 1990s and 2000s, SVMs were among the strongest general-purpose classifiers, used in text classification, image recognition, bioinformatics and many other fields.

Why it rose

Several qualities made SVMs attractive. The training problem is convex, so it has a single global optimum and no bad local minima. The margin gave a theoretical story about generalization. Kernels let practitioners encode domain knowledge as a similarity measure. And implementations such as LIBSVM made the method easy to use.

What changed later

As datasets grew and deep learning matured, other methods took over many tasks. Kernel SVM training scales poorly with the number of rows, and tuning a kernel by hand became less appealing than learning features automatically. On tabular data, tree ensembles such as Random forest, XGBoost and LightGBM often won. Even so, SVMs remain popular for small datasets, for high-dimensional problems and as a teaching tool for the ideas of margins and kernels.

Ideas that carried forward

The kernel view of similarity returned in modern research. Attention in transformers can be read as a similarity-weighted average, and some analyses connect it to kernel methods. In-context learning on tables, as in TabPFN, also weighs training rows by their relationship to a query row. The link is conceptual, not a claim that these models are SVMs.

How it works

This section builds the SVM from the ground up.

Step 1: Represent each row as a point

Each row of a table becomes a point in space. A table with five numeric columns gives points in five dimensions. Categories must be turned into numbers first, for example with one-hot encoding. Scaling is crucial, as explained below.

Step 2: Find the maximum margin boundary

For labels y of +1 or -1, the hard-margin problem asks for the w and b that satisfy y_i (w · x_i + b) >= 1 for all points while making ||w|| as small as possible. The points where the inequality is an equality are the support vectors.

Analogy: stretching a rubber band between two groups of nails. The band touches only the outermost nails. Those nails determine its shape.

Step 3: Allow mistakes with slack

The soft-margin problem adds slack variables. Each point can violate the margin by some amount, and the objective adds C times the total violation. The loss on a single point is the hinge loss: zero if the point is correctly classified beyond the margin, and growing linearly as it gets closer or crosses the boundary.

Step 4: Solve in the dual form

The problem can be rewritten so that it depends on the data only through dot products between pairs of points. In this dual form, each training point gets a weight alpha. Most weights are exactly zero. The points with non-zero weights are the support vectors. The decision function becomes a weighted sum of similarities to the support vectors:

f(x) = sum_i alpha_i y_i K(x_i, x) + b

Here K is the kernel. For a linear kernel it is the dot product.

Step 5: Use the kernel trick

Suppose a ring of blue points surrounds a cluster of red points. No straight line can separate them. If you add a feature equal to the squared distance from the center, the classes become separable by a plane in the new space. A kernel does this implicitly. It gives the dot product in a high-dimensional (even infinite-dimensional) feature space without ever constructing it.

Analogy: two dots on a flat sheet of paper are hard to separate by a line. Lift the sheet into a cone shape, and a flat cut at a certain height separates them. The kernel lets you compute as if the paper were lifted, without doing the lifting.

Step 6: The RBF kernel and gamma

The RBF kernel is K(x, z) = exp(-gamma * ||x - z||^2). The parameter gamma controls how far the influence of a single point reaches:

• Large gamma: each point influences only its immediate neighborhood. The boundary can become very wiggly and overfit.

• Small gamma: each point influences a wide region. The boundary is smooth, and may underfit.

C and gamma interact, so they are tuned together on a log scale.

Step 7: Handle many classes

SVMs are binary at heart. For many classes, two approaches are common. One-versus-rest trains one model per class against all the others. One-versus-one trains a model for every pair of classes and takes a vote. Libraries choose one of these for you.

Step 8: Get probabilities if needed

An SVM outputs a score, the signed distance to the boundary, not a probability. Platt scaling fits a small logistic model on the scores to produce calibrated probabilities. This costs an extra cross-validation inside training.

Step 9: Scale and train

Training time grows faster than linearly with the number of rows for kernel SVMs, often between quadratic and cubic in practice. For datasets with hundreds of thousands of rows, linear SVMs trained with specialized solvers, or approximate kernels, are the practical choice.

Why scaling matters

An SVM depends on distances or dot products. If one column ranges from 0 to 1,000,000 and another from 0 to 1, the large column dominates. Standardize each numeric column before training. Put the scaler inside a pipeline so it learns only from training folds.

Key concepts and terms

A worked example

This section includes a numeric walk-through and a code example.

A tiny numeric example

Take two features and four points.

The closest opposite pair is B at (2, 1) and C at (4, 3). The vector between them is (2, 2), whose length is about 2.83. The maximum-margin boundary is the line perpendicular to this vector, halfway between them. The midpoint is (3, 2), so the boundary is x1 + x2 = 5. B and C are support vectors. A and D could be deleted without changing the boundary.

Check a new point (3, 3). Its sum is 6, greater than 5, so it falls on the +1 side. The point (2, 2) has sum 4, so it falls on the -1 side. The margin is the distance from the line to B (or C). The value (5 - 3) / sqrt(2) is about 1.41, so the total margin width is about 2.83, matching the gap between B and C.

Code: pipeline with scaling and an RBF kernel

import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=600, noise=0.25, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0)

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("svm", SVC(kernel="rbf")),
])

grid = {
    "svm__C": [0.1, 1, 10, 100],
    "svm__gamma": [0.01, 0.1, 1, "scale"],
}
search = GridSearchCV(pipe, grid, cv=5, scoring="accuracy", n_jobs=-1)
search.fit(X_train, y_train)

print("best params:", search.best_params_)
print("cv accuracy:", round(search.best_score_, 3))
print("test accuracy:", round(search.score(X_test, y_test), 3))
print("support vectors:", search.best_estimator_["svm"].n_support_)

The two-moons data cannot be separated by a straight line. A linear SVM would fail on it. The RBF kernel draws a curved boundary. The last line prints how many support vectors each class uses. A model that uses almost all points as support vectors is likely overfitting or has a poor setting for C or gamma.

Code: linear SVM for wide, sparse data

from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer

texts = ["refund for late delivery", "invoice total is wrong",
         "package arrived damaged", "billing address update"]
labels = ["shipping", "billing", "shipping", "billing"]

clf = Pipeline([
    ("tfidf", TfidfVectorizer()),
    ("svm", LinearSVC(C=1.0)),
])
clf.fit(texts, labels)
print(clf.predict(["late package"]))

Linear SVMs are strong on text-like data, where the number of features is large and the data is sparse. This is also the style of model used for simple semantic classification of text fields in tables.

Code: SVR for regression

from sklearn.svm import SVR

reg = Pipeline([("scale", StandardScaler()),
                ("svr", SVR(kernel="rbf", C=10, epsilon=0.1))])
reg.fit(X_train[:, :1], X_train[:, 1])   # toy example

epsilon sets the width of the tube. Errors smaller than epsilon cost nothing.

How to read the grid search

Plot the cross-validated score as a heat map over C and gamma. A good region is a broad plateau, not a single sharp peak. If the best cell is at the edge of the grid, extend the grid in that direction and search again.

Strengths and limitations

Strengths

• Strong generalization through the margin principle.

• Convex training: a single global optimum.

• Works well when the number of features is large compared with the number of rows.

• Kernels let you plug in domain-specific similarity.

• Only support vectors matter, so the fitted model can be compact.

• Reasonably robust with a good scale and settings.

• Linear SVMs scale well to high-dimensional sparse data.

Limitations

• Kernel training is slow on large tables.

• Sensitive to feature scaling.

• Needs tuning of C, gamma and the kernel.

• No native probabilities.

• Not natively handling missing values or categorical features.

• Hard to interpret with non-linear kernels.

• Predictions cost time proportional to the number of support vectors.

• Noisy labels can inflate the number of support vectors.

• On mixed-type tabular data, tree ensembles often reach higher accuracy with less preprocessing.

How it compares with alternatives

A practical rule: if your table is small, numeric, and scaled well, an RBF SVM is a strong candidate. If your table has many categories, missing values and large row counts, start with trees. If your data is text or very high-dimensional and sparse, a linear SVM is hard to beat as a fast baseline.

Role in tabular foundation models

Tabular foundation models are pre-trained on many tables so that they can predict on a new table without task-specific training. SVMs are not part of these models, but they relate to them in several useful ways.

A classic baseline on the same input shape

An SVM reads a matrix with rows as samples and columns as features, the same input shape that tabular foundation models read. In benchmark studies of tabular models, SVMs are one of the standard classical baselines, alongside linear models, forests and boosting. They help show where a foundation model adds value, especially on small datasets, where SVMs are often at their best. See /model/benchmarks.

Similarity, kernels and in-context learning

A kernel SVM predicts using a weighted sum of similarities between a query and the support vectors. In-context learning on tables does something related: the model reads labeled example rows and the query row together and produces a prediction that depends on how the query relates to the examples. In a transformer, attention computes similarity scores between rows and uses them as weights. The two are different mechanisms, but both turn "which known rows look like this one" into a prediction. An SVM learns a fixed sparse set of important rows. A foundation model learns a general procedure for weighing the rows during pre-training. See /model/architecture.

Mixed types and missing values

SVMs need numeric, scaled, complete inputs. A pipeline must one-hot encode categories and impute missing cells. Tabular foundation models are built to accept mixed types and missing values more directly, which reduces the preprocessing work. When comparing the two, use the same pipeline for the SVM and note that its result depends on the quality of that preprocessing. See Imputation networks.

No column names

SVMs ignore column names. They treat each column as an unnamed dimension, so they fit the schema-agnostic setting naturally. Foundation models that also work without names are in the same position, and the SVM is an easy reference point for what can be done from values alone.

Pre-training on synthetic or real tables

SVMs are trained from scratch per dataset. They have no pre-training and cannot transfer knowledge between tables. Tabular foundation models pre-train on large numbers of generated or real tables, learning general priors about how features relate to targets. That is the main difference. The SVM must estimate everything from the new table, so it is sensitive to small sample sizes in the choice of C and gamma. A pre-trained model brings a prior. See Synthetic data generation.

Continual learning

Standard SVMs have no simple update rule. When new data arrives, you usually retrain. Online variants exist but are less common. Foundation models can take new rows as fresh context, and can be updated by further training. See Continued pre-training.

Anomaly detection and classification

The one-class form of SVM is a classical method for finding rows that do not look like normal data. Foundation models can also score how unusual a row is. The SVM version is a standard baseline in that comparison. See One-class SVM and Anomaly detection. For semantic or sector classification from text fields, a linear SVM over TF-IDF features is a fast and strong baseline that can be placed beside a foundation model. For more background on model families see /model/research.

Practical guidance and best practices

Prepare the data

• Standardize numeric columns inside a pipeline.

• One-hot encode low-cardinality categories. Avoid very high-cardinality encodings, which blow up the dimension.

• Impute missing values and consider missing indicators.

• Remove or cap extreme outliers, which can pull the boundary.

• For imbalanced classes, use class_weight="balanced" and evaluate with metrics that fit the goal.

Choose the kernel

1. Start with a linear kernel if the number of features is large or the data is sparse.

2. Try RBF as the general default.

3. Use polynomial only if you have a reason to expect low-order interactions.

4. Use a custom kernel when you have a meaningful similarity measure.

Tune on a log scale

Search C from 0.01 to 1000 and gamma from 0.0001 to 10 on a log grid. Use cross-validation. Start coarse, then refine around the best region. Beware of the edge of the grid.

Control cost

• Subsample for tuning, then refit on all data.

• Use LinearSVC or a stochastic gradient solver for large data.

• Use kernel approximations such as random Fourier features or Nystroem to approximate RBF at linear cost.

• Cache kernel computations if memory allows.

Check the fitted model

• Count support vectors. A very high share suggests overfitting or noise.

• Inspect errors by group.

• Calibrate probabilities if you need them.

• Compare with a simple baseline such as logistic regression and with a tree ensemble.

Report honestly

Use a held-out test set that was not used for tuning. Report the mean and spread over repeated cross-validation or several random splits.

A quick settings guide

Choosing between SVM and other models in a project

Think about four facts about your data before you pick a model.

1. How many rows do you have? Under about twenty thousand, a kernel SVM is practical. Above that, plan for a linear solver or a different model.

2. How many columns, and what types? Mostly numeric and dense suits SVMs. Many categories and gaps suit trees.

3. Do you need probabilities? If yes, plan for calibration, or choose logistic regression or a tree ensemble.

4. Do you need to explain each prediction? Linear models and trees are easier to explain than kernel SVMs.

Reproducible evaluation

• Fix the random seed for splits.

• Use repeated stratified cross-validation, for example five folds repeated three times.

• Record the exact scaler, encoder and imputer settings.

• Save the best parameters and the number of support vectors with each run.

• Compare against a dummy baseline, so you know how much signal exists.

Reading the support vectors

The support vectors tell you which training rows shaped the model. Looking at them can reveal mislabeled rows, duplicates and edge cases. Rows that are support vectors in many cross-validation folds are often the hard or noisy ones. Reviewing them is a cheap data quality check.

Common mistakes

1. Forgetting to scale features.

2. Fitting the scaler on the full data before splitting.

3. Using the default C and gamma without tuning.

4. Searching a linear grid instead of a log grid.

5. Using an RBF kernel on very large data and waiting hours.

6. Treating decision scores as probabilities.

7. One-hot encoding a huge-cardinality column and exploding the dimension.

8. Ignoring class imbalance and reporting accuracy.

9. Leaving outliers that dominate the margin.

10. Picking the kernel by test accuracy and then reporting that same number.

11. Reading feature importance from a non-linear kernel as if it were linear weights.

12. Expecting an SVM to match tree ensembles on large mixed-type tables without feature work.

Frequently asked questions

What does SVM stand for and what does it do?

SVM stands for support vector machine. It is a supervised model that separates classes with the widest possible boundary, called the maximum margin. The points nearest the boundary, the support vectors, define it. With kernels, it can draw curved boundaries. It is used for classification, regression and anomaly detection.

What is the kernel trick?

The kernel trick lets a linear method work as if the data had been moved into a richer feature space, without computing that space. A kernel function gives the similarity between two points as if they were in the higher-dimensional space. The SVM uses only these similarities, so it can fit curved boundaries at the cost of pairwise computations.

What do C and gamma control?

C controls how strongly the model punishes mistakes. A large value gives a narrow margin and a tight fit, and a small value gives a wider margin and a simpler boundary. Gamma, used in the RBF kernel, controls how far each point's influence reaches. A large value gives a very local, wiggly boundary, and a small value gives a smooth one.

Why do SVMs need scaled features?

Because they rely on distances and dot products. A column with a large numeric range will dominate the calculation and the other columns will barely matter. Standardizing each numeric column to a similar range lets all of them contribute fairly. The scaler should be fit on training data only, inside a pipeline.

Are SVMs still useful today?

Yes, in the right setting. They are strong on small to medium numeric datasets, on high-dimensional sparse data such as text, and as a baseline that is easy to explain. For large mixed-type tables, tree ensembles or pre-trained tabular models are often a better first choice, but SVMs still appear in careful benchmarks.

How do SVMs handle multiple classes?

They combine several binary classifiers. One-versus-rest trains one model per class against all others and picks the highest score. One-versus-one trains a model for each pair of classes and takes a vote. Software libraries usually do this automatically.

Can an SVM give probabilities?

Not directly. It outputs a signed distance to the boundary. Platt scaling or another calibration method can map these scores to probabilities by fitting a small model on held-out scores. This adds training time, and the quality of the probabilities should be checked with a reliability curve.

How do SVMs relate to tabular foundation models?

They serve as a classical baseline and share the same input shape: rows as samples and columns as features. They differ because an SVM is trained from scratch on each table, while a tabular foundation model is pre-trained on many tables and can use context rows at prediction time. Comparing the two shows how much a pre-trained prior helps on small data.

When should I choose a linear SVM over an RBF SVM?

Choose a linear SVM when you have many features, especially sparse ones like text, or when the dataset is large. It trains much faster and often matches the RBF kernel in those settings. Choose RBF when the data has a clear non-linear structure and is small enough to train comfortably.

What is a support vector regression model?

Support vector regression, or SVR, applies the same ideas to predicting numbers. It fits a function so that most points lie within a tube of width epsilon around it, and only points outside the tube affect the fit. It handles non-linear relationships with kernels and is controlled by C, epsilon and the kernel settings.

Glossary

• Hinge loss: a loss that penalizes points inside or beyond the margin on the wrong side.

• Hyperplane: a flat decision surface.

• Kernel: a similarity function between two points.

• Margin: the gap between the boundary and the nearest points.

• Nystroem approximation: a method that approximates a kernel with a smaller feature map.

• Platt scaling: converting scores to probabilities with logistic regression.

• RBF kernel: a kernel whose similarity decays with squared distance.

• Slack variable: the amount by which a point violates the margin.

• Support vector: a training point that defines the boundary.

• SVR: support vector regression.

Summary

A support vector machine finds the boundary with the widest margin between classes, and uses a small set of support vectors to define it. With a soft margin it tolerates noise, and with kernels it can draw curved boundaries. It is convex, principled and strong on small to medium numeric data, but it needs scaling, tuning and care with cost on large tables.

For tabular foundation models, the SVM is a classical reference point with the same input shape. It shows what can be done from values alone, and its sensitivity to preprocessing and sample size highlights what pre-training adds: priors, built-in handling of mixed types and missing values, and in-context use of labeled rows. Use SVMs as baselines, as fast linear text classifiers, and as the base of one-class anomaly detection. For related reading see scikit-learn, One-class SVM, Random forest, XGBoost, TabPFN, Anomaly detection, /model/architecture, /model/research and /model/benchmarks.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: