Clustering

Overview

Clustering is the task of splitting a set of data points into groups so that points in the same group are more alike than points in different groups. It needs no labels. You give the algorithm rows of data and a notion of similarity, and it proposes a grouping. It is one of the main tools of Unsupervised learning.

Typical uses are customer segmentation, document grouping, image and embedding analysis, anomaly screening and data exploration. The main families are centroid methods such as k-means, hierarchical methods that build a tree of merges, density methods such as DBSCAN, and probabilistic models such as Gaussian mixtures. No single method is best. The result depends on how you represent the data, how you measure distance and what shape you expect the groups to take.

What it is

Clustering assigns each data point to a cluster. A cluster is a set of points that are close to one another under some measure. The output may be hard (each point belongs to exactly one cluster), soft (each point has a probability for each cluster) or hierarchical (clusters nested inside clusters).

Analogy: sorting a pile of mixed buttons into boxes. Nobody tells you what the boxes should be. You group by color, size and shape until each box looks coherent. A different person might sort by number of holes. Both sortings can be valid. The "right" answer depends on the goal.

Clustering versus classification

Inputs and similarity

Clustering depends on a distance or similarity function. Common choices:

• Euclidean distance: straight-line distance. Sensitive to scale.
• Manhattan distance: sum of absolute differences. More robust to outliers.
• Cosine similarity: angle between vectors. Common for text and embeddings.
• Gower distance: mixes numeric and categorical columns. Useful for tabular data.
• Jaccard distance: overlap of sets. Useful for binary features.
• Dynamic time warping: compares time series that shift in time.

If features have different scales, the largest-scale feature dominates the distance. Standardize or normalize before clustering.

Types of clusters

Clusters come in different shapes.

• Compact blobs: round groups around a center. K-means suits these.
• Elongated or curved groups: moons, rings and chains. Density methods suit these.
• Nested groups: sub-groups inside groups. Hierarchical methods suit these.
• Groups of different densities: hard for most methods. HDBSCAN-style methods help.
• Overlapping groups: mixture models give soft assignments.

Where it came from

Grouping by similarity is older than computers. Biologists classified species by traits. In the 1950s and 1960s, researchers in taxonomy, psychology and statistics developed numerical methods. Hierarchical agglomerative clustering emerged from numerical taxonomy. The k-means idea was described by several authors in the 1950s and 1960s, and a widely used iterative version appeared in the 1950s to 1980s literature under names such as Lloyd's algorithm. Gaussian mixture models fitted with the expectation maximization algorithm were formalized in the 1970s. Density-based clustering gained attention in the mid 1990s with DBSCAN, which defined clusters as dense regions separated by sparse regions. Spectral clustering, using the eigenvectors of a similarity graph, spread in the 2000s. Later work added scalable variants, mini-batch k-means, hierarchical density methods and deep clustering, which learns representations and cluster assignments together. Contrastive and self-supervised embeddings changed practice: today many teams embed first, then cluster the embeddings. See Contrastive learning and Self-supervised learning.

How it works

K-means

K-means partitions data into k clusters, each represented by its centroid (the mean of its points).

1. Choose k.
2. Pick k initial centroids. The k-means++ rule spreads them out.
3. Assign each point to its nearest centroid.
4. Recompute each centroid as the mean of its assigned points.
5. Repeat steps 3 and 4 until assignments stop changing.

The objective is to minimize the sum of squared distances from points to their centroids, often called inertia.

Analogy: choosing k meeting places for a town so that everyone walks the shortest total distance. Each person goes to the nearest place. Then each meeting place moves to the center of its crowd. Repeat until stable.

Strengths: fast, simple, scales well. Limits: assumes round, similar-sized clusters, needs k, sensitive to outliers and to the starting points.

K-medoids

Instead of the mean, use an actual data point (the medoid) as the center. This makes the method more robust to outliers and lets you use any distance, including Gower distance for mixed data. It is slower than k-means on large data.

Hierarchical clustering

Agglomerative clustering starts with each point alone and merges the two closest clusters repeatedly until one cluster remains. The result is a tree called a dendrogram. You cut the tree at a chosen height to get clusters.

Linkage decides how to measure distance between clusters.

Analogy: a family tree built from the bottom up. Cousins group into branches, branches into larger families.

DBSCAN

DBSCAN defines clusters as dense regions.

1. Choose a radius eps and a minimum number of points min_samples.
2. A point is a core point if at least min_samples points lie within eps of it.
3. Core points that are close to each other join the same cluster.
4. Non-core points near a core point join as border points.
5. Points that fit nowhere are labeled noise.

It finds clusters of arbitrary shape and flags outliers, with no need to choose k. It struggles when clusters have very different densities and when dimensionality is high, because distances lose contrast.

HDBSCAN extends this idea to varying densities by building a hierarchy of density levels and selecting stable clusters. It needs fewer parameters.

Gaussian mixture models

A Gaussian mixture assumes the data come from several Gaussian distributions. The expectation maximization algorithm alternates two steps.

• E-step: compute, for each point, the probability that each component generated it.
• M-step: update each component's mean, covariance and weight using those probabilities.

The result is a soft assignment. Because components can have different shapes and sizes, mixtures are more flexible than k-means. Model selection criteria such as BIC and AIC help choose the number of components.

Spectral clustering

Build a graph where points are nodes and edges carry similarity. Compute the eigenvectors of the graph Laplacian, use the top ones as new coordinates, and run k-means there. This can separate shapes that k-means cannot, such as two interleaved moons. It is costly on large datasets.

Other methods

• Mean shift: moves points uphill on a density estimate; finds modes without choosing k.
• Affinity propagation: points exchange messages to choose exemplars.
• BIRCH: builds a compact summary tree to cluster large data in one pass.
• Self-organizing maps: a neural grid that preserves neighborhoods.
• Deep clustering: an Autoencoders or contrastive encoder learns features while a clustering loss groups them.

Choosing the number of clusters

There is rarely a single correct k. Tools to guide the choice:

• Elbow method: plot inertia against k and look for the bend.
• Silhouette score: compares within-cluster closeness to nearest-other-cluster distance; ranges from -1 to 1.
• Gap statistic: compares inertia to that expected under random data.
• BIC/AIC: for mixtures.
• Stability: cluster resamples of the data and check whether groups persist.
• Business use: the number of groups people can act on. Three segments may be useful when twelve are not.

Evaluating clusters

Internal indices use only the data: silhouette, Davies-Bouldin, Calinski-Harabasz. External indices compare clusters to known labels when you have them: adjusted Rand index, normalized mutual information, purity. Practical checks ask whether clusters are stable, interpretable and useful for the decision at hand.

Clustering at scale

When the table has tens of millions of rows, plain k-means can still run, but you need care.

• Mini-batch k-means updates centroids from small random batches. It is much faster and uses little memory, at a small cost in quality.
• Sampling lets you fit centroids on a sample and then assign the full table in a single pass.
• BIRCH builds a compact summary tree in one pass over the data.
• Approximate neighbor search speeds up density methods. Libraries such as FAISS run k-means on vectors on GPUs.
• Distributed engines such as Apache Spark include k-means and Gaussian mixtures for data that does not fit in one machine.

A useful rule is to prototype on a sample of fifty thousand rows, settle the features and method there, and only then scale up.

Clustering text and embeddings

Text is a common input. The usual workflow is:

1. Turn each document into an embedding with a sentence encoder.
2. Optionally reduce dimensions with PCA or UMAP.
3. Cluster with k-means or HDBSCAN.
4. Label each cluster by its most frequent terms or by reading a few examples.
5. Review and merge clusters that mean the same thing.

Cosine distance is the usual choice for embeddings. Normalizing vectors to unit length makes Euclidean distance behave like cosine distance, so k-means can be used directly.

Profiling a cluster

A cluster is only useful when you can describe it. After clustering, build a profile table.

Compare each cluster's mean with the overall mean, measured in standard deviations. Features with large gaps define the cluster. Give each cluster a plain name and a one-sentence description. If two clusters have the same description, merge them.

Clustering as a feature

Cluster ids can be added to a supervised model as a categorical feature, or the distance to each centroid can be used as a set of numeric features. This often gives a small gain for linear models and little for boosted trees, which can find such splits on their own. Always fit the clusterer on training data only, then apply it to validation and test data, to avoid leakage.

Key concepts and terms

A worked example

The example uses scikit-learn on synthetic data. See scikit-learn for the library.

import numpy as np
from sklearn.datasets import make_blobs, make_moons
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering
from sklearn.mixture import GaussianMixture
from sklearn.metrics import silhouette_score, adjusted_rand_score

# Part 1: round blobs, where k-means is a good fit
X, y = make_blobs(n_samples=600, centers=4, cluster_std=1.0, random_state=3)
X = StandardScaler().fit_transform(X)

for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(X)
    sil = silhouette_score(X, km.labels_)
    print(f"k={k} inertia={km.inertia_:.1f} silhouette={sil:.3f}")

best = KMeans(n_clusters=4, n_init=10, random_state=0).fit(X)
print("ARI vs truth:", adjusted_rand_score(y, best.labels_))

# Part 2: two moons, where k-means fails and DBSCAN works
M, ym = make_moons(n_samples=500, noise=0.07, random_state=1)
km2 = KMeans(n_clusters=2, n_init=10, random_state=0).fit(M)
db = DBSCAN(eps=0.2, min_samples=5).fit(M)
print("moons k-means ARI:", adjusted_rand_score(ym, km2.labels_))
print("moons DBSCAN  ARI:", adjusted_rand_score(ym, db.labels_))

# Part 3: soft assignments with a mixture model
gmm = GaussianMixture(n_components=4, covariance_type="full", random_state=0).fit(X)
probs = gmm.predict_proba(X[:3])
print(np.round(probs, 3))
print("BIC:", gmm.bic(X))

What to expect:

• On the blobs, the silhouette score peaks near the true k of 4, and the adjusted Rand index is close to 1.
• On the moons, k-means cuts the shapes in half and scores poorly, while DBSCAN follows the curves.
• The mixture model returns a probability for each cluster for each point, which is useful when groups overlap.

Mixed numeric and categorical data

Real tables mix types. A simple approach is to one-hot encode the categories and scale everything, though many binary columns can dominate the distance. A better one for small and medium tables is to use Gower distance with k-medoids or hierarchical clustering. For large data, cluster on embeddings: train an autoencoder or use a pre-trained model to produce a numeric vector for each row, then run k-means or HDBSCAN on those vectors.

A numeric walk-through of one k-means step

Take five one-dimensional points: 1, 2, 3, 10, 11, with k = 2 and initial centers 1 and 10.

• Assign: points 1, 2, 3 are nearest to 1; points 10, 11 are nearest to 10.
• Update: the new centers are the means, 2 and 10.5.
• Assign again: the same grouping results, so the algorithm stops.

Inertia is (1-2)^2 + (2-2)^2 + (3-2)^2 + (10-10.5)^2 + (11-10.5)^2, which equals 2.5.

Strengths and limitations

Strengths

• Needs no labels, so it works where labeling is costly.
• Reveals structure and suggests hypotheses.
• Fast methods exist for millions of rows.
• Output is easy to communicate: "these are your segments."
• Works as a preprocessing step for other models, for example as cluster-id features.
• Helps find outliers, duplicates and data quality problems.
• Supports labeling: cluster first, then label a few points per cluster.

Limitations

• There is no ground truth, so "good" is subjective.
• Results depend heavily on features, scaling and distance.
• Many methods require choosing k or other parameters.
• Some methods assume cluster shapes that real data does not have.
• High dimensions weaken distance measures.
• Clusters can be unstable: small data changes shift the result.
• Clusters may reflect artifacts such as recording differences rather than real groups.
• Categorical and mixed data need extra care.
• Labeling people with segments can lead to unfair treatment if not reviewed.

How it compares with alternatives

Clustering also differs from neighbors in the visualization tool family. UMAP projects data to two or three dimensions for plotting. It is not a clustering algorithm itself, but it is often used before clustering or to inspect results. Be careful: clusters seen in a projection are not always real, and cluster sizes and distances in a UMAP plot should not be over-read.

Related tasks:

• Anomaly detection treats rare points as the target.
• Recommendation can use clusters of users or items to handle cold start.
• Isolation forest isolates outliers without forming clusters.
• Active learning can label one example per cluster to bootstrap a classifier.

Role in tabular foundation models

Clustering and tabular foundation models meet at several points. A tabular foundation model learns from raw tables with mixed types, missing values and often no column names. Clustering is a way to understand structure in the same kind of table without labels.

Row embeddings as clustering input

A key weakness of clustering on tables is the distance function. Mixed types, missing cells and different scales make a hand-built distance fragile. A model that has learned from many tables can turn each row into a vector that captures the joint behavior of its cells. Clustering those vectors, with k-means or HDBSCAN, is often more robust than clustering the raw columns. This is the same recipe used for text and images with Contrastive learning and Autoencoders. Architectural background is in /model/architecture.

Rows, columns and mixed types

Tabular foundation models such as TabPFN, TabICL, TabSTAR, FT-Transformer and SAINT encode numeric and categorical cells into tokens and let attention relate cells in a row and rows in a table. Some of them use contrastive or masked objectives that produce useful representations without labels, which suits clustering. Row-wise and column-wise attention is also a natural way to see which columns drive a grouping.

No column names

Many tables arrive with coded or missing column names. A distance that depends on knowing which column is "income" and which is "age" cannot be hand-tuned. Learned representations rely on the values and their relationships, so clustering in the learned space does not require names.

Missing values and imputation

Standard clustering methods cannot handle missing cells, so they need imputation first. Poor imputation creates fake clusters: all rows with a filled-in median bunch together. Learned imputers from Imputation networks and masked pre-training such as Masked cell modeling fill gaps using the joint distribution of the row. Better imputation leads to more faithful clusters. A useful check is to compare clusters before and after imputation and see whether the "missing" group disappears.

Pre-training on synthetic or real tables

Models pre-trained on a large prior of synthetic tables see many different cluster structures, noise levels and feature types. Their internal representations may reflect general structure of tabular data. This is a research area; verify on your own data rather than assuming it works. For details on evaluation practices see /model/benchmarks and /model/research.

In-context learning and label propagation

Clustering gives a cheap way to build context for in-context learning. Cluster the full table, label a few examples in each cluster, and give those labeled rows to a model that predicts for the rest. This is a form of semi-supervised learning; see Semi-supervised learning and Active learning.

Anomaly detection

Points far from every cluster, or in tiny clusters, are anomaly candidates. DBSCAN's noise label does this directly. In a learned embedding space, distance to the nearest cluster center is an outlier score. This complements Anomaly detection with Isolation forest.

Continual learning

When new data arrives, clusters drift. Cluster ids can change meaning from one run to the next, which breaks reports. Continual learning methods such as Experience replay help representations adapt without forgetting. In clustering, a practical approach is to freeze old centroids, assign new points to them, and create a new cluster only when many points are far from all existing centers.

Sector and semantic classification

Clusters of text or category descriptions often match semantic groups, such as industry sectors, and can seed a taxonomy. A supervised semantic classifier, like the heads described in Semantic sector heads, then assigns stable names. A common workflow is: cluster, inspect, name the clusters, and then train a classifier to reproduce the names on new data.

Honest limits

Clustering still depends on what you ask. A foundation model's representation reflects its training, and clusters from it need validation on your own data. Check stability, interpretability and downstream usefulness.

Practical guidance and best practices

Prepare the data

• Remove duplicates and obvious errors.
• Handle missing values on purpose; do not leave it to defaults.
• Transform skewed variables with logs or ranks.
• Scale features. Standardization is a common default.
• Drop or down-weight redundant features. Ten highly correlated columns count as one idea ten times.
• Reduce dimensions with PCA or a learned embedding if you have hundreds of columns.

Pick a method

• Start with k-means for large numeric data and round groups.
• Use hierarchical clustering to explore small data and see the tree.
• Use DBSCAN or HDBSCAN when shapes are irregular or you need noise detection.
• Use Gaussian mixtures when overlap is expected and you want probabilities.
• Use Gower distance with medoids for mixed types on small to medium data.

Choose and check k

• Try a range of k.
• Compare silhouette, elbow and stability.
• Look at cluster sizes. A tiny cluster may be noise; a huge one may hide structure.
• Describe each cluster with feature means or typical rows. If you cannot explain the cluster, it may not be useful.

Validate

• Run the method several times with different seeds.
• Cluster bootstrap samples and compare with the adjusted Rand index.
• Test whether clusters predict something that matters, such as churn or response.
• Review with a domain expert.

Visualize carefully

• Project with PCA for a faithful linear view, or UMAP for a nonlinear one.
• Color points by cluster.
• Remember that projections distort distances. Use them for exploration, not proof.

Operate

• Save the scaler, the model and the feature list together.
• Assign new points with the saved model.
• Monitor drift in cluster sizes and distances over time.
• Re-cluster on a schedule and map new clusters to old ones by centroid similarity.

Ethics

• Do not use clusters as a proxy for protected traits in ways that lead to unfair treatment.
• Review segment definitions before using them in pricing, lending or eligibility decisions.

A short end-to-end checklist

1. State the decision the clusters should support.
2. Choose and clean the features that matter for that decision.
3. Handle missing values and outliers explicitly.
4. Scale and, if needed, reduce dimensions.
5. Try two or three methods and a range of cluster counts.
6. Measure silhouette and stability.
7. Profile each cluster and name it.
8. Check the clusters against a domain expert and against a downstream outcome.
9. Save the full pipeline and define how new points are assigned.
10. Monitor drift and set a refresh schedule.

When not to cluster

Skip clustering if you already have labels for the target, since supervised learning will use them directly. Skip it if you have very few rows, where any grouping is mostly noise. Skip it if the features do not carry information about the thing you care about. And skip it if the result cannot change any decision, since a segmentation nobody uses is a cost with no benefit.

Common mistakes

1. No scaling. One large-scale feature controls all distances.
2. Choosing k by habit. Always test a range and check stability.
3. Using k-means on non-round shapes. It will cut curved groups apart.
4. Clustering in too many dimensions. Distances lose meaning; reduce dimensions first.
5. Trusting a 2D plot. Projections can create or hide clusters.
6. Ignoring categorical columns or encoding them badly. One-hot columns can dominate.
7. Imputing with the mean and clustering. The mean-filled rows form a fake cluster.
8. Using the labels you want to predict as a feature. The clusters just reproduce the label.
9. Treating clusters as ground truth. They are a view of the data, not a fact.
10. Skipping outlier handling. A few extreme points can pull k-means centroids.
11. One run only. Random starts can give different answers.
12. No plan for new data. Without assignment and drift rules, segments decay.

Frequently asked questions

What is clustering in machine learning?

Clustering is an unsupervised method that groups data points so that points in the same group are more similar than points in other groups. It uses no labels. It is used for segmentation, exploration, anomaly screening and as a step in larger pipelines.

How do I choose the number of clusters?

Try a range of values and compare several signals: the elbow in inertia, the silhouette score, information criteria for mixture models and the stability of the groups across resamples. Also consider how many groups people can act on. There is rarely one true number, so pick the value that is stable and useful.

What is the difference between k-means and DBSCAN?

K-means needs the number of clusters and finds round groups around centroids. DBSCAN finds dense regions of any shape, labels sparse points as noise and needs a radius and a minimum point count instead of k. K-means is faster on large data, while DBSCAN handles odd shapes and outliers better.

Can I cluster categorical or mixed data?

Yes. For small and medium tables, use Gower distance with hierarchical clustering or k-medoids. For larger data, convert rows to numeric vectors with embeddings and then cluster those vectors. Plain k-means on one-hot columns often works poorly, because the binary columns can dominate the distance.

Do I need to scale my data before clustering?

Almost always. Distance-based methods treat each feature by its raw range, so a column measured in thousands overwhelms a column measured in fractions. Standardizing or normalizing puts the features on a comparable scale. Tree-based and rank-based approaches are less sensitive, but distance methods are not.

How do I know if my clusters are good?

Use internal indices like the silhouette score, check stability across resamples and seeds, and look at whether clusters can be described in plain words. The final test is usefulness: do the clusters predict or explain something that matters for your decision? If labels exist, external indices such as the adjusted Rand index can also be used.

Is UMAP a clustering algorithm?

No. UMAP is a dimensionality reduction method that projects data to a few dimensions for visualization or as a preprocessing step. It is often used before a clustering method such as HDBSCAN, but the picture alone is not proof of clusters. See the UMAP page for details.

Can tabular foundation models help with clustering?

They can help by providing row representations that handle mixed types, missing values and anonymous columns better than hand-built distances. Clustering the learned vectors with k-means or HDBSCAN is a common recipe. You should still validate stability and usefulness on your own data.

How does clustering relate to anomaly detection?

Anomalies are points that do not fit the main groups. Density methods like DBSCAN label such points as noise, and distance to the nearest cluster center can serve as an outlier score. Dedicated detectors like isolation forest are often better when outliers are the main goal.

How often should I re-cluster?

Re-cluster when the data distribution shifts, which you can detect by tracking cluster sizes and average distances to centroids. Many teams refresh monthly or quarterly. Between refreshes, assign new points to existing centroids so that segment ids stay stable for reports.

Glossary

• Agglomerative clustering: Bottom-up hierarchical method that merges the closest clusters.
• BIC: Bayesian information criterion, used to compare mixture models.
• Centroid: The mean of the points in a cluster.
• Cluster: A group of similar points.
• Cosine similarity: A measure based on the angle between two vectors.
• Dendrogram: A tree showing the order of merges.
• Density-based clustering: Grouping dense regions separated by sparse ones.
• Divisive clustering: Top-down hierarchical method that splits clusters.
• Elbow method: Choosing k at the bend of the inertia curve.
• EM algorithm: Expectation maximization, used to fit mixture models.
• Gower distance: A distance that handles mixed numeric and categorical data.
• Inertia: Sum of squared distances to centroids.
• K-means++: An initialization that spreads out starting centroids.
• Linkage: The rule for distance between clusters.
• Medoid: The most central actual point in a cluster.
• Silhouette score: A measure of how well points fit their cluster.
• Soft clustering: Assigning each point a probability for each cluster.
• Spectral clustering: Clustering using eigenvectors of a similarity graph.

Summary

Clustering groups similar rows without labels. The main families are centroid methods like k-means, hierarchical methods, density methods like DBSCAN and HDBSCAN, and probabilistic mixtures. The quality of a clustering depends on the features, the scaling, the distance and the number of groups, and it must be judged by stability and usefulness because no ground truth exists.

On tables with mixed types, missing cells and unnamed columns, hand-built distances are fragile. Learned row representations from tabular foundation models and masked or contrastive pre-training offer a more robust space to cluster in, and good imputation prevents fake clusters. Clusters also support anomaly screening, semi-supervised labeling and taxonomy building. Prepare the data, test several methods and values of k, validate with resamples and experts, and plan for drift. For more background see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Forecasting and tasks in my stack, under Foundation models and applied ML. Others in the same group: