Anomaly detection with masked-cell surprise
Many anomalies are ordinary values in an impossible combination. Predict every cell from the rest of its row and from the other rows, and the rows whose real values are most improbable rise to the top.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The table ranks every row by its most surprising cell, measured in predicted standard deviations; injected rows are marked. The chart shows all scores, and the panel compares masked-cell surprise with nearest-neighbour distance and per-column z-scores by AUROC.
Things to try
- Inject 12 anomalies and compare the three AUROC values.
- Click a row to see which cell gave it away.
- Alt-click any cell to corrupt it yourself and find it in the ranking.
How it works in real systems
The same idea underlies reconstruction-based detectors (autoencoders) and tabular foundation models used for anomaly scoring; isolation forests and local outlier factor are the classical baselines.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · A table with planted anomalies
import numpy as np
rng = np.random.default_rng(11)
def houses(n):
s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
T = np.c_[X, y]; T_bad = T.copy(); lab = np.zeros(len(T), bool)
for i, f in zip(rng.choice(len(T), 6, replace=False), [0, 4, 3, 1, 4, 2]):
T_bad[i, f] += rng.choice([-1, 1]) * T[:, f].std() * 2.5; lab[i] = True
Step 2 · Masked-cell surprise with cross-fitting
Predict every cell from the other columns with a model that never saw that row (5-fold cross-fitting), and score the row by its most surprising cell.
from sklearn.model_selection import KFold
from tabpfn import TabPFNRegressor
F = T_bad.shape[1]; surprise = np.zeros_like(T_bad)
for f in range(F):
Xo = np.delete(T_bad, f, axis=1)
for tr, te in KFold(5, shuffle=True, random_state=0).split(T_bad):
reg = TabPFNRegressor().fit(Xo[tr], T_bad[tr, f])
q10, q50, q90 = reg.predict(Xo[te], output_type="quantiles", quantiles=[0.1, 0.5, 0.9])
surprise[te, f] = np.abs(T_bad[te, f] - q50) / ((q90 - q10) / 2.56 + 1e-9) # in predicted standard deviations
row_score = surprise.max(1)
Step 3 · Compare with classical detectors
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import StandardScaler
Z = StandardScaler().fit_transform(T_bad)
print("masked-cell surprise", roc_auc_score(lab, row_score))
print("isolation forest ", roc_auc_score(lab, -IsolationForest(random_state=0).fit(Z).score_samples(Z)))
print("local outlier factor", roc_auc_score(lab, -LocalOutlierFactor(5).fit(Z).negative_outlier_factor_))
print("per-column z-score ", roc_auc_score(lab, np.abs(Z).max(1)))