Özer Cem KELAHMETKELAHMET
AI Simulations · Large tabular foundation model

Anomaly detection with masked-cell surprise

Many anomalies are ordinary values in an impossible combination. Predict every cell from the rest of its row and from the other rows, and the rows whose real values are most improbable rise to the top.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The table ranks every row by its most surprising cell, measured in predicted standard deviations; injected rows are marked. The chart shows all scores, and the panel compares masked-cell surprise with nearest-neighbour distance and per-column z-scores by AUROC.

Things to try

  • Inject 12 anomalies and compare the three AUROC values.
  • Click a row to see which cell gave it away.
  • Alt-click any cell to corrupt it yourself and find it in the ranking.

How it works in real systems

The same idea underlies reconstruction-based detectors (autoencoders) and tabular foundation models used for anomaly scoring; isolation forests and local outlier factor are the classical baselines.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · A table with planted anomalies

import numpy as np
rng = np.random.default_rng(11)
def houses(n):
    s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
    p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
    return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
T = np.c_[X, y]; T_bad = T.copy(); lab = np.zeros(len(T), bool)
for i, f in zip(rng.choice(len(T), 6, replace=False), [0, 4, 3, 1, 4, 2]):
    T_bad[i, f] += rng.choice([-1, 1]) * T[:, f].std() * 2.5; lab[i] = True

Step 2 · Masked-cell surprise with cross-fitting

Predict every cell from the other columns with a model that never saw that row (5-fold cross-fitting), and score the row by its most surprising cell.

from sklearn.model_selection import KFold
from tabpfn import TabPFNRegressor
F = T_bad.shape[1]; surprise = np.zeros_like(T_bad)
for f in range(F):
    Xo = np.delete(T_bad, f, axis=1)
    for tr, te in KFold(5, shuffle=True, random_state=0).split(T_bad):
        reg = TabPFNRegressor().fit(Xo[tr], T_bad[tr, f])
        q10, q50, q90 = reg.predict(Xo[te], output_type="quantiles", quantiles=[0.1, 0.5, 0.9])
        surprise[te, f] = np.abs(T_bad[te, f] - q50) / ((q90 - q10) / 2.56 + 1e-9)   # in predicted standard deviations
row_score = surprise.max(1)

Step 3 · Compare with classical detectors

from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import StandardScaler
Z = StandardScaler().fit_transform(T_bad)
print("masked-cell surprise", roc_auc_score(lab, row_score))
print("isolation forest    ", roc_auc_score(lab, -IsolationForest(random_state=0).fit(Z).score_samples(Z)))
print("local outlier factor", roc_auc_score(lab, -LocalOutlierFactor(5).fit(Z).negative_outlier_factor_))
print("per-column z-score  ", roc_auc_score(lab, np.abs(Z).max(1)))

More simulations

← Calibration · All AI Simulations · Data pipeline →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open