Anomaly detection benchmark: finding rows that do not fit, with no labels

The anomaly detection benchmark asks a model to rank rows by how strange they are.

This page explains the setting, how I built the detector, what made the difference, and what stayed open.

The short version is this.

A label-free selector decides how much to trust each channel.

The detector is first among thirteen methods by both average rank and AUROC.

There are no scores on this page on purpose.

The model card carries them with their protocols.

For background on the task, see Anomaly detection and Isolation forest.

For the wider list, see Benchmarks.

What this benchmark measures

An anomaly is a row that does not fit the normal pattern.

A card payment from a country the customer has never visited.

A sensor stuck at one value.

The benchmark ranks rows that do not fit the normal distribution.

A good method puts the anomalies near the top of the ranking.

The semi-supervised setting

The setting is semi-supervised.

The model is trained on normal rows only.

It never sees a labelled anomaly.

Then it is scored on a test set that contains both normal rows and anomalies.

This is the setting in which the published methods are measured, so I used it too.

Think of a security guard who has studied a building for a year.

The guard has only seen normal days.

On the day of a break-in, the guard notices what is wrong because it differs from everything seen before.

The guard was never shown a burglar.

ADBench

ADBench is a well-known benchmark suite for anomaly detection on tabular data.

It collects many datasets and many methods under one protocol.

I use its semi-supervised setting.

See Evaluation for how benchmark scores should be read in general.

Two metrics

• Average rank. For each dataset, rank all methods. Then average the ranks. A lower average rank is better.
• AUROC. The area under the receiver operating characteristic curve. It measures how well the score separates anomalies from normal rows. A value of 0.5 is a coin flip. A value of 1.0 is perfect.

AUROC is threshold-free.

It does not need you to pick a cut-off.

That makes it a good fit for a ranking task.

Who the model is compared with

Ten classical detectors and published model pools are the rivals.

Classical detectors include methods based on distance, density and isolation.

Published model pools combine several detectors.

Competitor numbers are third-party published values cited to their primary source.

I did not re-run them.

The final comparison has thirteen methods in all.

How I built it

The path was a sequence of wrong turns, each one fixed in turn.

The first chain used the wrong engine

My first chain used the weaker imputer.

In the model, imputation and anomaly scoring are linked.

Imputation reconstructs a hidden cell from the rest of the row.

If a cell is very different from what the model expects, that is a hint of an anomaly.

The idea is simple.

Hide a value.

Ask the model to guess it.

If the guess is far from the truth, the row is strange.

My first chain used the weaker of two imputers.

It was a poor engine for the job.

I switched to the imputation engine, the stronger one, and then fixed its integration bugs one at a time.

See Missing-value imputation for how that engine was built.

Bug 1: the output scale

The first bug was the output scale.

The imputation head returns values on its own internal scale.

To compare a guess with the real value, both must be on the same scale.

I fixed it with a per-column linear calibration.

For each column, I fit a simple straight line that maps the model's output to the real scale.

It is cheap and it works.

Bug 2: an empty head when no data is missing

The second bug was an empty head.

If the data has no missing values, the imputation head has nothing to learn from.

It comes out empty.

Training data is often complete. So this case is common.

My fix is to inject 15 percent random missingness before fitting.

I hide 15 percent of the cells at random, on purpose, then train the head to fill them.

This gives the head something to learn from, and it matches the way the model was pre-trained.

See Pretraining for the masked cell modeling objective.

Bug 3: import-time side effects

The third bug was import-time side effects.

Loading part of the code changed global state.

This made results depend on the order in which files were loaded.

This kind of bug is quiet.

Nothing crashes.

The numbers are just slightly wrong, and different between runs.

I removed the side effects so that the same input gives the same output.

The final detector: three model-based channels

The final detector has three model-based channels.

Each is a different way of asking, "how strange is this row?"

1. Imputation residual. The residual of the imputation head. Hide cells, let the model guess them, and measure how far off the guess is. A large residual means the row does not follow the patterns the model learned from normal rows.
2. Embedding neighbours. Nearest neighbours in the model's embedding space. The model turns each row into a vector. For a test row, find its closest normal rows in this space. If even the closest normal rows are far away, the test row is odd.
3. Raw neighbours. Nearest neighbours in the raw space. The same idea, but on the original values, with no model in between.

An isolation forest joins as an auxiliary channel.

It works differently.

It builds random trees that try to isolate each row.

Anomalies are isolated in fewer splits, because they sit away from the crowd.

See Isolation forest.

Why more than one channel

Different anomalies look different.

Some are extreme in a single column.

Some are normal in each column but odd in combination.

Some sit in a sparse region.

No single signal catches all of them.

It is like a team of inspectors.

One checks weights.

One checks labels.

One checks smell.

A bad item may fail only one inspection.

The hard part: which channel to trust

Having several channels raises a question. For a given dataset, which should I trust?

In the supervised setting, you could check labelled anomalies.

In the semi-supervised setting, you have none.

So the choice must be made without labels.

The pseudo-anomaly selector

Which channel to trust is decided without labels, by a pseudo-anomaly selector.

I build fake anomalies from the training rows. Two methods:

• Shuffling features. Take normal rows and shuffle some values between rows. Each value is still plausible by itself, but the combination is broken.
• Injecting uniform values. Replace some values with random draws across the range of the column. These are the kind of extreme values a real fault may produce.

Then I measure each channel on these fake anomalies.

A channel that separates the fakes from the real normal rows well is likely to be useful on this dataset.

A channel that does not is likely to be weak.

I convert these measurements into weights with a softmax.

A softmax turns a list of scores into weights that are positive and add up to one.

A channel with a much better score gets a much larger weight.

Rank fusion

The ranks are then fused.

Instead of adding raw scores, which live on different scales, I convert each channel's scores into ranks and combine the ranks with the softmax weights.

Ranks are robust.

A channel with huge raw values cannot drown out the others.

This is the same reason that Elo and average rank are used in other benchmarks, see Row-level prediction.

A worked picture

Imagine a dataset of machine readings.

Normal rows follow a smooth pattern.

The selector builds fakes by shuffling and by uniform injection.

It finds that the imputation residual separates the fakes strongly, the embedding neighbours moderately, and the raw neighbours weakly.

The weights shift toward the residual and the embedding channel.

The isolation forest gets a smaller share.

On another dataset, the pattern may be reversed.

The same procedure adapts without any labels and without any hand tuning.

The procedure is fixed.

Only the weights it produces differ.

What made the difference

Labels are not needed to combine the signals

No labels are needed to choose how to combine the signals.

This is the main practical point.

In real use you often have no labelled anomalies.

The selector needs only normal rows.

The signals come from what the model has learned

The signals come from what the model has learned about the table, not from distances alone.

A distance in raw space treats every column as equally important and every scale as equal.

The imputation residual and the embedding use the structure that the model learned during pre-training, see Architecture and Cell encoding.

Fixing the integration, one bug at a time

The jump from the weaker chain to the final detector did not come from a clever new idea.

It came from fixing three integration bugs in a strong engine.

First among thirteen

The detector is first among thirteen methods by both average rank and AUROC.

Two metrics agreeing is more convincing than one.

A method could top the average rank by being consistently good, yet lose on AUROC to a method that is great on a few datasets.

Leading on both suggests the lead is broad.

How the same capability appears in use

In the product outputs, anomaly detection runs unsupervised, with no target.

It gives an anomaly score for every row, with labels optional.

The benchmark setting is a little stricter than daily use.

In daily use, you may have a few labelled anomalies.

The benchmark assumes none.

The detector is also a part of a single checkpoint with the other capabilities.

See Unified checkpoint.

The protocol

Semi-supervised, normal rows only

The model is trained on normal rows only and scored on a test set.

This is the setting in which the published methods are measured.

Third-party competitor values

Competitor figures are never re-run by me.

They are third-party published values cited to their primary source.

One sealed configuration

My figures come from one sealed configuration per benchmark, with no per-dataset tuning.

The selector adapts its weights per dataset, but the procedure is the same everywhere.

That is different from hand-picking a setting per dataset.

I want to be careful here, because the two can look alike.

A fixed procedure that computes weights from the training rows is automatic and label-free.

A human choosing a different setting for each dataset after seeing its test score is tuning.

I did the first, not the second.

No blending

Results measured under different datasets, metrics or masking schemes are never blended into one number.

Contamination

Each run is checked for contamination against the training corpus.

Independent passes

The earlier uniform fusion was kept in the history.

The final claim was defended with independent passes before it was sealed.

An independent pass is a separate run, designed to try to break the claim.

If the result survived, it stayed.

If not, it would not have been sealed.

Rules at a glance

Lessons from the build

Start with the better engine

My first chain was built on the weaker imputer.

It is tempting to polish the wrong foundation.

Switching to the right engine was worth more than any tweak.

Calibrate the output

If a head returns values on its own scale, compare after calibration.

A linear map per column is simple and enough.

Give an empty head something to learn

When the data has no gaps, make gaps.

Injecting 15 percent random missingness fixed the empty head.

Beware silent side effects

Import-time side effects do not crash.

They quietly change results.

Make loading a file side-effect free.

Build your own test for the combiner

Pseudo-anomalies give you a way to test channels without labels.

They are not perfect, but they are enough to tell a strong channel from a weak one.

Fuse ranks, not raw scores

Raw scores live on different scales. Ranks put them on equal footing.

What stayed open

The uniform fusion history

The earlier uniform fusion was kept in the history.

Weighting channels equally was my first approach, and it was worse than the selector.

I keep it as a record, not as a claim.

The pseudo-anomalies are a stand-in

Fake anomalies approximate real ones.

They cover two kinds of oddness: broken combinations and extreme values.

Real anomalies come in other forms, such as subtle drift or a new category that appears.

The selector can only measure what the fakes can mimic.

Normal rows must be normal

The model is trained on normal rows only.

If the training rows contain hidden anomalies, the idea of normal becomes blurred.

Clean training data helps.

One setting

The benchmark covers the semi-supervised setting.

It does not claim anything about fully unsupervised or fully supervised detection.

See Limitations for the general boundaries of the model.

Narrow wins are still wins

First on two metrics among thirteen is a strong statement, but the margin on any one dataset can be small.

The claim is about the overall picture.

What this means for a user

If you have a table of normal operations and you want to flag rows that do not fit, you can use the detector without labels.

Give it the normal rows, and it ranks new rows.

Check the top of the ranking by hand at first.

An anomaly score tells you what is unusual.

It does not tell you what is wrong.

A rare but legitimate event can score high.

If you do have a few labelled anomalies, use them to confirm that the ranking agrees with your expert's sense of what matters.

Frequently asked questions

What is semi-supervised anomaly detection?

The model is trained only on normal rows and never sees a labelled anomaly.

At test time it ranks rows by how strange they are.

This is the setting in which the published methods in the benchmark are measured.

What does the detector use to score a row?

It uses three model-based channels: the residual of the imputation head, nearest neighbours in the model's embedding space, and nearest neighbours in the raw space.

An isolation forest joins as an auxiliary channel.

The scores are converted to ranks and fused with weights.

How are the channels weighted without labels?

A pseudo-anomaly selector builds fake anomalies from the training rows, by shuffling features and by injecting uniform values.

It measures each channel on those fakes and turns the results into weights with a softmax.

No real anomaly labels are used.

Why inject 15 percent missingness before fitting?

If the data has no missing values, the imputation head has nothing to learn from and comes out empty.

Hiding 15 percent of the cells at random gives it a task, and matches how the model was pre-trained with masked cells.

Why did the first chain fail?

It used the weaker imputer.

I switched to the imputation engine and fixed its integration bugs one at a time: the output scale, the empty head and import-time side effects.

Why use ranks for fusion?

Each channel produces scores on a different scale.

Combining raw scores lets the channel with the biggest numbers dominate.

Ranks put every channel on the same footing, so the weights alone decide the influence.

How many methods were compared?

Thirteen in all, including ten classical detectors and published model pools.

The detector is first among them by both average rank and AUROC.

The competitor figures are third-party published values.

Does the benchmark use labels at all?

Labels are used only to score the test set, as in the published protocol.

They are not used to train the model or to choose how channels are combined.

The selector relies on pseudo-anomalies built from normal rows.

Was the final claim checked?

Yes.

The earlier uniform fusion was kept in the history, and the final claim was defended with independent passes before it was sealed.

Each run was also checked for contamination against the training corpus.

Is an isolation forest alone enough?

It is a good detector, and I include it as an auxiliary channel.

But different anomalies look different, and a single signal misses some of them.

Combining model-based channels with an isolation forest, under a label-free selector, gave the strongest result.

Summary

The anomaly detection benchmark ranks rows that do not fit the normal distribution.

The model is trained on normal rows only and scored on a test set, the semi-supervised ADBench setting, against ten classical detectors and published model pools.

My first chain used the weaker imputer.

After switching engines I fixed three bugs: the output scale with a per-column linear calibration, an empty head with 15 percent injected missingness, and import-time side effects.

The final detector has three model-based channels and an isolation forest as an auxiliary channel.

A pseudo-anomaly selector, built from shuffled features and uniform values, weights the channels with a softmax, and the ranks are fused.

The detector is first among thirteen methods by both average rank and AUROC.

What stayed open: the earlier uniform fusion is kept in the history, and the pseudo-anomalies can only mimic some kinds of oddness.

Read next: Missing-value imputation, Row-level prediction, Probabilistic regression, and Benchmarks.