Metadata independence benchmark: predicting with no column names

The metadata independence benchmark asks one blunt question: does the model still work when the headers of a table are gone? For the tabular foundation model described here, the answer is yes. Its accuracy does not fall when column names are removed or replaced with random strings, and it was the only model in my comparison that behaved this way.

This page explains what the benchmark measures, why most models fail it, how I built the capability, how I kept the measurement honest, what made the difference, and what stayed open. There are no scores on this page on purpose. The model card carries them together with their protocols. For the full list of boards, see Benchmarks. For the design idea behind this result, see Column name independence.

The short version

• The benchmark uses twenty numerical classification datasets under three naming conditions: full names, random strings, and names removed.
• Models that read headers lose roughly seven AUC points when headers vanish.
• The model never uses a column's identity. Column embeddings are always zeroed, so it sees values and the relations between columns only.
• Rivals ran on the same folds with five seeds, 300 measurements in all, while the model used one fixed configuration with no source selection.
• Wins are concentrated on datasets where meaning matters. On some pure numeric classics a stronger prior-based rival still leads, usually by a few thousandths.

What this benchmark measures

Every table has two kinds of information. The first is the data itself: the numbers, categories and dates in the cells. The second is the metadata: the column names, the data dictionary, the schema notes that explain what each column is. Most machine learning treats the second kind as free help. If a column is called age, a model that has read a lot of text about ages can use that. The benchmark checks how much a model leans on that help by taking it away.

The three naming conditions

Each dataset is run three times, with only the headers changed.

The cell values never change. The same rows are used, the same labels, the same train and test split. Only the header row differs. That is what makes the comparison clean: any change in accuracy between conditions can be traced to the header and to nothing else.

The metric of record

The metric of record is accuracy with no names, together with the drop from the full-names condition. Both numbers matter. Accuracy with no names tells you how good the model is in the harsh case. The drop tells you how much the model was relying on names to begin with. A model can have a high no-names accuracy simply because it is strong everywhere, and still depend on headers. A model that depends on nothing should show a drop close to zero.

For reference, models that read headers lose roughly seven AUC points when headers vanish. That is the size of the gap the benchmark is built to expose.

Why numerical datasets

The twenty datasets are numerical classification datasets. This choice is deliberate. When a column holds free text, the text itself carries meaning, and removing the header does not remove that meaning. With purely numerical columns, the header is often the only place where the meaning lives. A column of numbers between 20 and 80 could be an age, a temperature or a score. The header says which. Take it away and a model must work out the role from the numbers and from how they move with the other columns.

Why this matters in the real world

Think of a spreadsheet you inherited from a colleague who left. The columns are called val_B, metric_14, field3. Nobody remembers what they mean. Or think of data exported from an old system that stores short codes instead of names. Or data that has been stripped of names on purpose, for privacy, before it was shared.

In all of these cases, a model that needs good headers is stuck until a person writes a data dictionary. That is weeks of work in a large organization. A model that does not need headers is useful on day one.

There is a second, quieter reason. Headers can be wrong. A column called value that is really a date. A column called price that now stores a count after a migration. A model that trusts names will trust these mistakes. A model that reads values will not.

• Legacy extracts. Columns named by a system that no one remembers.
• Privacy-constrained data. Names removed before the table is shared.
• Merged sources. The same quantity under different names in different tables.
• Machine-generated exports. Columns named by position, such as c1, c2, c3.
• Misleading headers. Names that no longer describe the content.

How most models use names

To see why the benchmark is hard, it helps to see how names get in. There are three common routes.

Route one: names as text

Some language-model-based approaches turn a row into a sentence: "age is 54, income is 71,000, tenure is 3 years". The header becomes part of the text. The model has learned from billions of words what "age" and "income" mean, and it uses that. Remove the header and the sentence loses its subject. The model has to guess from the numbers alone, and it was never trained for that.

Route two: names as learned embeddings

Other models learn a vector for each column name or each column position during training. This vector tells the network what kind of column it is looking at. When a new table arrives with unfamiliar names, the vector is weak or wrong, and the network is working with a blurred signal.

Route three: names as human preprocessing

Even where the model itself ignores names, the pipeline around it often does not. A person reads the headers, decides which columns are categories, which are dates, which should be dropped, and encodes them by hand. This is not in the model, but it is still a dependency on metadata. Without headers, that human step stalls.

What the benchmark exposes

A model that falls on this benchmark is telling you which route it uses. A drop of roughly seven AUC points is the typical price of route one or two when names disappear. The model on this site was designed to use none of the three.

How I built it

The design choice came first. The rest was measurement discipline.

The design choice: column embeddings are always zeroed

In the encoder, every cell becomes a vector that combines its value, its type, and its position. In many models there would also be a part that identifies the column. In this model that part is always zeroed. The model never gets a column's identity as an input. It gets the values in the column and the way those values relate to the values in other columns. See Architecture and Cell encoding for the details.

An everyday analogy: imagine a detective handed a spreadsheet with all the headers torn off. A good detective can still tell which column is a phone number, which is a date of birth, and which is a price, from the shape of the values and from how the columns move together. Ages are small positive numbers that rise together with years of employment. Prices are skewed with a long tail. Dates sort into runs. The model is trained to be that detective.

Why zeroing has to happen in training too

It would not be enough to hide names only at test time. If the model learned during training that a column called age always behaves a certain way, it would build a shortcut and stop looking at the values. That shortcut breaks the moment a table arrives with a column called c3.

So the zeroing is there from the start of pre-training and stays there. The model is forced to learn what a phone number, a date or a price looks like from the values and from relations between columns. This is why the result is a property of the model and not a trick applied afterwards. See Pre-training for how the self-supervised objective works and why closing this shortcut matters.

What this buys

• A column called c3 that holds phone numbers is understood as phone numbers.
• Renaming every column to random strings does not change what the model sees.
• Removing the names entirely does not change what the model sees.
• Misleading names, such as a column called value that is really a date, do not mislead it.

Notice the logic here. Since the model never receives the header in any form, the three naming conditions are, from the model's point of view, the same input. There is nothing to lose because there was nothing in use. This is different from a model that has learned to cope with missing names. Coping means a smaller loss. Not using means no loss.

When a name is the only clue, the model has nothing extra to lean on. It reads the data, and unusual or mixed formats can still lower its confidence. I come back to this in the section on what stayed open.

Measurement discipline

A claim that nothing changes is easy to get wrong in both directions. If the harness is buggy, a model can look name-independent when it is not, or the other way round. Most of the work on this benchmark went into making the measurement trustworthy.

Two measurement bugs

I fixed two measurement bugs before trusting any number. I will not pretend the details were glamorous. They were the ordinary kind that silently change a comparison: the kind where one condition is handled slightly differently from another. After the fixes, all three conditions went through exactly the same code path, with only the header row changing.

The lesson is general. In several of my benchmarks the largest gains came from fixing the harness, not the model. This is the first rule of the whole timeline: check the ruler before you trust the measurement. See Evaluation for how scores are tied to splits and snapshots.

A symmetric protocol

The second step was to make the protocol symmetric, so that no side had an advantage.

• Rivals run on the same folds as the model.
• Every method runs with five seeds.
• That makes 300 measurements in all, which is what twenty datasets, three naming conditions and five seeds multiply to.
• The model uses one fixed configuration. There is no source selection and no per-dataset tuning.
• A win that sat inside its own seed noise was dropped.

The last point deserves a moment. A seed is the starting number for a random process, which decides things like which rows land in which fold or how a model initializes. If you change only the seed and your result moves by more than the gap between you and a rival, then the gap is not evidence. I dropped such wins instead of reporting them.

What "no source selection" means

When a method has several variants, it is tempting to run them all and report the best one per dataset. That is source selection, and it quietly turns many tries into one headline. Here, the model has one sealed configuration, the same for every dataset and every naming condition. Rivals are allowed their own published configurations. The imbalance, if any, favours the rivals.

The protocol behind the benchmark

The benchmark follows the protocol of the first model's paper, arXiv:2605.06290, which defines the Data Language Model category. The paper is described on the Research page. Competitor figures are third-party published values cited to their primary source and are not re-run by me, in line with the standing rules for every board. See Benchmarks.

A worked picture

Here is what the same table looks like in the three conditions. The numbers are invented for illustration. They are not from any dataset in the benchmark.

Full names                      Random strings               Names removed
age   income  tenure  label     x7q2   k19    p_03   y        (blank) (blank) (blank) y
54    71000   3       1         54     71000  3      1        54      71000   3       1
31    42000   1       0         31     42000  1      0        31      42000   1       0
47    58000   9       1         47     58000  9      1        47      58000   9       1

For a model that reads headers, the left block is rich and the right block is poor. For the model on this site, the three blocks are the same input. The cell vectors are built from values, types and positions, and the column identity part is zero in all three.

What the model uses instead of names

Without names, the model leans on structure that is always present in a table.

Because the network alternates attention over rows and over columns, it can compare a column to its neighbours directly. That is how relations between columns reach the representation. See Architecture for the axial attention design and In-context prediction for how the labelled rows are used at prediction time.

What made the difference

The difference came from one decision and one habit.

1. The decision. Zero the column embeddings, always, in training and at use. This is what makes the three conditions identical to the model.
2. The habit. Make the protocol symmetric, fix the harness bugs first, and refuse wins that live inside seed noise.

It is the only model in the comparison whose accuracy does not fall when names are removed or replaced with noise. In practice, no renaming, mapping or data-dictionary work is needed before the model is useful. That is the practical meaning of the result: the first day of a project is not spent on headers.

Why a flat line is the right shape

A good result here looks boring. If you plot accuracy for the three conditions, you want three bars of the same height. The rivals show a staircase going down: full names, then random strings, then no names. The point of the benchmark is the shape of that staircase, and the model's shape is flat.

Why not just fine-tune on masked headers

A fair question is whether a name-reading model could be patched by training it on tables with headers hidden. That helps, and it narrows the gap, but it keeps the same architecture that has a path for names to enter. The model here has no such path. There is nothing to patch, and nothing for a later training run to quietly re-open.

What stayed open

I want to state the weak spots plainly.

• Wins are concentrated where meaning matters. The strongest gains are on datasets where the relations between columns carry the signal that a name would otherwise have supplied.
• Pure numeric classics. On datasets without semantics, a stronger prior-based rival still leads, usually by a few thousandths. These are tables where no name could have helped anyway, so the contest is about raw prediction strength.
• Names are sometimes the only clue. If two columns have identical value shapes and the only difference is the label, a name-blind model has less to work with. It may express that as lower confidence.
• Twenty datasets is a bounded sample. The result is a measurement on twenty numerical classification datasets, not a promise about every table in the world.

None of these takes away from the main finding. They define where it applies.

What the benchmark does not test

It does not test text columns, where the content itself carries meaning. It does not test regression, ranking or generation. It does not test tables where the header is the target of the task, for instance predicting a column's name. Those belong to other pages: Blind sector identification covers naming the industry of a table without names, and Row-level prediction covers the main classification and regression boards.

Standing rules for this board

The same rules apply to every entry on the timeline.

• Competitor figures are third-party published values cited to their primary source. I do not re-run them.
• My figures come from one sealed configuration per benchmark, with no per-dataset tuning.
• Results measured under different datasets, metrics or masking schemes are never blended into one number.
• Each run is checked for contamination against the training corpus, using an identifier blocklist and a distributional near-duplicate check.

Try the idea on your own data

You can test any model for name dependence with a small experiment. The code below is a sketch of the idea, not a specific tool.

# Sketch: measure name dependence for any tabular model.
import random, string

def random_names(columns):
    return ["".join(random.choices(string.ascii_lowercase + string.digits, k=5)) for _ in columns]

def run_condition(model, X_train, y_train, X_test, y_test, condition):
    if condition == "random":
        X_train.columns = X_test.columns = random_names(X_train.columns)
    elif condition == "removed":
        X_train.columns = X_test.columns = [""] * X_train.shape[1]
    model.fit(X_train, y_train)          # or in-context: pass labelled rows
    return accuracy(model.predict(X_test), y_test)

for seed in range(5):
    # same folds for every condition and every method
    for condition in ["full", "random", "removed"]:
        score = run_condition(model, *folds[seed], condition)

Three habits make the experiment honest. Use the same folds for every condition. Repeat with several seeds and look at the spread. Report the drop between conditions, not only the final number.

How this connects to the rest of the model

Name independence is not a stand-alone feature. It is the base for several other results.

• Blind sector identification. Naming the industry of a table from raw cell values only works if the model never needed the headers. See Blind sector identification.
• the model matching. The model matches columns across tables without seeing their names, and beats name-aware matchers. See Entity and schema matching.
• Imputation. Filling gaps does not depend on knowing what the column is called. See Missing-value imputation.
• Packaging. The merged checkpoint carries the same property into serving. See One checkpoint.

In the product

For a practical view of what this means for an integration, see Solutions, where the first question is "what is this data?", answered even when every header has been stripped. For the outputs that come back, see Capabilities.

The work in order

The page above describes the result. This section lists the steps in the order they happened, so the logic is easy to follow.

Why the order matters

• Choosing the datasets before running anything stops me from picking the ones that look good.
• Fixing the bugs before the comparison stops a harness error from becoming a headline.
• Setting the seeds before the comparison stops cherry-picking a lucky run.
• Dropping the noise wins after the comparison keeps the claim smaller than the data.

Common misreadings

Short results travel badly. These are the ways this one gets misread.

"The model ignores meaning"

It does not. It ignores the label of a column, not what the column means. The meaning is recovered from values and from relations. A column of birth years still looks like birth years.

"Names never help any model"

They help models that were built to use them.

"A flat line means the model is perfect"

A flat line means the header is not part of the input. The accuracy level itself is a separate question, answered by the other boards.

"Twenty datasets proves it for all tables"

It is a measurement on a bounded sample under a stated protocol. The claim is limited to what was measured.

"The model needs no human at all"

It needs no human to rename or document columns. A person still decides what to predict and whether to act on a result.

Questions to ask about any model

If you are choosing a tool for tables with poor headers, you can ask these of any vendor or paper, including this one.

• Does the input to the model include the column names in any form?
• What happens to accuracy when the names are random strings?
• What happens when the names are removed?
• Were the rivals run on the same folds, with several seeds?
• Was a single configuration used, or the best of many per dataset?
• Were wins that fall within seed noise reported or dropped?
• Is the comparison blended across different datasets or metrics?
• Was the evaluation checked for overlap with the training data?

A thought experiment

Imagine two analysts. The first is excellent but relies on the data dictionary. The second has no dictionary and has learned, over years, to read tables by their contents.

Hand both a clean table with good headers and the first analyst is quick. Hand both a table with headers stripped and the first analyst stops, while the second carries on at nearly the same speed.

The model is built to be the second analyst. That is also why the result is not a trick: it is the trained skill of reading contents.

Where the second analyst is slower

• When two columns hold values of the same shape and only a label separates them.
• When a format is unusual enough that its type is unclear.
• When the table is tiny and gives little context.

The analyst is honest about these cases by lowering confidence. So is the model.

Choosing a naming condition for your own tests

1. Run once with the headers as they are, to set a reference.
2. Run again with random strings, to confirm the model is unmoved by noise.
3. Run once more with no header at all, the harshest case.
4. Compare the three results using the same folds and the same seeds.
5. Read the held-out report for each run. See Evaluation for what that number claims.

If the three results match, you have confirmed on your own data what the benchmark shows on twenty datasets.

What to do if they do not match

• Check that the folds and seeds are identical across the three runs.
• Check that nothing else in your pipeline reads the header, such as a step that picks columns by name.
• Look at the weakest slice in the held-out report, which is where the model is least sure.

Reading the result next to the other boards

Each board answers a different question about the same model.

Fact sheet

The result in five lines

• Names removed: no fall in accuracy.
• Names scrambled: no fall in accuracy.
• Names misleading: no effect.
• Header readers: roughly seven AUC points lost.
• Pure numeric classics: a rival still leads, narrowly.

What the model reads instead

• Values.
• Types.
• Positions.
• Relations between columns.
• Labelled rows in context.

What the model does not read

• Column names.
• Data dictionaries.
• Table descriptions.
• The model notes.
• Any text about the table.

Two pipelines side by side

A header-dependent pipeline and a name-independent one differ in how many steps sit between the raw table and a prediction.

Header-dependent pipeline
  raw table
    -> read headers
    -> map headers to a data dictionary
    -> choose encoders by column meaning
    -> drop or rename columns by hand
    -> train or fit a model
    -> predict

Name-independent pipeline
  raw table
    -> model reads the cells
    -> predict

The second pipeline is shorter because the steps that depend on names are simply absent.

Do and do not

Do:

• Compare all three naming conditions on the same folds.
• Repeat with several seeds and read the spread.
• Report the drop as well as the final accuracy.
• Keep one configuration across conditions.

Do not:

• Pick the best variant per dataset and call it one result.
• Report a win that is smaller than the seed noise.
• Blend results from different datasets or metrics into one number.
• Treat a rival's published figure as something you re-ran.

Quick answers

• Does the model need headers? No.
• Does it need a data dictionary? No.
• Can it be misled by wrong names? No, it never sees them.
• Does it always win? No, see what stayed open.
• Is the result score-free here? Yes, scores sit in the model card with their protocols.
• Where is the design explained? In Architecture and Column name independence.

Terms used on this page

• Metadata. Information about the data rather than the data itself: column names, types, descriptions.
• Column embedding. A learned vector that tells a network which column a cell belongs to. In this model it is zeroed.
• Seed. The starting number of a random process, fixed so a run can be repeated.
• Fold. One split of a dataset into training and test parts.
• AUC. The area under the ROC curve, a score for how well a classifier ranks positives above negatives.
• Source selection. Choosing the best of several variants per dataset after seeing the results.
• Contamination. When test data has leaked into training data, which flatters the score.

Frequently asked questions

What is metadata independence in a tabular foundation model?

It means the model's output does not depend on column names or other descriptions of the table. It reads cell values and the relations between columns only. If you rename every column to random text, or remove the header row, the model sees the same input and gives the same quality of answer.

How was the benchmark set up?

I used twenty numerical classification datasets under three naming conditions: full names, random strings and names removed. The metric of record is accuracy with no names and the drop from full names. Rivals ran on the same folds with five seeds, giving 300 measurements, and the model used one fixed configuration.

Why do other models lose accuracy without names?

Many of them read the header as text, learn a vector for each column, or depend on a person to interpret the headers during preprocessing. When the header disappears, a piece of their input disappears too. Models that read headers lose roughly seven AUC points when headers vanish.

How does the model avoid using column names?

Its column embeddings are always zeroed, in training and at use. The encoder builds each cell vector from the value, the type and the position, and the part that would identify the column stays empty. The model therefore learns to recognise a phone number or a date from shape and context.

Is the model better than every rival on every dataset?

No. My wins are concentrated on datasets where meaning matters. On pure numeric classics without semantics, a stronger prior-based rival still leads, usually by a few thousandths. The claim is about stability across naming conditions, and about leading where the relations between columns carry the signal.

What did you do to make the comparison fair?

I fixed two measurement bugs first, then made the protocol symmetric: same folds, five seeds, one fixed configuration for the model and no source selection. A win that sat inside its own seed noise was dropped. Competitor figures are third-party published values, not re-runs.

Do I need a data dictionary before using the model?

No. In practice no renaming, mapping or data-dictionary work is needed before the model is useful. A column called c3 that holds phone numbers is understood as phone numbers. A data dictionary can still help people read the output, but the model does not require it.

Can misleading column names fool the model?

No, because the model never receives the names. A column called value that is really a date is judged on its values. This is a real advantage with old or migrated systems, where names drift away from content.

Where does name independence not help?

When the header is the only clue. If two columns look the same in value and differ only by label, a name-blind model has less to work with and may report lower confidence. Unusual or mixed formats can also lower confidence. See Limitations.

Does this benchmark cover text columns?

No. It uses numerical classification datasets, where the header is often the only carrier of meaning. Text carries its own meaning, so removing a header matters less there. Other boards cover different parts of the model.

Summary

The metadata independence benchmark removes or scrambles column names on twenty numerical classification datasets and measures what happens to accuracy. Models that read headers lose roughly seven AUC points. The model here does not, because its column embeddings are always zeroed and it works from values and the relations between columns.

The result rests on one design decision and careful measurement: two harness bugs fixed, rivals on the same folds with five seeds, 300 measurements, one fixed configuration, and no wins kept that sat inside seed noise. It stays honest about its limits, since on pure numeric classics a stronger prior-based rival still leads by a few thousandths. The practical meaning is simple: no renaming, mapping or data-dictionary work is needed before the model is useful. Read next: Column name independence, Benchmarks, Architecture and Limitations.