Schema evolution

Overview

Schema evolution is the ability of a data system to change the structure of its data over time, such as adding, renaming, dropping or widening columns, while old data stays readable and existing consumers keep working. A system with good schema evolution does not force a full rewrite of stored files for every change, and it tells producers and readers clearly which changes are safe and which are breaking.

Data sources change constantly. A vendor adds a field, an application renames a column, a counter outgrows a 32-bit integer. Without a plan, each change breaks a dashboard or a pipeline. Table formats such as Apache Iceberg and Delta Lake, and message systems with a schema registry, give explicit rules for evolution. The same issue appears in machine learning: models that learn from raw tables, such as TabPFN, must cope with tables whose columns vary from one dataset to the next.

What it is

A schema describes the shape of data: the names of fields, their types, whether they may be empty, and sometimes constraints and defaults. Schema evolution is the discipline and the set of technical features that allow this description to change while the data lives on.

Think of a printed form used by a clinic for thirty years. At some point the clinic wants to add a line for an email address. Old filled-in forms do not have that line. A good filing system lets staff read old forms (the email field is simply empty) and new forms side by side. Schema evolution is that filing system for data.

Two things evolve

• The logical schema: what readers and writers agree the fields are.
• The physical files: how bytes are laid out in storage, such as Parquet or Avro files.

A good design keeps these separate. The logical schema can change by updating metadata only, and old files are interpreted through the new schema when read.

Compatibility directions

Which direction matters depends on who upgrades first. If consumers upgrade before producers, backward compatibility is the key. If producers upgrade first, forward compatibility matters.

Kinds of change

Where it came from

Early relational databases treated the schema as fixed. Changing it meant an ALTER TABLE statement that could lock the table and rewrite every row. Over time, engines added faster paths: adding a nullable column became a metadata-only operation in many systems, while other changes remained expensive.

From databases to files and streams

When organizations began storing data as files in object storage, the schema lived in each file or in a separate catalog. Formats such as Avro carried the writer's schema with the data and supported rules for resolving differences between writer and reader schemas. Parquet stored a schema in each file footer, and engines had to reconcile files written at different times. Early data lake users learned the hard way that mixing files with different schemas could produce silent errors or failed queries.

In event streaming, producers and consumers evolve independently. A central registry of schemas, with compatibility checks at publish time, became the standard answer. This is the role of a schema registry alongside Apache Kafka.

Table formats

Modern open table formats took the next step. They keep schema information in metadata files that are versioned with the table. Iceberg assigns each column a unique id, so a rename changes only a name in metadata. Delta Lake records schema changes in its transaction log and supports options to merge new columns automatically on write. Both aim to make common changes cheap and safe. Details of these designs appear in the pages for Apache Iceberg, Iceberg cataloging and Delta Lake.

Governance and contracts

As data became a shared product, teams began to treat schemas as public interfaces. The idea of a data contract emerged: producers promise a shape and quality level, consumers rely on it, and automated checks flag violations. Detecting unplanned changes is the job of schema drift detection. Together, these practices make evolution a managed process instead of a surprise.

How it works

The mechanics vary by system, but the same few ideas repeat.

Step 1: Identify columns by something stable

If columns are identified only by name, a rename looks like a drop plus an add, and data in the old column appears lost. If columns are identified by position, any reorder corrupts reads. The robust approach is a stable column id that never changes and is never reused.

Analogy: a library tracks a book by its catalog number, not by its title or shelf position. The title can be corrected and the book can move to a new shelf, but the catalog number stays, so every loan record remains valid.

Step 2: Keep schema as versioned metadata

Each schema change creates a new schema version. Old versions stay in history. Every data file is tied to the schema it was written with, directly or through the snapshot it belongs to. Time travel queries can then read the table as it looked at an earlier moment, with the schema of that moment.

Step 3: Resolve differences at read time

When a reader requests columns under the current schema, the engine maps each file to that schema:

1. For every column in the current schema, find the matching column in the file by id or name.
2. If found and types match, read it.
3. If found with a narrower type, promote the values (for example int to long).
4. If not found, fill with the default or null.
5. Ignore columns in the file that are not in the current schema.

This is why adding a column does not require touching old files. The reader supplies the missing values itself.

Step 4: Enforce on write

Writers are checked against the table schema. There are two common modes:

• Enforcement: reject writes whose schema does not match. This protects consumers from accidental changes.
• Evolution on write: accept new columns and extend the table schema automatically, when explicitly enabled.

Many teams keep enforcement on by default and enable evolution only for controlled jobs, such as an ingestion layer where new source fields are expected.

Step 5: Check compatibility before publishing

In streaming systems, the registry compares a new schema with earlier versions under a chosen policy, for example backward, forward or full, and refuses incompatible ones. This moves the failure from runtime in a consumer to publish time in a producer, which is far cheaper.

Step 6: Migrate when a change is breaking

Some changes cannot be made safely in place. The standard recipe is expand and contract:

1. Expand: add the new column or structure next to the old one.
2. Dual write: producers fill both.
3. Migrate: backfill history and move readers to the new field.
4. Contract: stop writing the old field, then remove it after a grace period.

Analogy: a city moves a bus route to a new street. It runs both routes for a few weeks, tells riders, and only then retires the old stop.

Where changes get handled in a pipeline

This layering follows the medallion architecture pattern. Letting the early layers absorb change keeps the later layers stable.

Key concepts and terms

A worked example

Evolving a table with SQL

Start with a table of orders:

CREATE TABLE orders (
  order_id   BIGINT,
  customer   STRING,
  amount     INT,
  created_at TIMESTAMP
);

Over a few months, four requirements arrive.

-- 1. Add an optional column. Metadata only, old rows read as NULL.
ALTER TABLE orders ADD COLUMN currency STRING;

-- 2. Widen a type. Safe because every INT fits in a BIGINT.
ALTER TABLE orders ALTER COLUMN amount TYPE BIGINT;

-- 3. Rename a column. Safe in systems that track columns by id.
ALTER TABLE orders RENAME COLUMN customer TO customer_name;

-- 4. Drop a column that no reader uses any more.
ALTER TABLE orders DROP COLUMN legacy_flag;

Statements 1 and 3 usually touch only metadata in table formats that use column ids. Statement 2 is also metadata only when the format supports promotion. Statement 4 is metadata only too, but readers that still select the column will fail, so check data lineage first.

What a reader sees

Rows 1 and 2 were written before currency existed. The reader fills the gap with NULL. A default can be applied at read time or in a downstream view.

Automatic merge on write in Spark

# New source data has an extra column "channel".
new_df = spark.read.json("landing/orders_2025_02/")

(new_df.write
   .format("delta")
   .mode("append")
   .option("mergeSchema", "true")   # allow the table to gain the new column
   .save("/lake/silver/orders"))

Without the option, the write would fail with a schema mismatch. That failure is a feature: it forces a decision about whether the new column is wanted. See Apache Spark and PySpark for the engine side.

A compatibility check for an event schema

# Registry policy for the "orders-value" subject
compatibility: BACKWARD
rules:
  - add_field_with_default: allowed
  - remove_field: allowed        # backward: new readers ignore it
  - add_field_without_default: rejected
  - change_type_string_to_int: rejected

A producer that tries to publish a schema with a new required field and no default is refused at publish time.

Tiny compatibility checker

A simplified check, to show the logic:

def backward_compatible(old: dict, new: dict) -> list[str]:
    """Return problems that stop new readers reading old data."""
    problems = []
    for name, spec in new.items():
        if name not in old and spec.get("default") is None and not spec.get("nullable"):
            problems.append(f"new required field without default: {name}")
        if name in old and old[name]["type"] != spec["type"]:
            if (old[name]["type"], spec["type"]) not in {("int", "long"), ("float", "double")}:
                problems.append(f"unsafe type change: {name}")
    return problems

old = {"id": {"type": "long"}, "amount": {"type": "int"}}
new = {"id": {"type": "long"}, "amount": {"type": "long"},
       "channel": {"type": "string", "nullable": True}}
print(backward_compatible(old, new))   # []

Real systems also handle nested fields, enums, unions and renames through aliases, but the core idea is the same.

Strengths and limitations

Strengths

• Changes are cheap: most are metadata-only and need no rewrite of large files.
• Old data stays usable: history is not thrown away when the structure moves on.
• Safer releases: compatibility checks catch breaking changes before they reach consumers.
• Independent teams: producers and consumers can upgrade on separate schedules.
• Auditable: versioned schemas show exactly what changed and when.
• Reproducible: time travel returns the table in its older shape.

Limitations

• Not every change is safe. Renames in name-based systems, type changes across families and new required fields all need migration work.
• Semantics are invisible to the system. A column can keep its name and type while its meaning changes, for example a currency switch from euros to cents. No compatibility check catches that.
• Defaults can mislead. A null or default for old rows may be interpreted as real data downstream.
• Engines differ. A feature supported in one engine may be missing or limited in another reading the same files.
• Nested and complex types have more edge cases than flat columns.
• Automatic merge on write can let junk fields creep into a table if unchecked.
• Partitioning and clustering columns are harder to change than ordinary ones.
• Documentation drifts. A schema in code and a description in a catalog can disagree.

How it compares with alternatives

The systems named in this article differ in details. Iceberg's column ids make renames safe. Delta Lake offers merge options on write. In relational databases such as PostgreSQL, many ALTER TABLE operations are fast but some take locks. Tools such as dbt can manage downstream models and surface breaking changes through tests and contracts.

Role in tabular foundation models

Tabular foundation models learn from raw tables. Schema evolution matters to them in two ways: how they cope with tables of different shapes, and how the data pipeline around them handles change.

Each table has its own schema

A text model reads one kind of input: a sequence of tokens. A tabular model must read tables with a different number of columns, different types and different meanings every time. Models such as TabPFN and TabICL are pre-trained on a large number of synthetic tables whose structure varies on purpose. The model learns to treat the schema as an input, not a constant. In effect, pre-training is a rehearsal for schema variation.

Analogy: a translator who has only ever worked with one client's forms is helpless when a new form arrives. One who has handled thousands of forms of varied layout can find the relevant boxes in a new one. Pre-training on diverse synthetic tables builds that second kind of skill.

Row and column structure

Rows are treated as exchangeable observations, so the order of rows does not matter. Columns are also treated as a set whose order should not change the answer, because real tables get columns reordered all the time. Architectures that use attention across rows and across columns, such as axial attention, are naturally tolerant of reordering. This is the model-side counterpart of name-based or id-based column tracking. Details are in /model/architecture.

No column names

In practice, columns often arrive with meaningless names such as "f1" and "f2", or with names that changed between exports. A model that relied on names would be fragile. Foundation models trained on synthetic tables learn to infer a column's role from its values and its relationships with other columns. When meaningful names are present, text-aware models such as TabSTAR can use them as extra information. The robust design treats names as helpful but optional, which is exactly how a good table format treats names: as labels, with a stable id underneath.

Mixed types and type changes

An integer column that becomes a float after widening, a category that gains new levels, or a date stored first as text and later as a timestamp: these are routine schema changes. A model with a per-cell encoder for numbers, categories and text handles a changed type by changing the encoder path, not the model. Consistent type metadata from a data catalog and from type promotion rules makes this mapping reliable. See also tokenizer training for how cell encodings are fitted and frozen.

Missing values from added columns

When a column is added, every historical row has a missing value for it. This is the classic source of structured missingness: a block of rows that is empty in one column for a reason that is not random. A tabular model needs a clear missing-value indicator to tell "not recorded yet" from "zero". Self-supervised training with hidden cells, as in masked cell modeling, prepares a model for this. It is also the bridge to imputation: imputation networks can fill the historical gap for a new column using the relationships among older columns, with care taken to flag the imputed values.

In-context learning on a changed schema

In in-context learning, the model is given labeled rows as context and predicts for new rows without weight updates. After a schema change, no retraining is needed: the new table, with its new columns, is simply supplied as context. This is a practical advantage over classical models, where a new feature means a new training run and a changed feature pipeline. It does not remove the need for sound upstream checks, since a silent semantic change still produces a confident but wrong prediction.

Continual learning

When a model must keep learning as data evolves, the main risk is forgetting earlier knowledge. Methods such as elastic weight consolidation, synaptic intelligence, learning without forgetting and experience replay address this. Schema change adds a second risk: the input mapping itself can shift. Stable column ids, frozen encoders and versioned schemas keep the mapping steady, so that continual learning only has to deal with distribution change.

Anomaly detection and drift

Schema drift is itself a signal. A column that suddenly becomes entirely null, a numeric field that now holds strings, or a category with a flood of new values all point to an upstream change. Anomaly detection on column-level statistics, together with schema drift detection, turns these events into alerts before they reach training or scoring.

Sector and semantic classification

When a table is assigned to a domain or a semantic type, the model relies on stable features. If a source renames a column or changes a unit, the classification can shift for reasons that have nothing to do with the real domain. Versioned schemas and lineage records let teams explain such shifts. Heads such as semantic sector heads are most reliable when the incoming schema is known and tracked.

Summary of the mapping

Measured behavior of such models across varied tables is discussed in /model/benchmarks, and the wider research background in /model/research.

Practical guidance and best practices

Design for change from the start

• Choose a format and catalog that track columns by id.
• Give every column a stable, never-reused identifier.
• Prefer nullable columns with sensible defaults when adding fields.
• Avoid encoding meaning in column position.
• Use explicit types, including timestamp and decimal, instead of strings for everything.
• Keep a description for each column, including units and allowed values.

Set compatibility rules by boundary

A safe change process

1. Propose the change with a short description and the affected consumers, found through data lineage.
2. Classify it: additive, widening, renaming, narrowing or semantic.
3. Run the compatibility check in continuous integration.
4. Apply the change in a test environment and run downstream tests.
5. Roll out with expand and contract if the change is breaking.
6. Update the data catalog entry and notify owners.
7. Remove old structures only after a stated grace period.

Testing

• Add tests that read old files with the new schema and assert on values.
• Add tests that old consumers still read new data, if forward compatibility is promised.
• Keep sample files from each historical schema version in a test folder.
• Fail the build on unplanned changes in the schema of important tables.

Monitoring

• Track column count, null rates, distinct counts and type per column over time.
• Alert on sudden changes, even if the schema technically did not change.
• Log every schema change with who, when and why.

For machine learning pipelines

• Version the feature schema alongside the model.
• Freeze scalers, bin edges and category lists at training time.
• Reject or flag scoring inputs whose columns do not match what the model expects.
• Record whether a column was imputed, and keep that flag as a feature or in metadata.
• Retest accuracy after any semantic change, not only after structural ones.

Handling nested and complex types

Flat columns are the easy case. Structs, arrays and maps add questions.

• Adding a field inside a struct is usually safe if it is optional.
• Reordering struct fields is safe only in systems that match by name or id.
• Changing the element type of an array follows the same promotion rules as a plain column.
• Map keys are rarely changed, because they define lookup behavior.
• Deeply nested evolution is where engines differ most, so test on the exact stack you run.

Handling enums and categories

A category column that gains a new level is a quiet schema change. Readers that map levels to codes may fail on the unknown value. Good practice is to include an explicit "other" or "unknown" path in consumers, and to announce new levels through the catalog. For models, a category list is frozen at training time, so a new level needs an unknown bucket or a retraining decision.

Handling units and meaning

Because no tool detects a change in meaning, add lightweight guards:

1. Record the unit and the allowed range in the column description.
2. Add range and distribution checks that alert when values shift sharply.
3. Version the definition text, not only the type.
4. Require a review for any change to a column used in reports or models.

Rollback plan

Every schema change should have an undo path. In versioned table formats, rollback can be as simple as restoring an earlier table version. For event streams, rollback means publishing a compatible schema again, since old messages stay in the log. Write the rollback step into the change ticket before the change goes out.

A small change log template

Common mistakes

• Treating a rename as harmless in a system that identifies columns by name.
• Adding a required column without a default and breaking every old writer.
• Narrowing a type, such as long to int, and losing values silently.
• Changing the meaning of a column while keeping its name and type.
• Turning on automatic schema merge everywhere, so that any malformed source adds columns.
• Skipping the grace period when removing a column.
• Forgetting partition or sort columns, which are costlier to change.
• Relying on column position in downstream code.
• Letting the catalog description go stale after a change.
• Testing only with the newest data, not with files from older versions.
• Assuming all engines support the same evolution features on the same table.
• Replacing nulls from a new column with zeros, and so inventing history.
• Ignoring model impact: a schema change that is harmless to dashboards can still shift model inputs.

Frequently asked questions

What is schema evolution?

Schema evolution is the ability to change the structure of a dataset, such as its columns and types, over time while old data remains readable and existing readers keep working. Good implementations make common changes cheap, usually by updating metadata instead of rewriting files, and they provide rules to separate safe changes from breaking ones.

What is the difference between schema evolution and schema drift?

Schema evolution is a planned, managed change with rules and checks. Schema drift is an unplanned divergence, for example a source that starts sending a new field or a different type without notice. Drift detection finds the unplanned changes, and evolution rules decide how to absorb the ones that are wanted.

Which schema changes are usually safe?

Adding an optional column, adding a field with a default, and widening a numeric type, such as integer to long, are usually safe. Dropping a column or renaming one is safe only if no reader depends on it, or if the system tracks columns by stable id. Narrowing types and adding required fields without defaults are typically breaking.

What does backward compatible mean?

A change is backward compatible when new readers using the new schema can still read data written under the old schema. Adding an optional field is the standard example, because old data simply lacks the field and the reader fills in a default or null. Forward compatibility is the mirror case, where old readers can read new data.

How do Iceberg and Delta Lake handle schema changes?

Both keep schema information in versioned metadata. Iceberg assigns each column an id so renames and reorders do not break reads. Delta Lake records schema changes in its transaction log and offers a write option to merge new columns automatically. In both, many changes are metadata-only and do not rewrite existing data files.

How do I rename a column without breaking consumers?

If the table format tracks columns by id, rename in place and update documentation. Otherwise use expand and contract: add the new column, write to both, move consumers over, then retire the old one after a grace period. A temporary view that exposes the old name over the new column can also ease the transition.

Why use a schema registry for events?

A registry stores every version of an event schema and refuses new versions that break the chosen compatibility policy. This moves failures to publish time, so a bad change is caught in the producer's build instead of crashing consumers in production. It also gives every consumer one place to look up the exact structure of a message.

How do tabular foundation models cope with changing schemas?

They are pre-trained on many tables of varied shape, so the number, type and order of columns is an input, not a fixed part of the model. Because they learn in context from the rows you supply, a table with new columns can be used right away without retraining. They still depend on correct upstream checks, since a silent change in meaning is not visible to the model.

What happens to old rows when a column is added?

Old rows have no value for the new column, so readers return null or a configured default. This block of missing values is not random, and it should be flagged clearly. Downstream users and models should treat it as not recorded, not as zero, and imputation, if used, should be marked as such.

Can schema evolution fully protect against breaking changes?

No. It protects against structural breaks, such as missing columns and incompatible types. It cannot see a change in meaning, such as a unit switch or a new definition of a status code. Data contracts, documentation in a catalog, distribution monitoring and communication between teams are needed to catch those.

Glossary

• Additive change: a change that only adds structure, such as a new optional column.
• Avro: a row-oriented data format that carries schemas and defines resolution rules.
• Backward compatibility: new readers can read old data.
• Breaking change: a change that makes some readers or writers fail.
• Column id: a stable identifier for a column, independent of name and position.
• Compatibility policy: the rule a registry enforces, such as backward or full.
• Data contract: an agreement about the shape and quality of data between producer and consumer.
• Default value: the value used when a field is absent.
• Expand and contract: a migration pattern that adds the new form, runs both, then removes the old.
• Forward compatibility: old readers can read new data.
• Metadata-only change: a change that does not rewrite data files.
• Nullable: allowed to have no value.
• Parquet: a columnar file format that stores a schema in each file.
• Schema drift: unplanned change in structure or type.
• Schema enforcement: rejecting writes that do not match the table schema.
• Schema merge: automatically adding new columns on write.
• Schema registry: a service that stores versioned schemas and checks compatibility.
• Time travel: reading a table as it was at an earlier version.
• Type promotion: safe widening of a data type.

Summary

Schema evolution lets a dataset change its structure over time without breaking old data or existing consumers. It rests on stable column ids, versioned metadata, read-time resolution of differences, write-time enforcement and compatibility checks, plus migration patterns such as expand and contract for breaking changes. Table formats like Iceberg and Delta Lake, registries for events, and tools for contracts, lineage and drift detection make it practical. It cannot detect changes in meaning, so documentation and monitoring remain essential. For tabular foundation models, the same thinking applies: they treat each table's schema as an input, tolerate missing and reordered columns, learn from rows in context, and rely on frozen, versioned encodings for stability. Related reading includes Apache Iceberg, Delta Lake, Schema Registry, Data catalog and /model/architecture.

Related

It belongs to Table formats and lakehouse in my stack, under Data engineering and lakehouse. Others in the same group: