Delta Lake

Overview

Delta Lake is an open table format and storage layer that adds database-style reliability to files in a data lake. It stores data as Parquet files and adds a transaction log next to them. The log records every change, which gives tables ACID transactions, time travel, schema enforcement and safe concurrent writes. Without a layer like this, a folder of files is only a folder. With it, the folder behaves like a real table.

Delta Lake is most often used with Apache Spark and Databricks, and it is a core building block of the lakehouse idea. It is the usual storage choice for the bronze, silver and gold tiers of the medallion architecture. For tabular AI, it matters because reliable, versioned, typed tables are what models learn from, and time travel makes it possible to reproduce exactly which data a model saw.

What it is

Delta Lake is not a database server and not a new file format for data itself. It is a protocol and a set of libraries that organise ordinary Parquet files into a table. The key part is the Delta log, a folder of small JSON and checkpoint files that lists which data files belong to the table at each version.

A helpful comparison is a library with a catalogue. The books are the Parquet files. The catalogue says which books are on the shelf today, which were removed and which were added last Tuesday. Anyone can still walk into the stacks, but the catalogue is the source of truth. If two librarians update the catalogue at once, rules decide who goes first, so the catalogue never ends up half-written.

What Delta Lake provides

• ACID transactions. Writes either fully succeed or have no effect, and readers see a consistent snapshot.
• Scalable metadata. The log can describe tables with millions of files, and engines read it efficiently.
• Time travel. You can query an earlier version of a table by version number or timestamp.
• Schema enforcement. Writes that do not match the table schema are rejected.
• Schema evolution. Columns can be added in a controlled way.
• Upserts and deletes. MERGE, UPDATE and DELETE work on files that are otherwise immutable.
• Unified batch and streaming. The same table can be a sink and a source for streams.
• Audit history. Every operation is recorded, with who and what changed.

What it is built on

Delta Lake uses Parquet columnar files for data. Parquet is efficient for analytics because it stores values column by column, compresses well and lets engines skip unneeded columns. Delta adds the log, statistics and rules on top. Storage can be any object store or file system, such as Amazon S3.

Where it came from

Delta Lake grew out of a practical problem. As companies moved analytics onto cheap object storage, they found that data lakes were unreliable. A failed job could leave half-written files. Two jobs writing at the same time could corrupt a table. There was no clean way to update or delete a row, which became important with privacy rules. Readers sometimes saw partial results.

Engineers working on large Spark pipelines built a transaction log layer to fix this, and the project was later released as open source and moved under a neutral foundation. Around the same time, other open table formats such as Apache Iceberg and Apache Hudi addressed the same pain with different designs. The three are often compared, and engines increasingly read more than one.

Key ideas behind Delta come from database research that is much older: write-ahead logging, multi-version concurrency control and optimistic concurrency. Delta's contribution was to apply them to files on object storage, where there is no central database server to coordinate writers.

Over time the project added features such as change data feed, deletion vectors, liquid clustering and richer connectors for engines beyond Spark. Exact feature names and release timing vary by version, so check the documentation of the version you use.

How it works

The folder layout

A Delta table is a directory that looks roughly like this:

/tables/orders/
  _delta_log/
    00000000000000000000.json
    00000000000000000001.json
    00000000000000000010.checkpoint.parquet
  part-0000-....snappy.parquet
  part-0001-....snappy.parquet

The _delta_log folder holds numbered commit files. Each commit lists actions, such as "add this data file" or "remove that data file". The current state of the table is the result of replaying the commits in order.

Step 1: A write creates new files

When you append rows, the engine writes new Parquet files. Existing files are never changed. This is like adding new pages to a binder instead of erasing old ones.

Step 2: The commit is atomic

After the files are written, the engine tries to create the next numbered log file. This single step is the commit. Object stores offer an atomic way to create a file only if it does not exist, or a coordinator service provides the same guarantee. If the commit succeeds, the new files become part of the table at once. If the job fails before that point, the extra files are never referenced and are cleaned up later.

Step 3: Readers use snapshots

A reader first reads the log to learn which files make up the latest version, then reads only those files. Because files are immutable, a reader that started before a commit keeps a stable view. This is called snapshot isolation.

Step 4: Concurrent writers use optimistic concurrency

Two writers may both try to commit version 12. One wins. The other checks whether its changes conflict with what the winner did. If the two touched different files or different partitions, the loser can retry on top of the winner. If they conflict, it fails with a clear error. This is optimistic concurrency: assume no conflict, check at the end.

An everyday comparison is two people editing a shared recipe by email. If one changes the sugar and the other changes the oven time, the changes can be merged. If both change the sugar, someone must decide.

Step 5: Updates and deletes rewrite files

Because Parquet files are immutable, an UPDATE or DELETE finds the files that contain affected rows, writes new copies without or with changed rows, and commits "remove old file, add new file" in one transaction. Some versions can instead mark rows as deleted with a small side file, called a deletion vector, and rewrite later. This is cheaper for small changes.

Step 6: Checkpoints keep reads fast

Replaying thousands of small JSON files would be slow, so every so often Delta writes a checkpoint, a Parquet file that holds the full state at that version. Readers start from the latest checkpoint and replay only newer commits.

Step 7: Statistics enable data skipping

Delta records minimum and maximum values per column for each data file. A query with a filter such as order_date = '2026-01-05' can skip any file whose range does not include that date. Combined with partitioning and clustering, this cuts the amount of data read by a large factor.

Step 8: Time travel reads old versions

Since old files are kept until cleaned, the log can be replayed to any earlier version. That is how time travel works. It is useful for audits, debugging and reproducing model training sets.

Step 9: Maintenance keeps things tidy

• Compaction merges many small files into fewer larger ones.
• Vacuum deletes files that are no longer referenced and older than a retention period.
• Optimisation of layout orders or clusters data so related rows sit together.

Without maintenance, tables slowly become slow because of too many small files.

Key concepts and terms

A worked example

This example uses PySpark with fake paths. It creates a Delta table, updates it, travels back in time and runs a merge.

Create and write

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.appName("delta_demo")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog",
            "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    .getOrCreate()
)

path = "s3://example.internal/lake/silver/customers"

df = spark.createDataFrame(
    [(1, "Ada", "NL", 120.5), (2, "Ben", "DE", 80.0), (3, "Cleo", "FR", None)],
    ["customer_id", "name", "country", "balance"],
)

df.write.format("delta").mode("overwrite").save(path)

This is version 0 of the table. A folder _delta_log now holds 00000000000000000000.json.

Append and update

from delta.tables import DeltaTable

new_rows = spark.createDataFrame(
    [(4, "Dev", "ES", 15.0)],
    ["customer_id", "name", "country", "balance"],
)
new_rows.write.format("delta").mode("append").save(path)   # version 1

table = DeltaTable.forPath(spark, path)
table.update(
    condition="customer_id = 2",
    set={"balance": "balance + 10"},
)                                                           # version 2

Time travel

v0 = spark.read.format("delta").option("versionAsOf", 0).load(path)
v0.show()

Version 0 still shows three rows with the original balances, even though the table now has four rows and a changed balance. This is time travel at work, and it needs no extra copy of the data.

Merge (upsert)

updates = spark.createDataFrame(
    [(3, "Cleo", "FR", 55.0), (5, "Eli", "IT", 9.0)],
    ["customer_id", "name", "country", "balance"],
)

(
    table.alias("t")
    .merge(updates.alias("s"), "t.customer_id = s.customer_id")
    .whenMatchedUpdateAll()
    .whenNotMatchedInsertAll()
    .execute()
)

Customer 3 gets a balance, customer 5 is inserted. The whole merge is one transaction.

Inspect the history

DESCRIBE HISTORY delta.`s3://example.internal/lake/silver/customers`;

The result lists versions, timestamps, operations and parameters. This is the audit trail.

Schema enforcement in action

bad = spark.createDataFrame([(6, "Fay", "UK", "not a number")],
                            ["customer_id", "name", "country", "balance"])
bad.write.format("delta").mode("append").save(path)

This write fails, because balance is a number and the new data has a string. The table stays clean. To add a new column on purpose, you would use the schema evolution option for that write.

Maintenance

OPTIMIZE delta.`s3://example.internal/lake/silver/customers`;
VACUUM  delta.`s3://example.internal/lake/silver/customers` RETAIN 168 HOURS;

OPTIMIZE compacts small files. VACUUM removes files no longer needed after seven days, which also ends time travel for versions older than that.

A small numeric picture

Say a table has files A and B at version 5. A job deletes rows in file B. It writes file C with the surviving rows, then commits "remove B, add C" as version 6. Readers of version 5 still use A and B. Readers of version 6 use A and C. File B remains on disk until vacuum. Nothing was ever half-changed.

Strengths and limitations

Strengths

• Reliability. ACID commits remove the classic half-written table problem.
• Reproducibility. Time travel makes audits and experiments repeatable.
• Flexibility. One table serves batch jobs, streaming jobs and interactive SQL.
• Upserts on cheap storage. Merge and delete work on object stores.
• Performance features. Statistics, skipping, compaction and clustering reduce scan cost.
• Open format. Data stays in Parquet, and the protocol is public.
• Ecosystem. Strong support in Spark, and growing support in other engines.
• Governance hooks. History, constraints and change feeds support audit and quality work.

Limitations

• Small-file pressure. Frequent small writes need regular compaction.
• Write conflicts. Heavy concurrent writes to the same files can cause retries and failures.
• Not a low-latency OLTP store. Single-row lookups with millisecond targets are better served by a database.
• Retention trade-off. Longer time travel means more storage.
• Engine differences. Not every engine supports every feature, so check compatibility.
• Operational care. Vacuum with too short a retention can break long-running readers or time travel.
• Format competition. Teams must choose between Delta, Iceberg and Hudi, or run a bridge between them.
• Storage cost of rewrites. Updating a few rows may rewrite whole files if deletion vectors are not used.

How it compares with alternatives

Delta and Iceberg both solve the same core problem, and many teams pick based on the engine and platform they already use. Interoperability layers let one format be read as the other in some setups, but you should test the exact combination.

Role in tabular foundation models

Delta Lake does not train or run a model, but it is a very good home for the tables that tabular foundation models read and write. These models learn from rows and columns, handle mixed types and missing values, and can predict from a handful of examples through in-context learning. Their results are only as good as the data they are fed, and Delta Lake improves that data in specific ways.

Row and column structure

A Delta table has a defined schema and consistent rows. Each column has one type, and each row follows the same shape. That matches what a tabular model expects. Models such as TabPFN, TabICL and TabSTAR can take a slice of a Delta table, converted to a DataFrame, as context and query rows.

Mixed types and missing values

Delta stores numbers, strings, dates, booleans, decimals and nested types, and it keeps nulls as true nulls. This is helpful, because a tabular model can treat a null as missing, and can fill it in through imputation. See imputation networks. Schema enforcement also prevents a numeric column from silently becoming text, which is a common cause of silent model failure.

No column names

Some tabular models are designed to cope with tables whose columns have no meaningful names. In a Delta table, names are always present, and a model that can use them may benefit from extra signal. A model that ignores names can still work from the values. You can also drop names for a blind test, to check that a model is not relying on a naming accident.

Pre-training on synthetic or real tables

Tabular foundation models are pre-trained on large collections of synthetic tables and sometimes real ones. If real tables are used, they must be versioned and traceable. Delta Lake gives each snapshot a version and a history, so a training corpus can be pinned to exact versions. That makes it possible to say which data went into a model, which supports audits and licence checks. See the model research and model architecture pages for how such pre-training is described.

In-context learning

In-context learning needs a small, well-chosen sample of labelled rows. With Delta, you can sample efficiently using data skipping and partition filters, so only the needed files are read. You can also pin the sample to a table version, so the same context gives the same predictions tomorrow.

Continual learning

When a model is updated as new data arrives, you need to know what is new. Delta's change data feed lists inserted, updated and deleted rows between versions. A continual learning loop can read only the delta, update the model or its context, and record the version it has processed. This is much cheaper than rereading the whole table.

Imputation

Imputation writes predicted values back into a table. With Delta, the write is an atomic merge, and the original values stay available in earlier versions. This is good practice: never overwrite the source without being able to see what was imputed. A common pattern adds a flag column such as balance_imputed.

Anomaly detection

Anomaly scores can be stored as a column or a separate Delta table. Time travel lets you compare anomaly scores before and after a pipeline change. Constraints and expectations in Delta pipelines act as simple rules, while model-based anomaly detection catches subtler patterns.

Sector and semantic classification

Classifying companies or records into sectors, or tagging columns with semantic types, produces new label columns. Delta makes it simple to keep labels versioned, to compare two label sets, and to roll back if a classifier update made things worse. Benchmark discussions on the model benchmarks page show why repeatable evaluation sets matter, and Delta snapshots are a practical way to freeze them.

Where Delta sits in the pipeline

In a typical flow, raw files land in the bronze layer, cleaning produces silver tables, and business-ready tables go to the gold layer. All three can be Delta tables. A tabular foundation model reads silver or gold features, and writes predictions, scores and imputed values back as new Delta tables, with full history.

Practical guidance and best practices

Table design

• Choose partition columns with low cardinality, such as date, and avoid over-partitioning.
• For large tables, prefer clustering features over deep partition trees where available.
• Keep a clear primary key even though Delta does not enforce uniqueness by itself.
• Use constraints for rules like non-null and value ranges.
• Use decimal types for money, not floating point.
• Keep timestamps in UTC.

Writing

• Batch small writes into larger ones when you can.
• Use MERGE for upserts, with a join condition that matches a unique key.
• Make merges idempotent, so retries are safe.
• Limit the number of files touched per merge by filtering the target with a partition predicate.
• Use schema evolution only on purpose and review the changes.

Reading

• Filter on partition and clustered columns.
• Select only the columns you need.
• Use time travel for audits and reproducibility, not as a backup of everything.
• For streaming reads, set sensible trigger and rate options. See Spark Streaming.

Maintenance

• Schedule compaction on busy tables.
• Run vacuum with a retention period that matches your time travel and long-reader needs.
• Monitor file counts, average file size and table size.
• Watch for failing commits and repeated conflicts.
• Checkpoint streams carefully and never share a checkpoint between jobs.

Governance and security

• Control access with the catalog and storage policies.
• Mask or remove personal data where not needed.
• Delete personal data properly: after a delete, old versions may still hold the data until vacuum removes them.
• Register tables in a data catalog.
• Keep history for audit but limit who can read old versions.

For machine learning teams

• Pin training sets to a table version and record it with the model.
• Snapshot evaluation sets, so scores stay comparable over time.
• Store features with an as-of date to avoid leakage.
• Write predictions with the model version and run id.
• Use the change data feed for incremental feature updates.

Choosing partitions and clustering

Partitioning splits a table into folders by column value. It helps when queries filter on that column and each partition holds a healthy amount of data. A rough rule is that a partition should hold at least a few hundred megabytes. A daily partition on a table that gets ten megabytes a day is too fine.

Clustering features go a step further. They order the data inside files by chosen columns, so that min and max statistics become tight and data skipping works well. Pick columns that appear often in filters and joins. Revisit the choice when query patterns change, since a layout that was right last year can be wrong today.

Handling late and duplicate data

Event data often arrives late or twice. Two habits keep Delta tables correct:

• Use a stable event id and merge on it, so a repeated record updates instead of duplicating.
• Keep an ingestion timestamp next to the event timestamp, so you can find what arrived when.

Deduplicate in silver, not in bronze. Bronze should keep what was received so that you can replay.

Reading the change data feed

The change data feed lists what changed between two versions, with a change type per row such as insert, update before, update after and delete.

changes = (
    spark.read.format("delta")
    .option("readChangeFeed", "true")
    .option("startingVersion", 12)
    .load("s3://example.internal/lake/silver/customers")
)
changes.select("customer_id", "_change_type", "_commit_version").show()

A downstream job can use this to update a gold table, to refresh features, or to feed a model only the rows that changed. The feature must be enabled on the table first, and it adds a little storage.

Cloning and testing safely

A clone creates a new table that shares data files with the source at a chosen version. It is a cheap way to test a risky change.

CREATE TABLE sandbox.customers_test SHALLOW CLONE silver.customers;

You can run an experimental merge on the clone, compare the result, and drop the clone, without touching the source. This is useful for validating a new pipeline or a data fix before production.

Reading a Delta table without Spark

Many other engines and libraries can read Delta tables, so the data is not locked to one tool. A Python process can load a table into a DataFrame for a quick look, then pass a sample to a model.

# Pseudocode: any Delta-capable reader works here
df = read_delta("s3://example.internal/lake/gold/customer_features")
sample = df.sample(5000)

Check which features each reader supports, such as deletion vectors or newer protocol versions, before depending on it.

Protocol versions and upgrades

A Delta table records the minimum reader and writer protocol versions it needs. When you turn on a newer feature, the table may require newer clients. Upgrade readers before writers, test with a clone, and keep a list of which tools read each table. A silent upgrade of one writer can lock older readers out.

Cost thinking

Storage on an object store is cheap, but not free, and time travel keeps extra files. Compute cost depends on how much data a query touches. Good layout, compaction and filters reduce that. Track three numbers per large table: total size, file count and bytes scanned per common query. Changes in these are early warnings.

A short troubleshooting guide

Common mistakes

1. Too many small files. Streaming or frequent appends without compaction slow every query.
2. Over-partitioning. Partitioning by a high-cardinality column creates millions of tiny folders.
3. Vacuuming too soon. A short retention can break time travel and long jobs.
4. Assuming unique keys. Delta does not enforce primary keys, so duplicates can appear unless merges are correct.
5. Unbounded merges. A merge that scans the entire target is slow and expensive.
6. Ignoring schema changes. Turning on automatic evolution everywhere hides upstream breakage.
7. Using it as an OLTP database. Row-level, millisecond workloads belong elsewhere.
8. Deleting data but keeping history. A delete does not erase old versions until cleanup runs.
9. Sharing streaming checkpoints. Two jobs on one checkpoint corrupt each other's progress.
10. Not pinning versions for ML. Results cannot be reproduced when the table keeps changing.
11. Mixing writers with different library versions. Protocol upgrades can make a table unreadable by older clients.
12. Skipping monitoring. Slowdowns arrive quietly when file counts grow.

Frequently asked questions

What is Delta Lake in simple terms?

Delta Lake is a layer that turns a folder of data files into a reliable table. It keeps a log of every change so that updates are atomic, readers see a consistent view, and old versions can be queried. The data itself stays in open columnar files, which keeps it portable.

Is Delta Lake a database?

No. It is a storage format and protocol, not a server. You query Delta tables with engines such as Spark, SQL warehouses or other compatible tools. The tables behave like database tables in terms of transactions and schema rules, but there is no single process that owns the data.

What does ACID mean for a Delta table?

Atomicity means a write fully happens or not at all. Consistency means the table obeys its schema and constraints. Isolation means readers and writers do not see each other's partial work. Durability means committed changes are safely stored. Delta achieves this through its transaction log and immutable data files.

How does time travel work?

Delta keeps old data files until they are vacuumed, and the log records which files made up each version. To read an old version, the engine replays the log up to that version and reads the matching files. You can choose a version number or a timestamp. Once vacuum removes old files, those versions are gone.

How is Delta Lake different from Apache Iceberg?

Both add transactions, schema evolution and time travel to lake files. They differ in metadata design, catalog approach and ecosystem history. Delta has deep roots in Spark, while Iceberg is known for broad multi-engine support. The best choice usually depends on your platform, engines and governance setup.

Can Delta Lake handle streaming data?

Yes. A Delta table can be both a streaming sink and a streaming source, which lets one table serve batch and streaming readers. Exactly-once style behaviour comes from the combination of checkpoints and transactional commits. See Spark Streaming for the processing side.

Do I need Databricks to use Delta Lake?

No. The format is open and can be used with open-source Spark and other engines through connectors. Managed platforms such as Databricks add convenience features, but the core protocol and libraries do not depend on one vendor.

How do I delete personal data from a Delta table?

Run a delete for the affected rows, then run vacuum after the retention period so that old files containing that data are physically removed. Until vacuum runs, earlier versions may still hold the rows. Plan retention and access to history with your privacy rules in mind.

Why are my Delta queries getting slower?

The most common cause is a large number of small files, caused by frequent appends or streaming. Compaction and better clustering usually fix it. Other causes include missing partition filters, merges that scan the whole table, and stale statistics. Check file counts and sizes first.

Is Delta Lake good for machine learning data?

Yes. It gives typed, versioned tables with history, which makes training sets reproducible. You can pin a version, sample efficiently, and track changes with the change data feed. Many tabular models and classic algorithms such as XGBoost read Delta tables through a DataFrame.

Glossary

• ACID: A set of guarantees for reliable transactions.
• Change data feed: A record of row-level changes between table versions.
• Checkpoint: A compact snapshot of log state, used to speed up reads.
• Commit: An atomic unit of change recorded in the log.
• Compaction: Merging small files into larger ones.
• Data skipping: Using file statistics to avoid reading irrelevant files.
• Deletion vector: A marker that hides deleted rows without rewriting a file.
• Delta log: The ordered record of all actions on a table.
• Immutable file: A file that is never changed after it is written.
• Lakehouse: An architecture combining lake storage with warehouse-like reliability.
• MERGE: An upsert command that inserts, updates or deletes based on a match.
• Optimistic concurrency: Checking for conflicts at commit time instead of locking.
• Parquet: A columnar file format used for analytics.
• Partition: A division of table data by column value.
• Schema enforcement: Rejecting writes that do not match the schema.
• Schema evolution: Controlled changes to a table schema.
• Snapshot isolation: Each reader sees a stable version of the table.
• Time travel: Querying older versions of a table.
• Vacuum: Removing files no longer referenced by the log.
• Version: The sequence number of a table state.

Summary

Delta Lake makes lake storage trustworthy. Data stays in open columnar files, and a transaction log adds atomic commits, snapshot reads, time travel, schema rules and efficient upserts. It works with Apache Spark, PySpark and Spark Streaming, and it is a common choice for bronze, silver and gold tables.

Use it when you need reliable, versioned tables on object storage. Plan partitioning, compaction and retention, keep merges bounded, and pin versions for reproducible work. For tabular foundation models, Delta Lake provides typed, versioned and governed tables as inputs, and a safe place to write back predictions, imputed values, anomaly scores and labels with full history.

Related

It belongs to Table formats and lakehouse in my stack, under Data engineering and lakehouse. Others in the same group: