Apache Iceberg

Overview

Apache Iceberg is an open table format for large analytic tables stored as files in object storage. It adds a layer of metadata on top of data files such as Parquet, so that a pile of files behaves like a real table: it has a schema, it supports inserts, updates and deletes, it offers consistent reads while writes happen, and it can be rewound to an earlier state. Many engines can read and write the same Iceberg table.

The key idea is simple. Instead of finding data by listing folders, Iceberg tracks files in metadata. Each commit creates a new snapshot, and readers see a complete snapshot or nothing. This removes many classic data lake problems, such as half-written tables, slow directory listings and fragile partition schemes. For teams that prepare tables for tabular foundation models, Iceberg gives versioned, reproducible, evolving tables, which is exactly what training sets, context tables and evaluation sets need. See /model/architecture for how such models consume rows and columns.

What it is

Iceberg is a specification and a set of libraries, not a database server. There is no Iceberg service to run. A table is a set of data files plus a tree of metadata files, and a catalog that points to the current metadata. Engines such as Apache Spark, Apache Flink, Trino and Amazon Athena implement the specification, so they can all work on the same table.

What Iceberg gives you

• ACID transactions: commits are atomic. Readers never see partial writes.
• Snapshots and time travel: every commit makes a snapshot you can read later.
• Schema evolution: add, drop, rename and reorder columns without rewriting data.
• Hidden partitioning: partitioning by transforms of columns, applied for you.
• Partition evolution: change the partitioning scheme as data grows.
• Row-level changes: updates, deletes and merges, using delete files or rewritten files.
• Scan planning with statistics: skip files using column-level min and max values.
• Engine independence: open specification, many implementations.

What it does not do

Iceberg does not store data by itself and does not run queries. It does not replace a catalog service, although it needs one. It does not remove the need for maintenance, such as compacting small files and expiring old snapshots.

An everyday analogy

Think of a large library where books are shelved by whoever has time. Without a catalog, finding a book means walking every aisle. A table format is the catalog and the librarian's logbook. The catalog lists every book and its shelf. The logbook records every change: which books were added, removed or replaced, and when. A reader can ask for the library as it was on a given day, and the librarian can reconstruct it from the logbook. The books are the data files, and Iceberg is the catalog and logbook.

Where it came from

Iceberg was created at a large streaming video company to fix problems with big tables on Hive-style layouts, where a table was defined by a directory tree. Listing directories on object storage was slow, concurrent writes could produce inconsistent results, and renaming a column or changing partitions needed risky rewrites. The project was open sourced and became a top-level Apache Software Foundation project.

The surrounding landscape includes other open table formats, notably Delta Lake and Apache Hudi, which solve similar problems with different designs. Over time, major engines and cloud services added Iceberg support, and the specification grew through versions, adding row-level delete files and more features. Exact version numbers and feature dates are best checked in the official specification.

The problem it solved

Before table formats, a data lake table was a convention: files in a folder, a catalog entry that named the folder, and partitions found by listing subfolders. Problems followed:

• A reader could see a half-written set of files.
• Two writers could overwrite each other.
• Changing a column type could break old files.
• Partition columns leaked into queries, so users had to filter on them by hand.
• Listing millions of files made planning slow.

Iceberg replaced the convention with a contract.

How it works

The metadata tree

An Iceberg table has several layers.

1. Catalog: a small store that holds, for each table name, a pointer to the current metadata file.
2. Metadata file: a JSON file describing the schema, partition spec, properties and the list of snapshots.
3. Manifest list: one per snapshot. It lists the manifest files that make up that snapshot, with summary information.
4. Manifest files: lists of data files, with per-file statistics such as row count and column min and max.
5. Data files: the actual rows, usually in Parquet, sometimes ORC or Avro.

Analogy: the catalog is the front desk, the metadata file is the table of contents, the manifest list is a chapter index, manifests are page lists, and data files are the pages.

Reading a table

1. The engine asks the catalog for the current metadata file.
2. It picks the current snapshot, or one you named for time travel.
3. It reads the manifest list and prunes manifests using partition summaries.
4. It reads the remaining manifests and prunes data files using column statistics.
5. It scans only the surviving data files.

No directory listing is needed. Planning cost depends on metadata size, not on the number of folders.

Writing a table

1. The writer creates new data files.
2. It writes new manifests that list them, and a new manifest list for a new snapshot.
3. It writes a new metadata file that includes the new snapshot.
4. It asks the catalog to swap the pointer from the old metadata file to the new one, atomically.

If another writer committed in the meantime, the swap fails. The writer then checks whether its changes conflict, and retries on top of the new state if not. This is optimistic concurrency control. Readers keep using the old snapshot until the swap, so they never see partial work.

Row-level changes

There are two main approaches.

• Copy on write: rewrite the affected data files with the changes applied. Reads are simple and fast. Writes cost more.
• Merge on read: write small delete files that mark rows as removed, and merge them at read time. Writes are cheap. Reads do extra work until compaction.

Delete files can identify rows by file and position, or by equality on column values. Compaction later folds deletes into data files.

Hidden partitioning

In older layouts a user had to add a date partition column and filter on it. Iceberg stores a partition spec as transforms of source columns, such as day of a timestamp, month, a hash bucket of a key, or the first characters of a string. Queries filter on the original column, and Iceberg maps the filter to partitions. Users do not need to know the layout. Analogy: a post office sorts mail by postcode, but you only write the address.

Partition evolution

If data grows and daily partitions become too small, you can change to a coarser or finer spec. Old data keeps its old layout, and new data uses the new one. Queries work across both because each file records which spec it was written under.

Schema evolution

Iceberg tracks columns by unique ids, not by name or position. Renaming a column changes only the name mapped to an id. Adding a column assigns a new id, so old files simply have no data for it and read as null. Dropping a column removes the mapping. This avoids the classic bug where a dropped and re-added column name picks up stale data. See Schema evolution.

Snapshots and time travel

Each snapshot is a full description of the table at a moment. You can query a table as of a timestamp or a snapshot id, roll back to an earlier snapshot, or compare two snapshots to see what changed. Snapshots are kept until expired, so retention is a choice.

Maintenance operations

See Iceberg compaction for the details of keeping files healthy.

Key concepts and terms

A worked example

A team keeps a customer events table that is appended to daily, corrected by late updates, and read by analysts and a model pipeline. The example uses Spark SQL with fake names.

Step 1: Configure a catalog

from pyspark.sql import SparkSession

spark = (SparkSession.builder
  .appName("iceberg-demo")
  .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
  .config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.lake.type", "glue")
  .config("spark.sql.catalog.lake.warehouse", "s3://example-lake-bucket/warehouse")
  .getOrCreate())

The catalog here is backed by the AWS Glue Data Catalog. See AWS Glue Data Catalog. Other catalog types exist, such as REST, Hive and JDBC catalogs.

Step 2: Create a table with hidden partitioning

CREATE TABLE lake.analytics.customer_events (
  event_id     string,
  customer_id  string,
  event_type   string,
  amount       double,
  event_ts     timestamp
)
USING iceberg
PARTITIONED BY (days(event_ts), bucket(16, customer_id))
TBLPROPERTIES (
  'format-version' = '2',
  'write.format.default' = 'parquet',
  'write.target-file-size-bytes' = '268435456'
);

Step 3: Append and update

INSERT INTO lake.analytics.customer_events
SELECT event_id, customer_id, event_type, amount, event_ts
FROM staging.new_events;

MERGE INTO lake.analytics.customer_events t
USING staging.corrections s
ON t.event_id = s.event_id
WHEN MATCHED THEN UPDATE SET amount = s.amount
WHEN NOT MATCHED THEN INSERT *;

The merge is one atomic commit. Readers see the table either before or after, never in the middle.

Step 4: Inspect snapshots and travel back

SELECT snapshot_id, committed_at, operation
FROM lake.analytics.customer_events.snapshots
ORDER BY committed_at DESC;

SELECT count(*) FROM lake.analytics.customer_events
TIMESTAMP AS OF '2026-01-15 00:00:00';

CALL lake.system.rollback_to_snapshot('analytics.customer_events', 1234567890123456789);

The snapshot id above is a placeholder. Time travel makes it possible to rebuild a training table exactly as it was on a given date.

Step 5: Evolve the schema

ALTER TABLE lake.analytics.customer_events ADD COLUMN channel string;
ALTER TABLE lake.analytics.customer_events RENAME COLUMN amount TO amount_usd;

No data files are rewritten. Old rows read the new column as null.

Step 6: Maintain

CALL lake.system.rewrite_data_files(
  table => 'analytics.customer_events',
  strategy => 'binpack',
  options => map('target-file-size-bytes', '268435456')
);

CALL lake.system.expire_snapshots(
  table => 'analytics.customer_events',
  older_than => TIMESTAMP '2026-01-01 00:00:00',
  retain_last => 10
);

A numeric walk-through of pruning

Imagine the table holds 10,000 data files of about 256 MB each, in daily partitions across 27 months. A query for one week of events for one customer bucket, using a filter on event_ts and customer_id, can prune by partition to roughly 7 days times 1 of 16 buckets of each day's files. If each day has about 12 files, that is about 7 times 12 divided by 16, which is a handful of files. Column statistics can prune further. The query reads a few files rather than 10,000, and the planner never lists a directory.

Inspecting a table through metadata tables

Engines expose metadata as queryable tables. They are the quickest way to understand a table's health.

-- how many files and how big are they?
SELECT count(*) AS files,
       sum(file_size_in_bytes) / 1024 / 1024 AS total_mb,
       avg(file_size_in_bytes) / 1024 / 1024 AS avg_mb
FROM lake.analytics.customer_events.files;

-- what changed in each commit?
SELECT committed_at, operation, summary['added-records'] AS added, summary['deleted-records'] AS deleted
FROM lake.analytics.customer_events.snapshots
ORDER BY committed_at DESC
LIMIT 10;

-- how is data spread across partitions?
SELECT partition, record_count, file_count
FROM lake.analytics.customer_events.partitions
ORDER BY record_count DESC
LIMIT 20;

If the average file size is a few megabytes while the target is hundreds, compaction is overdue.

Branches and tags

Branches and tags are named references to snapshots. A tag marks a version you want to keep, such as the table used to train a published model. A branch lets you write changes on the side, validate them, and then publish by fast-forwarding the main branch. This write, audit, publish pattern keeps bad data away from readers.

ALTER TABLE lake.analytics.customer_events CREATE TAG `model_v3_training` RETAIN 365 DAYS;
ALTER TABLE lake.analytics.customer_events CREATE BRANCH `staging_load`;

Reading from Python

Libraries exist to read Iceberg tables without Spark, which suits small jobs and notebooks.

from pyiceberg.catalog import load_catalog

catalog = load_catalog("lake", **{"type": "glue"})
table = catalog.load_table("analytics.customer_events")
df = table.scan(
    row_filter="event_ts >= '2026-01-01T00:00:00'",
    selected_fields=("customer_id", "amount", "event_ts"),
).to_pandas()
print(df.head())

Strengths and limitations

Strengths

• Reliable commits with snapshot isolation on plain object storage.
• Time travel, rollback and reproducible reads.
• Safe schema and partition evolution without rewrites.
• Hidden partitioning removes a class of user errors.
• Works across engines, which avoids lock-in.
• Metadata-driven planning scales to very large tables.
• Row-level deletes and merges for corrections and privacy requests.
• Open specification with an active community.

Limitations

• It needs a catalog, and the choice of catalog affects portability.
• Streaming and frequent small commits create many small files and many snapshots. Maintenance is required.
• Merge on read adds read cost until compaction runs.
• Concurrent writers to the same partitions can conflict and retry.
• Engine support varies by feature and version. Check support before relying on a new feature.
• Time travel retention costs storage until snapshots are expired.
• It is a table format, not a low latency serving store.

How it compares with alternatives

The three open formats have been moving closer in features, and tools exist to read one as another in some cases. Choose by the engines you use, the features you need and the catalog you can run. See also Databricks and Snowflake, which support open formats to different degrees.

Picking a format in plain steps

1. List the engines that must read and write the table. Check each one's support for the formats.
2. List the features you need: row-level updates, time travel, partition evolution, streaming ingest.
3. Check the catalog options available in your environment.
4. Consider the team's skills and the maintenance tooling you already have.
5. Run a small pilot with real data and measure planning time, write time and file counts.

Migrating an existing folder of Parquet files

An existing Parquet table can often be registered as an Iceberg table without rewriting data, by adding the files to a new table's metadata. This is quick, but the files keep their old layout and sizes. A follow-up compaction can reorganise them. For a safer path, create a new Iceberg table and copy data with a CREATE TABLE AS SELECT, then switch readers once validated.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns of mixed types, cope with missing values, may see columns with weak or no names, and predict from labelled example rows supplied in context. They are pre-trained on synthetic or real tables and can serve imputation, anomaly detection, continual learning and semantic or sector classification. See /model/research and /model/benchmarks.

Iceberg does not train or run such models. Its role is to be the dependable table layer under the data they use.

Reproducible training and evaluation tables

Evaluating a tabular model fairly means feeding every method the same data. Iceberg snapshots give an exact name for a table version. Store the snapshot id with each experiment, and you can rebuild the identical input later. This helps with audits, regression checks and comparing model versions.

Point-in-time correct features

Leakage happens when a feature uses information from after the prediction time. With time travel and event timestamps, you can build features as of a past date and verify them. A frozen snapshot of the label table next to a frozen snapshot of the feature table pins down what the model could have known.

Handling schema change without breaking models

A model that reads columns by position or by type is sensitive to schema changes. Iceberg tracks columns by id, so a rename does not shift data into the wrong column. Adding a column makes old rows read as null, which a tabular model treats as a missing value. Teams can then check changes with Schema drift detection and Data contract validation before a new column is exposed to a model.

Missing values and imputation

Because added columns read as null for old rows, tables often carry structured missingness. Iceberg stores per-file null counts in its statistics, which makes it cheap to profile missing values per column without scanning data. A pipeline can compare the null share before and after a model fills gaps, and store both versions as separate columns or separate snapshots.

Context tables for in-context learning

A context table of labelled rows can be a branch or a tagged snapshot. A scheduled job refreshes it, validates it, and tags the good version. Serving code reads the tag, so a bad refresh never reaches the model. Rolling back is a metadata change, not a data copy.

Anomaly detection and corrections

When a model flags rows as anomalous and a reviewer fixes them, the corrections can be applied with MERGE or DELETE. Row-level changes with an audit trail in snapshots mean both the original and the corrected tables are recoverable. This is useful for privacy requests as well, where rows must be removed and later proof is needed.

Streaming and continual learning

Streams from Amazon MSK or Apache Kafka can be written into Iceberg with Flink or Spark, giving a continuously updated table with atomic commits. A continual learning or monitoring job reads the newest snapshots, compares them with earlier ones, and decides whether to update or re-evaluate a model.

Pre-training corpora

A collection of many source tables needs deduplication, versioning and lineage. Iceberg tables hold such corpora with clear versions, and metadata tables such as files and snapshots expose how many rows and files each version has. See Data lineage and Synthetic data generation for related ideas.

Governance

Access control is usually applied in the catalog and the query engine, for example with AWS Lake Formation, not in the table format itself. Sensitive columns can be tokenized upstream with Data tokenization. Because deletes are real commits, retention and expiry policies can be enforced, but remember that old snapshots keep deleted data until they are expired.

A simple governance pattern for model data

This keeps every number a model saw traceable to a snapshot, and every number it produced traceable to a model version.

Handling late and corrected data

Real tables receive late arrivals and corrections. With Iceberg, a nightly MERGE can apply them. For model evaluation, record both the as-known-then view, using an older snapshot, and the as-corrected-now view, using the latest. Comparing the two shows how much a model depends on data that was later revised.

Practical guidance and best practices

Choose the catalog deliberately

• Pick one catalog as the source of truth for each table.
• Prefer a catalog that several engines can use, such as one that implements the REST catalog protocol or the Glue Data Catalog.
• Avoid writing to the same table through two different catalogs.
• See Iceberg cataloging for the trade-offs.

Design partitions with care

• Partition by time at a granularity that gives reasonably large partitions.
• Use bucket transforms for high-cardinality keys that you join or filter on.
• Do not over-partition. Many tiny partitions create many tiny files.
• Remember that partition spec can change later, so start simple.

Manage file size

• Target data files in the hundreds of megabytes.
• Set write properties for target file size and distribution mode.
• Run compaction on a schedule and after heavy streaming ingestion.
• Rewrite manifests when planning slows down.

Control snapshots and storage

• Expire snapshots on a schedule that matches your time travel needs.
• Keep a minimum number of recent snapshots for rollback.
• Remove orphan files carefully, with a safe age threshold, so that in-flight writes are not deleted.
• Use tags for snapshots that must be kept, such as those used for published models.

Choose a row-level strategy

• Use copy on write for read-heavy tables with infrequent updates.
• Use merge on read for write-heavy tables with frequent small updates, and compact often.
• Prefer merging in batches over many tiny single-row commits.

Operate safely

• Test schema and partition changes on a copy.
• Limit who can run expire and delete operations.
• Monitor commit failures and retries.
• Keep a record of table properties and owners.
• Query metadata tables to check file counts, sizes and snapshot history.

Connect it to the rest of the stack

Bring raw files in with AWS DataSync, process with EMR Serverless or other Spark runtimes, and query with Amazon Athena. Use Apache Airflow to orchestrate the maintenance jobs.

Common mistakes

• Never running compaction. Small files accumulate, and reads and planning slow down.
• Never expiring snapshots. Storage grows without bound and deleted data lingers.
• Expiring too aggressively. Time travel and rollback windows disappear.
• Over-partitioning. Daily partitions on a tiny table create thousands of small files.
• Mixing writers and catalogs. Two catalogs for one table can corrupt the pointer logic.
• Assuming deletes are physical. A delete removes rows from the current snapshot, but old files stay until snapshots expire.
• Using single-row commits at high frequency. Each commit adds metadata. Batch them.
• Ignoring engine support. A feature that works in one engine may not be implemented in another.
• Treating Iceberg as a security layer. It does not enforce access control by itself.

• Renaming by dropping and adding. That loses the column id link. Use rename.

• Ignoring metadata size. Millions of tiny commits produce huge manifests. Rewrite manifests and batch writes.

• Forgetting that readers pin snapshots. A long-running query can keep old files alive, so expiring files too soon can break it. Keep a safety window.

Frequently asked questions

What is Apache Iceberg?

Apache Iceberg is an open table format that organises data files in object storage into reliable tables. It adds metadata for schema, partitioning and snapshots, and provides atomic commits, time travel, schema evolution and row-level changes. Engines such as Spark, Flink, Trino and Athena can all use the same Iceberg table.

Is Iceberg a database?

No. Iceberg is a specification and a set of libraries. It does not run as a server and does not execute queries. You need a query engine to read and write tables, and a catalog to track table locations. Together they provide database-like behaviour over files.

How is Iceberg different from Parquet?

Parquet is a file format that stores columns of data efficiently. Iceberg is a table format that organises many files, such as many Parquet files, into a table with a schema, snapshots and transactions. Iceberg tables usually use Parquet for their data files.

What is hidden partitioning?

Hidden partitioning means the table defines partitions as transforms of regular columns, such as the day of a timestamp, and the engine applies them automatically. Users filter on the original column and do not need to know or maintain extra partition columns. It reduces mistakes and lets partitioning change over time.

What is time travel in Iceberg?

Time travel is the ability to read a table as it was at an earlier snapshot or timestamp. Every commit creates a snapshot, so you can query old states, compare versions or roll back a bad change. It works until the old snapshots are expired.

How does Iceberg compare with Delta Lake?

Both add transactions and versioning to data lakes. Iceberg uses a tree of metadata files with manifests and snapshots and is designed for engine independence, while Delta Lake uses an ordered transaction log and began with strong Spark integration. Both have evolved, so check the features you need in the engines you use.

Do I need to compact Iceberg tables?

Yes, in most real workloads. Frequent writes, streaming and merges create many small files and delete files. Regular compaction merges them into larger files and folds in deletes, which speeds up reads and planning. Also rewrite manifests and expire snapshots as needed.

Can Iceberg help with machine learning on tables?

It helps by making data reproducible and consistent. You can name the exact snapshot used for training or evaluation, build point-in-time features, evolve schemas safely and apply corrections with an audit trail. The learning itself happens in other tools, with Iceberg providing the dependable data layer.

How does schema evolution work without rewriting data?

Iceberg identifies each column by a unique id stored in the metadata and in file schemas. Adding a column creates a new id, and old files read it as null. Renaming changes only the name for an id. Dropping removes the column from the schema. None of these changes touch the data files.

Which catalog should I use with Iceberg?

Pick a catalog that your engines support and that fits your governance needs. Common choices include the Glue Data Catalog, a REST catalog service, a Hive metastore or a JDBC-backed catalog. Use one catalog per table as the source of truth and avoid mixing catalogs for the same table.

Glossary

• ACID: atomicity, consistency, isolation and durability, the properties of reliable transactions.
• Branch: a named, movable reference to a line of snapshots.
• Catalog: the store that maps table names to current metadata.
• Compaction: merging small files into larger ones.
• Copy on write: rewriting files when rows change.
• Delete file: a file that marks rows as deleted.
• Field id: a stable identifier for a column.
• Manifest: a file listing data files and their statistics.
• Merge on read: applying deletes at read time.
• Optimistic concurrency: assuming no conflict and checking at commit time.
• Orphan file: a data file not referenced by any snapshot.
• Partition spec: the definition of how data is partitioned.
• Snapshot: the table state after a commit.
• Tag: a named reference to one snapshot.
• Time travel: reading an earlier table state.
• Transform: a function used in a partition spec.

Summary

Apache Iceberg turns files in object storage into reliable, evolving tables. A catalog points to a metadata file, which leads through manifest lists and manifests to data files. Commits are atomic, snapshots allow time travel and rollback, hidden partitioning and partition evolution remove layout headaches, and column ids make schema changes safe. Regular compaction and snapshot expiry keep it healthy.

For tabular foundation models, Iceberg supplies versioned, reproducible tables: exact training and evaluation snapshots, point-in-time features, tagged context tables, safe schema changes that read as missing values, and auditable corrections. It does not run the model, enforce access control or replace a catalog, but it gives the model's data a solid and honest foundation.

Related

It belongs to Table formats and lakehouse in my stack, under Data engineering and lakehouse. Others in the same group: