Iceberg cataloging

Overview

Iceberg cataloging is the practice of registering each Apache Iceberg table in a catalog. The catalog stores one key fact per table: the location of the current metadata file. Every engine that reads or writes the table asks the catalog for that pointer first. When a writer finishes a change, it asks the catalog to swap the pointer from the old metadata file to the new one in a single atomic step.

This small job matters a lot. It is what lets Spark, Trino, Athena and Flink share the same tables without corrupting each other's work. Common catalog choices are the AWS Glue Data Catalog, a Hive metastore, a REST catalog service, a JDBC database and Nessie. They differ in features, security and operations, but they all answer the same question: "Where is the latest version of this table?"

For teams that feed tabular data to machine learning models, a good catalog gives a stable, versioned view of each table. That makes training sets reproducible and lets many jobs read the same data safely.

What it is

A catalog is a service or a small database that maps table names to table metadata locations. In Iceberg, a table is not a folder of files that a engine scans. A table is a tree of metadata files that point to data files. The catalog holds the entry point to that tree.

Think of a library. The books are the data files. The card catalog tells you which shelf holds which book. Without the card catalog, every reader would wander the shelves and might pick up a book that is being moved. The Iceberg catalog is the card catalog for the lake.

What a catalog stores

• A namespace (like a database or schema) and a table name.
• The path of the current metadata file, often a JSON file in object storage.
• Optional properties, such as owner, description and table settings.
• In some catalogs, access rules, views and branch or tag information.

What a catalog does not store

• It does not store the data files. Those live in object storage such as Amazon S3.
• It does not store the full table history. The metadata files hold that.
• It does not run queries. Engines do that.

Why not just list a folder?

You could find the newest metadata file by listing a folder. That is slow and unsafe. Two writers could create files at the same time, and a reader could pick the wrong one. A catalog gives one authoritative answer and a safe way to update it.

Where it came from

Before table formats like Iceberg, most data lakes used the Hive metastore. It mapped a table name to a folder and a set of partitions. Engines listed the files in those folders. This worked for read-heavy batch jobs. It was weak for concurrent writes, deletes and schema changes, because there was no atomic way to publish a set of files together.

Apache Iceberg was created to fix those problems. It moved the list of files out of folder listings and into metadata files. That left a smaller job for the catalog: keep a pointer to the latest metadata and update it atomically. The project defined a catalog interface so that many back ends could do that job.

Over time several implementations appeared:

• Hive metastore catalog, reusing existing infrastructure.
• AWS Glue Data Catalog, a managed option on AWS.
• JDBC catalog, which keeps the pointer in a relational database.
• Hadoop catalog, which uses file system rename and suits tests, not shared production use on object storage.
• Nessie, a catalog with Git-like branches and tags.
• REST catalog, an open HTTP specification that lets any service act as a catalog.

The REST catalog specification became a common way to connect engines to catalogs, because clients only need to speak one protocol. Cloud vendors and open source projects now offer services that implement it.

How it works

This section walks through what happens when an engine uses an Iceberg catalog.

Step 1: Configure the catalog in the engine

Each engine is told about one or more catalogs by name. A catalog has a type and some settings, such as a URI and a warehouse location. The engine uses the name as a prefix in table identifiers, like lake.sales.orders.

Analogy: a phone's contacts list. You add a contact once, with a name and a number. After that you only use the name.

Step 2: Load the table

When a query names a table, the engine asks the catalog to load it. The catalog returns the table's current metadata location. The engine reads that metadata file from storage.

Step 3: Read the metadata tree

The metadata file lists the schema, the partition spec, the snapshots and the current snapshot. A snapshot points to a manifest list. The manifest list points to manifests. The manifests list data files with statistics. The engine uses those statistics to skip files that cannot match the query.

Step 4: Plan and run the query

The engine plans work from the file list. It never has to list the folder. The read is tied to one snapshot, so it sees a consistent view even if writers commit during the query.

Step 5: Write new data files

A writer creates new data files and new manifests. It then creates a new metadata file that includes a new snapshot. None of this is visible to readers yet, because the catalog still points to the old metadata.

Step 6: Commit with compare-and-swap

The writer asks the catalog: "Change the pointer for this table from the old metadata to the new metadata, but only if the current pointer is still the old one." This is a compare-and-swap. If another writer committed first, the swap fails.

Analogy: two people editing a shared document by email. Each sends their version back with "based on version 7." If version 8 already exists, the second person must merge and try again.

Step 7: Retry on conflict

If the swap fails, Iceberg checks whether the changes conflict. Many appends do not conflict, so the writer rebuilds its metadata on top of the new state and tries again. Real conflicts, such as two jobs rewriting the same files, cause the job to fail.

Step 8: Readers see the new state

After a successful swap, the next load returns the new metadata. Old readers keep the snapshot they started with. No one ever sees a half-finished commit.

How each catalog type does the atomic swap

Key concepts and terms

A worked example

This example shows a Spark session that uses a REST catalog and a Glue catalog side by side. Host names and bucket names are placeholders.

Configure two catalogs

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("iceberg-catalog-demo")
    .config("spark.sql.extensions",
            "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    # A REST catalog
    .config("spark.sql.catalog.rest", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.rest.type", "rest")
    .config("spark.sql.catalog.rest.uri", "https://catalog.example.internal/api")
    .config("spark.sql.catalog.rest.warehouse", "example-warehouse")
    # A Glue catalog
    .config("spark.sql.catalog.glue", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.glue.catalog-impl",
            "org.apache.iceberg.aws.glue.GlueCatalog")
    .config("spark.sql.catalog.glue.warehouse", "s3://example-bucket/warehouse")
    .getOrCreate()
)

Create a table and write rows

CREATE NAMESPACE IF NOT EXISTS glue.sales;

CREATE TABLE glue.sales.orders (
  order_id   BIGINT,
  customer   STRING,
  amount     DECIMAL(12,2),
  order_ts   TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(order_ts));

INSERT INTO glue.sales.orders VALUES
  (1, 'c-100', 25.50, TIMESTAMP '2026-01-05 10:00:00'),
  (2, 'c-101', 80.00, TIMESTAMP '2026-01-05 11:30:00');

Inspect what the catalog points to

SELECT * FROM glue.sales.orders.snapshots;
SELECT * FROM glue.sales.orders.metadata_log_entries;

The metadata_log_entries view shows each metadata file the table has used. The latest one is what the catalog points to.

Read the same table from another engine

In Trino, a catalog file points at the same Glue catalog:

connector.name=iceberg
iceberg.catalog.type=glue

Then:

SELECT customer, SUM(amount) AS total
FROM iceberg.sales.orders
GROUP BY customer;

Trino loads the table through Glue, reads the same metadata and sees the same snapshot as Spark. Athena can read it the same way, and Flink can write to it with a matching catalog definition.

Numeric walk-through of a conflict

Suppose the current metadata is v3. Two jobs start at the same time.

1. Job A reads v3, writes files, builds v4a.
2. Job B reads v3, writes files, builds v4b.
3. Job A asks the catalog: swap v3 to v4a. This succeeds.
4. Job B asks: swap v3 to v4b. The current value is v4a, so it fails.
5. Job B reloads v4a. Both jobs only appended data, so there is no conflict. Job B builds v5 on top of v4a and swaps v4a to v5. This succeeds.

Final state: v5 contains the data from both jobs.

Strengths and limitations

Strengths

• One source of truth for table location, so engines agree.
• Atomic commits protect readers from partial writes.
• Multi-engine access: the same table can serve batch, SQL and streaming.
• Small footprint: the catalog stores a pointer, not data.
• Open interface: the REST specification avoids tying clients to one vendor.
• Governance hooks: many catalogs add access control, auditing and credential vending.
• Versioning options: Nessie style catalogs add branches and tags across tables.

Limitations

• The catalog is a critical dependency. If it is down, tables cannot be loaded or committed.
• Features differ by catalog. Views, multi-table commits and branches are not available everywhere.
• Lock based or database based catalogs can become a bottleneck with very frequent commits.
• Mixing catalogs for the same table is dangerous. Two catalogs may hold different pointers.
• Migration between catalogs needs care, since the pointer must be registered in the new place without losing history.
• Permissions are split between the catalog and the storage layer, which can confuse teams.

How it compares with alternatives

A related choice is the catalog itself. The table below compares common options.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns with mixed types, missing values and sometimes no useful column names. A catalog does not train or run such models, but it shapes the data they see.

Stable, versioned inputs

Pre-training and fine-tuning need many tables. A catalog gives each table a name and a snapshot history. A training run can record the snapshot id it used. Months later, the same run can be repeated against the same snapshot, even if the table has changed since. That is key for honest evaluation and for continual learning, where a model is updated as new data arrives and you need to know exactly what it saw.

Row and column structure

Iceberg stores the schema, including column ids, types and nullability, in metadata. A data loader can read this schema from the catalog and build the right encoders for numeric, categorical, date and text columns. Column ids stay stable when names change, which helps pipelines that must map columns across versions. See Schema evolution.

Missing values and imputation

Metadata statistics, such as null counts per file, help you find columns with many gaps before training. Imputation research, such as filling holes in a table, needs reliable tables where "missing" is recorded the same way across partitions. A catalog does not fix missing values, but a governed table makes it easier to apply one rule everywhere.

Anomaly detection

Because each commit is a snapshot, you can compare two snapshots and find rows that changed, appeared or vanished. This is a base for anomaly checks: a sudden jump in nulls, a new category or a shifted distribution can be spotted by comparing a new snapshot to an older one. See Schema drift detection and Anomaly detection.

In-context learning on tables

Models such as TabPFN take a labeled table as context and predict new rows without gradient updates. The context table must be a clean, well defined slice. A catalog backed table lets a service pull exactly that slice, at a known snapshot, and pass it to the model. If the table changes, the snapshot id tells you which version produced a given prediction.

Synthetic and real pre-training tables

Models that pre-train on synthetic tables and on real ones need a registry of sources. A catalog can hold both kinds under separate namespaces, with properties that mark origin, license and refresh date. This helps teams keep synthetic and real data apart when they measure results. See /model/research and /model/architecture for how such models are built, and /model/benchmarks for how they are compared.

Sector and semantic classification

When columns have no helpful names, a model may infer meaning from values. Catalog properties and documentation fields can still carry business tags, such as "customer id" or "postal code." These tags are useful labels for evaluating semantic classification of columns. See Data catalog for the wider idea.

Governance and lineage

Catalogs often connect to access control and lineage tools. Knowing who may read which table, and where a table came from, is part of responsible model training. See Data lineage and AWS Lake Formation.

Practical guidance and best practices

Pick one catalog per table

Every table should be owned by exactly one catalog. Other systems may read through that catalog, but they should not hold their own pointer.

Prefer an open protocol where you can

A REST catalog lets new engines connect without special code. It also lets the service add features like credential vending and server side planning.

Plan for high availability

Treat the catalog like a database. Use backups, monitoring and a clear failover plan. If you use a managed service, understand its limits and regional behavior.

Secure both layers

Control who can call the catalog and who can read the storage. If engines read files directly, a user who can read the bucket can bypass catalog rules. Credential vending or storage policies can close this gap.

Use consistent naming

Pick a pattern for namespaces such as domain.layer. Keep names lowercase and avoid special characters, because different engines handle case and punctuation differently.

Keep commit rates reasonable

Each commit writes new metadata and swaps a pointer. Very frequent tiny commits create many metadata files and more conflicts. Batch writes where you can, and schedule maintenance such as Iceberg compaction.

Record snapshot ids in downstream jobs

For training and reporting jobs, store the snapshot id with the output. It makes results reproducible and easy to audit.

Use branches for risky changes

With a catalog that supports branches, write to a branch, validate, and then publish. This pairs well with data contract validation.

Test failover and conflict handling

Run a test where two writers commit at once. Check that your job retries correctly and that your alerts fire when it cannot.

Clean up old metadata

Expire snapshots and remove orphan files on a schedule. Without cleanup, the metadata tree and storage keep growing. See Apache Iceberg for maintenance procedures.

Understanding the REST catalog in more detail

The REST catalog specification describes a set of HTTP calls. A client can list namespaces, create and drop tables, load a table, and commit changes. The client sends the changes it wants to make, and the server decides if they can be applied.

This design has several practical benefits.

• Clients stay thin. They do not need database drivers or metastore libraries.
• The server can enforce rules, such as who may drop a table.
• The server can hand out short-lived storage credentials, so users need no long-lived keys.
• The server can plan scans on behalf of the client in some designs.
• New engines can join by implementing one protocol.

Analogy: a hotel front desk. Guests do not walk into the back office and edit the room book. They ask the desk, and the desk applies the rules.

A simple ownership checklist for a new table

1. Choose the catalog that will own the table.
2. Choose a namespace that matches the data domain and layer.
3. Decide on a storage location and check that the catalog may write there.
4. Set table properties such as format version and default file format.
5. Add a description and an owner tag so people can find and question the table.
6. Grant read access to consumers through the catalog and the storage layer.
7. Add the table to your lineage and quality tooling.
8. Schedule maintenance, including compaction and snapshot expiry.

Moving a table between catalogs

Migration comes up when teams change platforms. The steps are usually these.

1. Stop or pause writers to the table.
2. Record the current metadata file location from the old catalog.
3. Register that location as a table in the new catalog.
4. Run a few reads in the new catalog and compare row counts and snapshot ids.
5. Switch writers to the new catalog.
6. Remove the entry in the old catalog, taking care not to delete data files.

If you skip step 1, a writer may commit in the old catalog after you registered the pointer, and the two catalogs will disagree.

Monitoring a catalog

A catalog should have simple health checks.

Handling permissions with care

There are two doors to a table. One is the catalog. The other is the storage bucket. A strong setup closes the second door for normal users and lets only the catalog, or a trusted engine role, touch the files. Credential vending is one way to do this: the catalog checks the user and then gives a short-lived key scoped to that table path.

If you cannot close the storage door, treat catalog rules as advice, not as a hard wall. Document that clearly for auditors.

Catalog design for machine learning platforms

A platform that serves many model projects often uses a layout like this.

catalogs:
  prod:
    type: rest
    uri: https://catalog.example.internal/api
    namespaces:
      - raw_events
      - clean_events
      - features
      - training_sets
  sandbox:
    type: rest
    uri: https://sandbox-catalog.example.internal/api
    namespaces:
      - experiments

Training sets live in their own namespace. Each one is written once, tagged with a model run id, and never changed afterward. Experiments live in a sandbox catalog so that mistakes cannot touch production tables.

Tags for reproducible training

Where the catalog supports tags, create one for each training run.

ALTER TABLE lake.features.customer_daily
CREATE TAG `train_2026_q1` AS OF VERSION 4821039847501;

The tag keeps that snapshot from being removed by snapshot expiry. Later you can read it with a time travel query and reproduce the exact input.

SELECT * FROM lake.features.customer_daily VERSION AS OF 'train_2026_q1';

What can go wrong during a commit

Common mistakes

• Registering the same table in two catalogs, which leads to diverging pointers.
• Using the Hadoop catalog on object storage for shared tables, where rename is not atomic.
• Forgetting that the catalog and storage have separate permissions.
• Copying table folders by hand to a new location and not updating metadata paths.
• Letting very many small commits pile up without compaction.
• Mixing engine versions that support different table format versions.
• Hard coding metadata file paths in jobs instead of loading by table name.
• Ignoring catalog downtime in pipeline design.
• Dropping a table in the catalog and assuming the data files are gone. In many setups, files remain until purged.
• Relying on case sensitive names across engines that fold case.

Frequently asked questions

What is an Iceberg catalog?

An Iceberg catalog is a service or database that maps table names to the location of each table's current metadata file. Engines ask it where a table lives and ask it to publish new versions. It does not hold the data files themselves.

Why does Iceberg need a catalog at all?

Iceberg tables are made of metadata files and data files in object storage. Something must say which metadata file is the latest and must update that choice atomically. The catalog does both, which keeps readers and writers consistent.

Which catalog should I choose?

If you run mostly on AWS and want a managed service, the Glue Data Catalog is a common choice. If you need many engines and clouds, a REST catalog is attractive. If you want branches and tags across tables, consider Nessie. Choose the one that matches your security, scale and operations needs.

Can Spark, Trino, Athena and Flink use the same catalog?

Yes, as long as each engine supports that catalog type and a compatible table format version. They all load the same metadata and see the same snapshots. Check each engine's documentation for the settings it needs.

What happens if two jobs write at the same time?

Each job tries to swap the catalog pointer from the version it started with. The first one wins. The second reloads the new state and retries if the changes do not conflict. If they do conflict, the second job fails with an error.

Is the Hive metastore still a good choice?

It works and many sites already run it, so it is a practical choice for existing setups. It adds the burden of running a service and a database. Newer catalogs may offer better features for new projects.

How do I move a table from one catalog to another?

You register the table's current metadata location in the new catalog and then stop writing through the old one. Tools and procedures exist for registering existing tables. Do it during a quiet window and verify snapshots afterward.

Does the catalog affect query speed?

Only a little. It adds a small lookup when a table is loaded. The speed of queries depends mostly on file layout, statistics and the engine. The catalog can affect commit speed when commits are very frequent.

How does a catalog help machine learning work?

It gives each table a stable name and a version history, so a training job can record the exact snapshot it used. That makes experiments repeatable and helps with audits. It also centralizes permissions and metadata.

Do I need a catalog for a quick local test?

Not always. A local file based catalog is fine for tests. For shared work, use a real catalog so that commits are safe.

Glossary

• Atomic commit: A change that either happens fully or not at all.
• Branch: A named line of table history that can be written and later merged or published.
• Catalog: A service that tracks table names and metadata locations.
• Compare-and-swap: An update that works only when the stored value matches the expected value.
• Credential vending: A catalog issuing short-lived storage keys to a client.
• Data file: A file, often Parquet, that holds table rows.
• Manifest: A file that lists data files and their statistics.
• Metadata file: A JSON file describing schema, partitions and snapshots.
• Namespace: A grouping of tables inside a catalog.
• Optimistic concurrency: Letting writers work without locks and retrying on conflict.
• REST catalog: A catalog that follows the Iceberg REST specification.
• Snapshot: The state of a table at one commit.
• Tag: A named pointer to a snapshot that does not move.
• Warehouse: The default storage root for new tables.

Summary

Iceberg cataloging gives every table a name, a current metadata pointer and a safe way to change that pointer. The catalog, whether Glue, Hive, REST, JDBC or Nessie, is small but critical. It makes atomic commits possible and lets Spark, Trino, Athena and Flink work on the same tables without stepping on each other.

For teams that build models on raw tables, the catalog supplies stable names, schemas, snapshots and governance. Those features make training sets reproducible and support imputation, anomaly checks and in-context learning on well defined table slices. Choose one catalog per table, secure both the catalog and the storage, keep commit rates sane, and record snapshot ids in downstream jobs.

Related

It belongs to Table formats and lakehouse in my stack, under Data engineering and lakehouse. Others in the same group: