Iceberg compaction

Overview

Iceberg compaction is the maintenance work that merges many small data files into fewer large ones. It also folds in row level deletes, rewrites manifests and, together with snapshot expiry, removes files that no longer matter. Without it, tables that receive frequent small writes, such as those fed by streams, slowly become slow and costly to query.

The reason is simple. Query engines pay a fixed cost to open each file. A table with a million tiny files spends most of its time opening files, not reading data. Compaction trades a little background compute for much faster reads, smaller metadata and lower storage and request costs. In Apache Iceberg this is done with procedures such as rewrite_data_files, rewrite_manifests, expire_snapshots and remove_orphan_files.

For anyone who trains or serves models on tables, compaction keeps the source data tidy and fast to scan. That helps feature pipelines, training set builds and any service that reads a context table on demand.

What it is

Compaction is a set of table maintenance actions. It does not change what the table means. The rows you can query before and after are the same. Only the physical layout changes.

Analogy: a school library receives books in small boxes every day. If you never shelve them, finding a book means opening hundreds of boxes. Compaction is the weekly job of unpacking the boxes and putting books on proper shelves in order.

The main actions

Why files get small

• Streaming jobs commit every few seconds or minutes, each commit writing tiny files.
• Highly partitioned tables spread a small batch over many partitions.
• Many parallel writer tasks each write a little.
• Frequent row level updates create many small delete files.
• Late arriving data lands in old partitions in small pieces.

What counts as a good file size

Many teams aim for data files in the range of a few hundred megabytes, with the Iceberg default target at 512 MB for the write target size. The right number depends on the engine and the storage. Files that are too large limit parallelism. Files that are too small cause overhead. Aim for a size where each task has meaningful work.

Where it came from

The small file problem is much older than Iceberg. It appeared in Hadoop, where the name node tracked every file and tiny files strained it. Later, object storage made each file open a network request, so the cost moved from memory to latency and request fees.

Early lakes used folders of Parquet files. Compaction meant rewriting a partition folder by hand and swapping it in, which risked readers seeing partial results. Table formats such as Apache Iceberg, Delta Lake and Apache Hudi changed that. Because they track files in metadata and commit changes atomically, a rewrite job can build new files in the background and swap them in with one commit. Readers either see the old files or the new ones, never a mix.

Iceberg added maintenance procedures and actions for Spark and other engines, and later added support for row level deletes through delete files. Delete files made updates cheap to write but added a new reason to compact. Managed services from cloud vendors and data platforms now offer automatic compaction as a feature, since so many teams need it.

How it works

Step 1: Understand the immutable file model

Iceberg data files never change after they are written. An update or delete writes new files and records which old files are replaced. That makes commits safe, and it is also why small files accumulate.

Analogy: a ledger in ink. You never erase a line. You add new lines and note which old lines are void. Over time the ledger gets long and messy. Compaction is the clean copy.

Step 2: Select the files to rewrite

A compaction job looks at the table's current snapshot and picks files that are worth rewriting. Common filters are:

• Files smaller than a threshold, for example less than 75 percent of the target size.
• Files larger than a high threshold, which are split.
• Files with too many associated delete files.
• Only the partitions that changed recently, to save compute.

Step 3: Group files into rewrite units

The job groups selected files, usually inside the same partition, into groups sized for one task. This keeps each task's input within a memory budget.

Step 4: Choose a rewrite strategy

Bin-pack is like packing boxes of clothes tightly. Sort is like packing boxes by category so you can find a shirt without opening every box.

Step 5: Write new files

Tasks read the old files, apply any deletes, and write new, larger files. The old files stay in place while this happens, so readers keep working.

Step 6: Commit the replacement

The job commits one change that removes the old file entries and adds the new ones. The change is atomic through the catalog. See Iceberg cataloging. If another writer touched the same files meanwhile, the commit may fail and be retried, or the group may be skipped.

Step 7: Expire snapshots

The old files are still referenced by older snapshots, which exist so that time travel works. When you expire snapshots older than a retention period, files used only by those snapshots become removable and are deleted.

Step 8: Remove orphan files

Failed jobs may leave files that no snapshot references. A separate orphan removal job lists the storage location and deletes files older than a safety age that no metadata mentions.

Step 9: Rewrite manifests

Manifests list data files. Many small commits create many small manifests, which slows planning. Rewriting manifests regroups them, often by partition, so that the planner reads fewer manifest files.

How delete files fit in

Iceberg format version 2 supports two kinds of row level delete files.

• Position deletes name a file and a row position to remove.
• Equality deletes list column values that identify rows to remove.

Readers must merge these with data files at query time. Many delete files make reads slow. Compaction with the right options applies the deletes and writes clean data files, so later reads need no merging.

Key concepts and terms

A worked example

This example uses Spark SQL procedures. Catalog and table names are placeholders.

Step 1: See the problem

SELECT
  COUNT(*)                         AS file_count,
  ROUND(AVG(file_size_in_bytes)/1048576, 2) AS avg_mb,
  ROUND(MIN(file_size_in_bytes)/1048576, 2) AS min_mb,
  ROUND(MAX(file_size_in_bytes)/1048576, 2) AS max_mb
FROM lake.events.clicks.files;

Suppose the output is 48,000 files with an average of 2 MB. That is a classic small file table, holding about 96 GB in files that should number a few hundred.

Step 2: Compact with bin-pack

CALL lake.system.rewrite_data_files(
  table => 'events.clicks',
  strategy => 'binpack',
  where => 'event_date >= current_date() - 7',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10'
  )
);

The where clause limits work to recent partitions. The partial progress options let the job commit completed groups early.

Step 3: Compact with sorting

CALL lake.system.rewrite_data_files(
  table => 'events.clicks',
  strategy => 'sort',
  sort_order => 'zorder(user_id, event_type)'
);

This lays out rows so that queries filtering on user_id and event_type can skip more files.

Step 4: Apply deletes

CALL lake.system.rewrite_data_files(
  table => 'events.clicks',
  options => map('delete-file-threshold', '2')
);

Files with at least two delete files attached are rewritten, and the deletes are applied.

Step 5: Rewrite manifests

CALL lake.system.rewrite_manifests('events.clicks');

Step 6: Expire snapshots

CALL lake.system.expire_snapshots(
  table => 'events.clicks',
  older_than => TIMESTAMP '2026-01-01 00:00:00',
  retain_last => 10
);

Step 7: Remove orphan files

CALL lake.system.remove_orphan_files(
  table => 'events.clicks',
  older_than => TIMESTAMP '2026-01-01 00:00:00'
);

Use an age that is safely older than any running write job, or you may delete files that are about to be committed.

Step 8: Check the result

Run the first query again. A healthy result might show 200 files with an average near 480 MB. Queries that scanned 48,000 files now open a few hundred, and planning time drops with it.

Numeric view of the gain

Say each file open costs about 30 ms of overhead, spread across 100 parallel tasks.

• Before: 48,000 files times 30 ms is 1,440 seconds of overhead, about 14 seconds across 100 tasks.
• After: 200 files times 30 ms is 6 seconds of overhead, well under a second across the tasks.

The figures are illustrative, but the pattern is real: overhead scales with file count, not with data size.

A scheduling sketch

# Pseudo schedule for a streaming-fed table
jobs = [
    ("every 1 hour",  "rewrite_data_files on last 2 days, binpack"),
    ("every 1 day",   "rewrite_data_files on last 7 days, sort"),
    ("every 1 day",   "rewrite_manifests"),
    ("every 1 day",   "expire_snapshots retain 7 days"),
    ("every 1 week",  "remove_orphan_files older than 3 days"),
]

Strengths and limitations

Strengths

• Faster reads, since fewer files are opened and planned.
• Lower cost in request fees and compute.
• Smaller metadata, so planning is quicker.
• Better data skipping when sort or Z-order is used.
• Safe to run while readers are active, thanks to snapshot isolation.
• Applies deletes so reads need less merge work.
• Frees storage by dropping data that no snapshot needs.

Limitations

• It uses compute and I/O, which costs money.
• It causes write amplification, since data is written again.
• It can conflict with concurrent writers, especially updates to the same files.
• Expiring snapshots removes the ability to time travel to those points.
• Aggressive compaction on hot partitions may fight with streaming writes.
• Sorting and Z-order cost more than bin-pack.
• Poor settings can make things worse, such as target sizes that do not fit the engine.

How it compares with alternatives

A related decision is between compaction in the same engine as the writer and a separate maintenance cluster. A separate cluster keeps maintenance from slowing ingestion, and it is a common pattern with Apache Spark or Amazon EMR.

Role in tabular foundation models

Tabular foundation models learn from raw tables, so the quality and layout of the stored tables matters even though compaction never changes the rows.

Fast, repeatable scans for pre-training

Pre-training on many real tables means reading a lot of Parquet. Compacted files make those reads efficient and predictable. A table of 48,000 tiny files can bottleneck a data loader that must open each one.

Consistent row and column structure

Compaction preserves schema, types and nulls exactly. That matters because a model that learns from mixed types and missing values needs the stored table to match the original. Rewrites do not change nulls to empty strings or change types. If you ever see that happen, it points to a bug in the job, not to normal compaction.

Sorting for sampling and context building

In-context learning models such as TabPFN take a context table of labeled rows. A service that builds those contexts often filters by an entity or a time window. Sort compaction by those columns lets the service fetch the right rows quickly without scanning the whole table.

Continual learning and incremental data

Continual learning updates a model as new data arrives. Streaming-fed tables grow by small commits, which makes compaction essential. A good routine compacts recent partitions often, so a nightly incremental training read sees tidy files, while older partitions are left alone.

Imputation and anomaly detection

Imputation and anomaly checks often scan entire columns to compute statistics. Compacted files with good column statistics make these scans quick. Delete handling also matters: if rows are removed for compliance, applying deletes during compaction ensures downstream jobs never see them. See Anomaly detection.

Snapshots and reproducibility

Snapshot expiry has a direct effect on reproducibility. If a training run depends on snapshot 123, make sure it is tagged or the retention window covers it. Expiring it will remove the files that only that snapshot used. Plan retention with the model team. See Apache Iceberg and Iceberg cataloging.

Fit with the medallion layers

Raw landing tables tend to have the smallest files, and curated tables tend to be larger. Compaction schedules usually differ by layer. See Medallion architecture, Silver layer and Gold layer. For how such models are built and compared, see /model/architecture and /model/benchmarks.

Practical guidance and best practices

Measure before you tune

Query the files, manifests and snapshots metadata tables. Track file count, average size and the share of files under a small threshold. Only compact tables that need it.

Compact the hot edge more often

Streaming data lands in the newest partitions. Compact those every hour or so, and compact older partitions rarely. Use a where filter so that jobs touch only what changed.

Use partial progress for large jobs

Partial progress commits finished groups as it goes. If the job fails near the end, completed work is kept. Set a sensible maximum number of commits so you do not create many snapshots.

Pick the strategy that matches queries

Use bin-pack by default. Use sort when most queries filter on one or two columns. Use Z-order when filters span several columns of similar importance.

Set target file sizes with care

Match the target to your engine and storage. Test a few values on a copy. Keep the write target and the compaction target consistent.

Separate maintenance compute

Run maintenance on its own cluster or job queue. This protects ingestion and query latency.

Handle delete files

If you do row level updates, include delete file thresholds in your compaction plan. Check the number of delete files per data file regularly.

Choose retention with eyes open

Set snapshot retention to match your needs for rollback, audit and reproducible training. Tag snapshots you need to keep.

Be careful with orphan removal

Use an older_than age that is longer than your longest write job. Run it on a schedule, but not at the same time as big ingestion jobs.

Reduce small files at the source

Where possible, increase batch intervals, repartition before writing, or use a distribution mode that groups rows by partition. Compaction is a fix, but fewer small files to start with is cheaper.

Alert on health

Set alerts for average file size dropping, file count rising, and compaction jobs failing or running too long.

Reading the metadata tables

Iceberg exposes its own metadata as queryable tables. These are the best tools for deciding what to compact.

A quick health query by partition:

SELECT
  partition,
  file_count,
  record_count,
  ROUND(total_data_file_size_in_bytes / file_count / 1048576, 1) AS avg_file_mb
FROM lake.events.clicks.partitions
ORDER BY file_count DESC
LIMIT 20;

Partitions with a high file count and a low average size are the first candidates.

A decision guide

1. Is average file size under about a quarter of the target? Run bin-pack on those partitions.
2. Are many queries slow on one filter column? Switch that table to sort compaction.
3. Do data files have many attached delete files? Add a delete file threshold.
4. Is planning slow even though data files look fine? Rewrite manifests.
5. Is storage growing faster than the data? Expire snapshots and remove orphans.
6. Do jobs keep failing on conflicts? Move compaction to partitions that are closed to writes.

Compaction for streaming tables in detail

A streaming job, for example one built with Apache Flink or Spark Streaming, commits at every checkpoint. If the checkpoint interval is one minute, the table gets about 1,440 commits a day. Each commit may write one small file per partition per writer task.

There are three levers.

• Lengthen the checkpoint interval, which makes bigger files but adds latency.
• Reduce writer parallelism or shuffle by partition, so fewer tasks write to each partition.
• Run compaction as a separate job that trails the stream by a safe margin.

Many teams use all three. Compaction targets partitions older than the last hour, so it does not collide with active writers.

Cost thinking

Compaction has a cost, and it should pay for itself. A rough rule: if a partition is read many times per day, compaction pays off quickly. If a partition is rarely read, leave it small until it is read, or compact it once when it is closed.

Testing a compaction plan safely

Before running on production, copy a table sample into a test namespace and run the procedure there. Compare row counts, a checksum of key columns and a few representative query times before and after. Then roll out to one partition, check, and widen.

-- Row count and a simple checksum before and after
SELECT COUNT(*) AS n, SUM(hash(user_id, event_type)) AS chk
FROM lake.events.clicks;

If the numbers differ after a rewrite, stop and investigate. A correct compaction must preserve both.

Rolling back

Because compaction is a normal commit, you can roll back to the snapshot before it, as long as that snapshot has not been expired.

CALL lake.system.rollback_to_snapshot('events.clicks', 4821039847501);

This is another reason to expire snapshots only after you trust the new layout.

Common mistakes

• Compacting the whole table every time rather than recent partitions.
• Running orphan file removal with a very short age, which can delete files of in-flight writes.
• Expiring snapshots needed for a training run or an audit.
• Over-partitioning, which makes small files unavoidable.
• Using Z-order on many columns, which weakens its benefit.
• Letting compaction and a streaming job rewrite the same files, causing repeated conflicts.
• Ignoring delete files, so reads slow down even after data compaction.
• Setting a target file size far larger than what tasks can handle in memory.
• Forgetting manifests, so planning stays slow after data files are fixed.
• Assuming compaction is free. It writes data again and has a cost.

Frequently asked questions

What is Iceberg compaction?

Iceberg compaction is a set of maintenance actions that rewrite many small data files into fewer large ones, apply row deletes and tidy metadata. The data stays the same, but queries run faster and cost less. It is usually done with the rewrite_data_files procedure.

Why do I have so many small files?

Frequent small commits, such as those from streaming jobs, write a few rows at a time into many partitions. Over-partitioning and many parallel writer tasks also create small files. Each commit adds its own files, and Iceberg never edits files in place.

How often should I compact?

It depends on how fast data arrives. A streaming table may need compaction of its newest partitions every hour, while a daily batch table may only need a weekly pass. Watch file count and average file size to decide.

Is it safe to compact while queries are running?

Yes. Readers use a snapshot, and compaction creates new files and swaps them in with one commit. Readers that started earlier keep seeing the old files until their query ends. Do not delete the old files too soon, which is why snapshot expiry is a separate step.

What is the difference between compaction and snapshot expiry?

Compaction rewrites data files into better shapes. Snapshot expiry removes old snapshots and deletes files that only they used. You need both: compaction creates new files, and expiry lets storage shrink again.

Does compaction change my data?

No. The rows, columns and values are the same before and after. Only the physical layout changes. If values change, the job or its settings are wrong.

When should I use sort or Z-order compaction?

Use sort when queries usually filter on one column or a short ordered list of columns. Use Z-order when queries filter on several columns with no single dominant one. Both cost more than bin-pack, so use them where the skipping benefit is clear.

What are delete files and why do they matter?

Delete files record rows to remove without rewriting the data files. They make updates cheap to write, but readers must merge them at query time. Compaction can apply them so that reads become simple again.

Can compaction conflict with streaming writes?

It can, if both touch the same files. Iceberg detects conflicts and retries or fails the commit. To reduce conflicts, compact partitions that are no longer receiving writes, or schedule jobs to avoid overlap.

Do I need compaction for tables that are loaded once a day?

Often only a little. A daily job that writes reasonably large files may never create a small file problem. Still, check file sizes occasionally and expire snapshots on a schedule.

Glossary

• Bin-pack: A strategy that combines files up to a target size without sorting.
• Compaction: Rewriting many small files into fewer large ones.
• Delete file: A file that marks rows as deleted without changing data files.
• Expire snapshots: Removing old snapshots and the files only they reference.
• Manifest: A metadata file listing data files and statistics.
• Orphan file: A file in storage that no table metadata refers to.
• Partial progress: Committing finished rewrite groups early.
• Retention: How long old snapshots are kept.
• Small file problem: Poor performance caused by too many tiny files.
• Sort order: The column ordering used to arrange rows within files.
• Target file size: The desired size of data files.
• Write amplification: Extra writes caused by rewriting existing data.
• Z-order: A way to sort by several columns to cluster related rows.

Summary

Iceberg compaction keeps lakehouse tables healthy. It merges small files, applies deletes, regroups manifests, and, with snapshot expiry and orphan cleanup, removes files that are no longer needed. The rows never change, but scans, planning and costs all improve.

Treat it as routine operations. Measure file counts and sizes, compact the hot edge often, choose bin-pack, sort or Z-order to match your queries, and set retention with reproducibility in mind. For teams that build tabular models, tidy and well-sorted tables mean faster pre-training reads, quicker context building and cleaner incremental updates.

Related

It belongs to Table formats and lakehouse in my stack, under Data engineering and lakehouse. Others in the same group: