Apache Hudi

Overview

Apache Hudi is an open-source table format and data lake platform. Its name stands for Hadoop Upserts Deletes and Incrementals, which describes what it adds to plain files in a data lake. With Hudi you can update or delete individual records, commit changes as atomic transactions and ask "what changed since yesterday at noon?" without rescanning the whole table.

Plain data lakes store immutable files such as Apache Parquet. That works well for appending new data but badly for change. If one customer's address changes, you must find and rewrite the files that hold that customer. Hudi does that bookkeeping for you. It keeps an index from record keys to files, a timeline of every commit and background services that tidy the files up.

An analogy: a plain data lake is a bound ledger, where correcting one entry means copying out the whole page. Hudi is a ledger with a sticky-note system. Corrections go on notes stuck to the page, a clerk periodically rewrites pages with the notes folded in, and a dated logbook records every change so you can read just the latest corrections.

What it is

Hudi is more than a file layout. It ships writers for Apache Spark and Apache Flink, an ingestion tool called Hudi Streamer and table services. Query engines such as Spark, Flink, Hive, Presto and Trino can read Hudi tables.

Where it came from

Hudi was developed at Uber in 2016, under the code name "Hoodie". Uber needed fresh, queryable copies of fast-changing operational data, such as trips, in its Hadoop data lake, without rewriting huge tables every few hours. Uber open-sourced the project in 2017 and submitted it to the Apache Incubator in January 2019. The Apache Software Foundation announced Hudi as a top-level project on 4 June 2020.

Hudi 1.0 arrived in December 2024. It introduced a timeline stored as an LSM tree so long histories stay efficient, secondary indexes on non-key columns, expression indexes, partial updates on merge-on-read tables and non-blocking concurrency control for several streaming writers. It also moved to table format version 8, while still able to write the older version 6 format for compatibility.

How it works

Writing a batch of changes

1. A writer receives a batch of records, for example change events from a database.
2. If the batch contains several versions of one key, Hudi keeps the one with the highest value in an ordering field (traditionally called the precombine field), such as an update timestamp.
3. The index looks up each record key. Keys already present are updates and are routed to their existing file group. New keys are inserts and go to new or under-sized file groups.
4. Hudi writes new files. What it writes depends on the table type, described below.
5. The writer adds a completed instant to the timeline. Until then, readers do not see the change, so commits are atomic.

Copy-on-write versus merge-on-read

In a copy-on-write table, an update rewrites the whole base file that contains the record, producing a new file slice. Reads are simple and fast because every file is already a clean Parquet file. Writes cost more because changing one row rewrites a file.

In a merge-on-read table, updates are appended to log files beside the base file. Writes are cheap and fast. Readers who want the newest data merge base and log files on the fly. A compaction service later merges logs into a new base file. MOR tables suit frequent, small updates such as streaming change data capture.

Reading

Hudi offers several query types. A snapshot query returns the latest committed state. A read-optimised query on a MOR table reads only base files, which is faster but may lag behind. An incremental query returns only records changed after a given commit time. Time travel reads the table as it was at an earlier instant.

Keeping the table healthy

Cleaning deletes old file slices that no reader needs any more. Compaction merges log files into base files for MOR tables. Clustering rewrites data to group related records, for example by sorting on a filter column, and to fix small files. These services can run inline with writes or asynchronously.

Key concepts and terms

A worked example

This scenario is illustrative. A delivery company keeps an orders table of 400 million rows in object storage, partitioned by order date. Its operational database produces about 2 million changes an hour, mostly status updates to orders from the last three days.

1. A change data capture tool publishes database changes to Apache Kafka.
2. Every five minutes a Hudi Streamer job reads about 170,000 new events, de-duplicates them by order_id using updated_at as the ordering field and upserts them.
3. The table is merge-on-read. Each update lands in a small log file, so a write touches only recent partitions and completes in under a minute.
4. Asynchronous compaction runs every hour and folds logs into fresh Parquet base files. Cleaning keeps the last 24 hours of versions so long-running readers are not broken.
5. Analysts run snapshot queries in Trino for near-real-time dashboards. A downstream job runs an incremental query each hour, reads only the orders changed since its last run and updates a daily revenue table.
6. When a customer asks to be forgotten, the team issues a delete by key, and the next compaction removes the record from base files.

Strengths and limitations

Strengths:

• Strong record-level upserts and deletes, driven by keys and indexes.
• Incremental queries make change-driven pipelines cheap.
• Choice of copy-on-write or merge-on-read to balance read and write cost.
• Built-in table services for compaction, clustering and cleaning.
• Ingestion tooling for streams and database changes.

Limitations:

• Many configuration options; good results need tuning of indexes, file sizes and services.
• Merge-on-read reads are slower until compaction runs.
• Table services consume compute and must be scheduled and monitored.
• Engine support for every feature varies, so check the reader you plan to use.
• Mixing writer versions across a table upgrade needs care; read the release notes.

How it compares with alternatives

Practical guidance and best practices

1. Choose a stable, unique record key and a reliable ordering field before the first write.
2. Use copy-on-write for read-heavy tables with infrequent updates and merge-on-read for frequent, small updates.
3. Partition by a coarse field such as date; too many partitions create small files.
4. Pick an index type that matches your update pattern; random updates across the whole table benefit from a global or record-level index.
5. Run compaction and cleaning on a schedule and monitor how many log files build up.
6. Use clustering to sort data by common filter columns and fix small files.
7. Use incremental queries for downstream jobs instead of full rescans.
8. Test upgrades between major versions on a copy of the table first.

Common mistakes

• Using a non-unique record key, so different records overwrite each other.
• Forgetting the ordering field, so an older event wins over a newer one.
• Leaving compaction off on merge-on-read tables until queries slow down.
• Setting cleaning so aggressively that long-running queries lose the files they were reading.
• Treating Hudi as a database for single-row, millisecond lookups.
• Copying configuration from a tutorial without matching it to data volume and update patterns.

Frequently asked questions

What problem does Hudi solve?

It makes data lake tables mutable and incremental. You can upsert and delete individual records with atomic commits, and downstream jobs can read only the changes since their last run instead of rescanning everything.

Should I use copy-on-write or merge-on-read?

Copy-on-write gives the simplest and fastest reads but more expensive writes. Merge-on-read gives fast writes and fresher data but needs compaction and slightly slower reads until compaction runs. Read-heavy reporting tables often use copy-on-write; streaming CDC tables often use merge-on-read.

Can Spark, Trino and Flink all use the same Hudi table?

Yes, Hudi has integrations for these and other engines. Feature support differs between engines and versions, so check that your reader supports the query type you need, such as incremental or snapshot queries on merge-on-read tables.

How is Hudi different from Iceberg and Delta Lake?

All three add transactions and table metadata on top of files. Hudi puts particular weight on record-level indexing, upserts, incremental processing and built-in table services. Iceberg emphasises engine neutrality and schema and partition evolution. Delta Lake is closely tied to Spark. Each has added many of the others' features over time.

Does Hudi replace a database?

No. Hudi is for analytical tables on lake storage. Operational systems that need many small, fast transactions should stay in a database, with changes streamed into Hudi.

Can I query old versions of a table?

Yes, through time travel queries, as long as the cleaner has not yet removed the files for that version.

Glossary

• CDC: change data capture, streaming inserts, updates and deletes from a database.
• Hudi Streamer: Hudi's ingestion utility, formerly called DeltaStreamer.
• Ordering field: column used to choose the newest version of a record.
• Read-optimised query: a query on a MOR table that reads only base files.
• Savepoint: a marked instant that cleaning will not remove, for recovery.
• Snapshot query: a query that returns the latest committed data.
• Time travel: reading a table as of an earlier commit.
• Small-file problem: performance loss from very many tiny files.

Summary

Apache Hudi turns static data lake files into tables that can change. It routes each record to a file group through an index, writes changes as atomic commits on a timeline and, with merge-on-read, absorbs frequent updates cheaply before compaction folds them in. Incremental queries let downstream pipelines process only what changed. Hudi needs more tuning than plain files, and it competes closely with Apache Iceberg and Delta Lake, but it is a strong choice for change data capture and near-real-time lakehouse ingestion.

Related

It belongs to Storage and formats in my stack, under Big data. Others in the same group: