Lakehouse architecture

Overview

A lakehouse is a data architecture that combines the low-cost, flexible storage of a data lake with the reliability and performance features of a data warehouse. Data sits in open file formats, usually Apache Parquet, in cloud object storage. A table format layered on top, such as Apache Iceberg, Delta Lake or Apache Hudi, adds what plain files lack: atomic transactions, schemas that can evolve safely, snapshots for time travel and metadata that lets engines skip data. Many engines can then read and write the same tables, from SQL warehouses to Spark jobs and Python notebooks.

The idea grew from a common pain. Organisations kept two systems. A data lake held raw and semi-structured data cheaply for data science and machine learning, but it was hard to keep consistent and slow for business reporting. A data warehouse gave clean tables and fast SQL, but it was more expensive, held data in proprietary storage and needed copies of data from the lake. Keeping the two in sync caused delays, extra cost and arguments about which numbers were right. A lakehouse aims to serve both kinds of work from one copy of the data.

An everyday analogy: a lake is a large, cheap storage unit where you throw boxes; a warehouse is a tidy shop where everything is labelled but you pay for every shelf. A lakehouse is the storage unit with a proper inventory system, locks on the doors and an index at the front desk, so it can be run like the shop without moving the boxes.

What it is

A lakehouse is a set of layers rather than one product.

The key point is that storage and compute are separate and the formats are open. Several engines can work on the same tables, and you can change engines without migrating the data.

Where it came from

Data warehouses date back decades. Data lakes became popular with Hadoop in the late 2000s and moved to cloud object storage in the 2010s. The weaknesses of plain lakes, such as no transactions, slow file listing and painful updates, led to new open table formats. Hudi came from Uber, Iceberg from Netflix and Delta Lake from Databricks, which open-sourced it in 2019. Each one added a metadata layer that tracks exactly which files form a table at each version.

The word "lakehouse" was popularised by Databricks around 2020. A 2021 paper at the CIDR conference by Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia, titled "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics", set out the architecture and argued that it would replace the two-tier lake-plus-warehouse design. Since then, most major data platforms and cloud warehouses have added support for open table formats, especially Iceberg, and open catalogue standards such as the Iceberg REST catalogue protocol have spread.

How it works

Writing data

1. An engine prepares new data, for example a Spark job processing yesterday's orders.
2. It writes new Parquet files to object storage. These files are not yet part of the table.
3. It writes new table metadata that lists the table's files for the new version, together with statistics such as column minimums and maximums.
4. It asks the catalogue to switch the table's pointer from the old metadata to the new one in a single atomic operation. If another writer committed first, the operation fails and the writer retries against the new state.
5. Readers that started earlier keep seeing the old version; new readers see the new one. Nobody sees a half-written table.

Reading data

An engine asks the catalogue for the table's current metadata, reads the list of files and their statistics, prunes files and partitions that cannot match the query, then reads only the needed columns from the remaining Parquet files. Because the file list comes from metadata, the engine does not need to list directories in object storage, which is slow at scale.

Updates, deletes and evolution

Table formats support row-level changes. Copy-on-write rewrites affected files. Merge-on-read writes delete files or change logs and merges them when reading. Schema evolution lets you add, rename or drop columns through metadata. Some formats also let you change the partition scheme without rewriting old data.

Maintenance

Lakehouse tables need regular housekeeping: compacting small files into larger ones, expiring old snapshots, removing orphaned files and sometimes re-sorting data for better skipping. See Iceberg compaction for one example.

Organising data

Many teams organise lakehouse tables in layers, often called the medallion architecture: a bronze layer of raw data, a silver layer of cleaned and conformed data and a gold layer of business-ready tables.

Key concepts and terms

A worked example

This scenario is illustrative. A mid-sized insurer runs a lake on object storage for claims documents and event logs, and a separate warehouse for finance reports. Each night, 200 GB is copied from lake to warehouse, and the two often disagree.

1. The team chooses Iceberg tables with a REST catalogue. Claims events land in bronze tables as they arrive from change data capture streams.
2. Spark jobs clean and deduplicate them into silver tables every hour, using MERGE to apply updates by claim ID.
3. Gold tables of daily claims and reserves are built with dbt on a SQL engine reading the same silver tables.
4. Finance analysts query gold tables from their BI tool through a warehouse engine that reads Iceberg directly. The nightly copy is retired.
5. Data scientists read silver tables from notebooks with DuckDB and Spark to train a fraud model, using time travel to reproduce the exact training data from a given snapshot.
6. Nightly maintenance compacts small files from streaming writes and expires snapshots older than 14 days.
7. Access rules in the catalogue hide personal fields from most users, and the same rules apply whichever engine they use.

Strengths and limitations

Strengths:

• One copy of data serves BI, data engineering, data science and ML.
• Open formats reduce lock-in and allow several engines.
• Cheap object storage scales to very large volumes.
• Transactions, time travel and schema evolution make lakes reliable.
• Storage and compute scale independently.

Limitations:

• More moving parts than a single managed warehouse: formats, catalogues, engines and maintenance.
• Performance for highly concurrent, low-latency BI may need extra tuning or caching.
• Table maintenance must be planned and monitored.
• Interoperability between engines varies by format version and feature.
• Governance across many engines depends on a strong, shared catalogue.

How it compares with alternatives

Practical guidance and best practices

1. Pick one primary table format for the organisation and check that every engine you need supports the features you plan to use.
2. Choose a catalogue early; it becomes the centre of access control and interoperability.
3. Keep data in Parquet with sensible file sizes and sort or cluster by common filters.
4. Schedule compaction, snapshot expiry and orphan-file cleanup from the start.
5. Use layered tables, such as bronze, silver and gold, with clear ownership and quality checks.
6. Govern access in the catalogue rather than in each engine separately.
7. Use time travel and snapshot tags to make ML training data and audits reproducible.
8. Migrate incrementally: move one domain or workload at a time and compare results with the old system.

Common mistakes

• Treating a lakehouse as just "Parquet in a bucket" without a table format or catalogue.
• Letting streaming writes create millions of small files with no compaction.
• Running several catalogues that disagree about which metadata is current.
• Expecting every engine to support every feature of every format version.
• Granting broad bucket-level access, bypassing the catalogue's permissions.
• Rebuilding a warehouse's complexity on top of the lake without clear data ownership.

Frequently asked questions

Is a lakehouse a product or an architecture?

It is an architecture. Vendors sell products built on it, but the core idea, open files plus a table format, a catalogue and multiple engines, can be assembled from open-source parts or managed services.

Which table format should I choose?

Iceberg is widely supported across engines and vendors. Delta Lake is tightly integrated with Spark and Databricks. Hudi is strong for frequent upserts and incremental processing. Check your engines, cloud services and update patterns, and avoid mixing formats without a reason.

Does a lakehouse replace a data warehouse?

It can replace much of a warehouse's role, and many warehouses now read and write open table formats themselves, which blurs the line. Some organisations keep a warehouse for demanding BI workloads and use the lakehouse for everything else.

Is a lakehouse good for machine learning?

Yes. ML tools can read Parquet and table formats directly, data scientists work on the same governed data as analysts, and time travel helps reproduce training sets. Feature pipelines often build on silver and gold tables.

Can a lakehouse handle streaming data?

Yes. Engines such as Flink and Spark can write to table formats continuously, and some formats support incremental reads. Frequent small commits need compaction to keep reads fast.

What role does the catalogue play?

It names tables, points engines to each table's current metadata and performs the atomic switch that makes commits safe. Modern catalogues also handle access control, credentials and auditing across engines.

Glossary

• Medallion architecture: organising tables into bronze, silver and gold layers.
• Object storage: storage of files as objects accessed over an API.
• Orphan files: data files no longer referenced by any table version.
• REST catalogue: a catalogue that exposes table operations through a standard HTTP API.
• Snapshot expiry: removing old table versions and their unused files.
• Two-tier architecture: a separate data lake and data warehouse.
• Copy-on-write: applying updates by rewriting affected files.
• Merge-on-read: recording changes separately and combining them at read time.

Summary

A lakehouse stores data once, in open formats on cheap object storage, and adds a table format and catalogue that give it warehouse qualities: transactions, schemas, snapshots and fast, selective reads. Many engines can then share the same tables for BI, data engineering, streaming and machine learning. The price is more components and ongoing maintenance, and engine support for each format's features varies. Formats such as Apache Iceberg, Delta Lake and Apache Hudi, organised with the medallion architecture, are the usual building blocks.

Related

It belongs to Architecture in my stack, under Big data. Others in the same group: