Özer Cem KELAHMETKELAHMET
Topics · Big data

Data storage and formats: object storage, Parquet, Arrow and table formats

Modern data platforms are built in layers: cheap object storage at the bottom, columnar files on top, a table format that adds transactions, and Arrow to move data between engines in memory.

Overview

Almost every analytics stack today stores data the same way. Files sit in object storage; a columnar file format makes them fast to scan; a table format turns folders of files into reliable tables; and engines exchange results in Arrow's in-memory format. Understanding these four layers explains most of the design choices in big data.

The four layers

Layer Role Main options Analogy
Object storage Durable, cheap, unlimited space Amazon S3, GCS, Azure Blob, HDFS The warehouse building
File format How bytes are laid out Parquet, ORC, Avro Boxes sorted by product
Table format Which files make up a table, and when Iceberg, Delta Lake, Hudi The inventory ledger
In-memory format Exchanging data between engines Apache Arrow A standard pallet every forklift can lift

Why columnar files win for analytics

Analytical queries read a few columns across many rows. Parquet stores each column together, compressed, with minimum and maximum values for every block, so a query can skip blocks that cannot match and read only the columns it needs. A query for the average price of a billion orders reads only the price column, often a few percent of the file.

What a table format adds

A folder of files has no transactions: two writers can collide, a reader can see half a write, and changing a column means rewriting everything. A table format keeps metadata that lists the exact files in each version of the table. That gives atomic commits, safe concurrent writes, schema and partition changes without rewriting data (schema evolution), and time travel to earlier versions. Iceberg v3, now generally available on major platforms, adds deletion vectors, row lineage and a variant type for semi-structured data.

Choosing a table format

Iceberg is the most engine-neutral and is supported by Spark, Trino, Flink, Snowflake, BigQuery and DuckDB. Delta Lake is native to Databricks and widely supported elsewhere. Hudi is strongest for frequent record-level updates and streaming ingestion. Many teams now pick Iceberg for new lakes because of the breadth of engines and catalogs that read it; see Iceberg cataloging.

Keeping tables healthy

Streaming and frequent small writes create many small files, which slow every query. Schedule compaction, expire old snapshots, and partition by columns that queries filter on, such as date. A data catalog records owners and schemas so tables stay discoverable.

Frequently asked questions

Is Parquet a database?

No. It is a file format. A table format and an engine together give you database-like behaviour on top of it.

Do I still need a data warehouse?

Many teams run warehouse workloads directly on lakehouse tables; others keep a warehouse such as Snowflake or BigQuery that reads the same Iceberg tables.

Summary

Object storage for space, Parquet for fast scans, a table format for transactions and history, and Arrow for moving data between engines: these four layers are the foundation of the modern lakehouse.

Back to the topic

This page is part of Big data, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open