# Data storage and formats: object storage, Parquet, Arrow and table formats

Modern data platforms are built in layers: cheap object storage at the bottom, columnar files on top, a table format that adds transactions, and Arrow to move data between engines in memory.

## Overview

Almost every analytics stack today stores data the same way. Files sit in object storage; a columnar file format makes them fast to scan; a table format turns folders of files into reliable tables; and engines exchange results in Arrow's in-memory format. Understanding these four layers explains most of the design choices in big data.

## The four layers

| Layer | Role | Main options | Analogy |
|---|---|---|---|
| Object storage | Durable, cheap, unlimited space | [Amazon S3](/tools/amazon-s3), GCS, Azure Blob, [HDFS](/tools/hadoop-hdfs) | The warehouse building |
| File format | How bytes are laid out | [Parquet](/tools/apache-parquet), ORC, Avro | Boxes sorted by product |
| Table format | Which files make up a table, and when | [Iceberg](/tools/apache-iceberg), [Delta Lake](/tools/delta-lake), [Hudi](/tools/apache-hudi) | The inventory ledger |
| In-memory format | Exchanging data between engines | [Apache Arrow](/tools/apache-arrow) | A standard pallet every forklift can lift |

## Why columnar files win for analytics

Analytical queries read a few columns across many rows. Parquet stores each column together, compressed, with minimum and maximum values for every block, so a query can skip blocks that cannot match and read only the columns it needs. A query for the average price of a billion orders reads only the price column, often a few percent of the file.

## What a table format adds

A folder of files has no transactions: two writers can collide, a reader can see half a write, and changing a column means rewriting everything. A table format keeps metadata that lists the exact files in each version of the table. That gives atomic commits, safe concurrent writes, schema and partition changes without rewriting data ([schema evolution](/tools/schema-evolution)), and time travel to earlier versions. Iceberg v3, now generally available on major platforms, adds deletion vectors, row lineage and a variant type for semi-structured data.

## Choosing a table format

Iceberg is the most engine-neutral and is supported by Spark, Trino, Flink, Snowflake, BigQuery and DuckDB. Delta Lake is native to Databricks and widely supported elsewhere. Hudi is strongest for frequent record-level updates and streaming ingestion. Many teams now pick Iceberg for new lakes because of the breadth of engines and catalogs that read it; see [Iceberg cataloging](/tools/iceberg-cataloging).

## Keeping tables healthy

Streaming and frequent small writes create many small files, which slow every query. Schedule [compaction](/tools/iceberg-compaction), expire old snapshots, and partition by columns that queries filter on, such as date. A [data catalog](/tools/data-catalog) records owners and schemas so tables stay discoverable.

## Frequently asked questions

### Is Parquet a database?

No. It is a file format. A table format and an engine together give you database-like behaviour on top of it.

### Do I still need a data warehouse?

Many teams run warehouse workloads directly on lakehouse tables; others keep a warehouse such as [Snowflake](/tools/snowflake) or [BigQuery](/tools/bigquery) that reads the same Iceberg tables.

## Summary

Object storage for space, Parquet for fast scans, a table format for transactions and history, and Arrow for moving data between engines: these four layers are the foundation of the modern lakehouse.

## Back to the topic

This page is part of **[Big data](/topics/big-data)**, which maps every concept, tool, simulation and article on the subject.
