# Apache Parquet

Apache Parquet is an open columnar file format with efficient compression and encoding, the standard storage format for analytics, lakehouses and ML data.

## Overview

Apache Parquet is an open-source file format for storing tables on disk. Instead of writing one row after another, as CSV does, it writes the values of each column together. Analytical queries usually touch a few columns across many rows, such as "total revenue by region". With Parquet a reader can fetch just the revenue and region columns and ignore the rest.

Storing columns together also compresses much better. Values in one column share a type and often repeat, so encodings such as dictionaries and run lengths shrink them a lot. Parquet files carry their own schema and statistics, which lets engines skip whole chunks of data that cannot match a filter. These properties made Parquet the common storage layer of data lakes, [lakehouse](/tools/lakehouse) table formats and machine learning data sets.

Picture a school register. A row layout is a card for each pupil with every detail on it. To find the average age you must pick up every card. A columnar layout is one sheet per question: one sheet of names, one of ages, one of postcodes. To find the average age you read a single sheet. If the ages sheet also notes "youngest 11, oldest 12" at the top, you can tell at a glance that it holds no 16-year-olds.

## What it is

A Parquet file has a nested structure.

| Element | Role |
|---|---|
| File | Starts and ends with the magic bytes `PAR1`; holds one or more row groups |
| Row group | A horizontal slice of the table, a set of rows |
| Column chunk | The data for one column inside one row group |
| Page | The smallest unit, holding encoded and compressed values; data pages and dictionary pages |
| Footer | File metadata: schema, row group locations, column statistics and encodings |
| Statistics | Minimum, maximum and null counts used to skip data |
| Page index | Optional per-page statistics and offsets for finer skipping |
| Bloom filter | Optional structure to test whether a value may be present in a column chunk |
| Logical types | Annotations such as string, date, timestamp, decimal on top of a few physical types |

Parquet is a file format only. It does not handle transactions, concurrent writers or updates to individual rows. Table formats such as [Apache Iceberg](/tools/apache-iceberg), [Delta Lake](/tools/delta-lake) and [Apache Hudi](/tools/apache-hudi) add those features, and they nearly always store their data in Parquet.

## Where it came from

Parquet grew out of a joint effort by Twitter and Cloudera. It improved on Trevni, an earlier columnar format created by Doug Cutting, the creator of Hadoop. Its handling of nested data uses the record shredding and assembly approach described in Google's Dremel paper. Parquet 1.0 was released in July 2013. The project joined the Apache Incubator and became a top-level Apache Software Foundation project on 27 April 2015.

Since then the format has added features while keeping old files readable: more encodings, logical types for dates, times and decimals, page indexes, bloom filters and more compression codecs. Today implementations exist in Java, C++ (as part of [Apache Arrow](/tools/apache-arrow)), Rust, Go and other languages, and nearly every analytical engine reads it.

## How it works

### Writing

1. The writer buffers rows until it has enough for a row group.
2. For each column in the group it splits values into pages.
3. Each page is encoded. Dictionary encoding replaces repeated values with small integer codes. A hybrid of run-length encoding and bit packing stores those codes, and repeated values, compactly. Delta encodings suit sorted numbers and timestamps.
4. The encoded page is compressed with a codec such as Snappy, gzip, Zstandard, LZ4 or Brotli.
5. The writer records statistics for each column chunk, and optionally page indexes and bloom filters.
6. When the file closes, the writer appends the footer, which describes the schema and where every row group and column chunk lives.

### Reading

1. The reader fetches the footer from the end of the file. One small read reveals the whole layout.
2. It keeps only the columns the query needs. This is column pruning, or projection.
3. It compares filters with the statistics. If a query asks for `amount > 1000` and a row group's maximum is 800, the whole group is skipped. This is predicate pushdown.
4. It reads and decompresses only the remaining pages, then decodes them into memory, often straight into Arrow arrays.

### Nested data

Parquet stores nested structures, such as lists and structs, as flat columns. Two small numbers per value, the repetition level and the definition level, record where lists start and which optional fields are null. This lets the reader rebuild the original nesting exactly.

### Sizing

The Parquet documentation recommends large row groups, 512 MB to 1 GB, and data pages of about 8 KB. Many writers use smaller defaults, and engines differ in their preferences, so check the guidance for the engine that reads your files.

```python
import pyarrow.parquet as pq
pq.write_table(table, "sales.parquet", compression="zstd")
t = pq.read_table("sales.parquet", columns=["region", "amount"],
                  filters=[("region", "=", "EU")])
```

## Key concepts and terms

| Term | Meaning |
|---|---|
| Columnar storage | Storing each column's values together |
| Row group | Horizontal chunk of rows within a file |
| Column chunk | One column's data within a row group |
| Page | Unit of encoding and compression |
| Footer | Metadata block at the end of the file |
| Dictionary encoding | Replacing repeated values with integer codes |
| Run-length encoding | Storing a value once with a repeat count |
| Predicate pushdown | Skipping data using statistics and filters |
| Column pruning | Reading only the needed columns |
| Repetition and definition levels | Numbers that encode nesting and nulls |
| Logical type | Meaning layered on a physical type, such as a date stored as an integer |

## A worked example

This example is illustrative and the numbers are typical rather than guaranteed. A retailer exports a year of transactions as CSV: 2 billion rows, 18 columns and 500 GB.

1. The team converts it to Parquet with Zstandard compression, writing one file per day and sorting each day by `store_id`. Dictionary encoding works well on columns such as `region`, `payment_type` and `store_id`. The result is about 70 GB, a seven-fold reduction.
2. A finance query asks for monthly totals by region. It needs three columns: `date`, `region` and `amount`. The engine reads only those column chunks, roughly a sixth of the file data.
3. A second query filters on one store. Because each file is sorted by `store_id`, row group statistics show narrow ranges, and most row groups are skipped without being read.
4. The same files are read by a Python notebook with pandas, a Spark job and [DuckDB](/tools/duckdb) on a laptop, with no conversion.

## Strengths and limitations

**Strengths:**

- Reads only the columns and row groups a query needs.
- Strong compression thanks to per-column encodings.
- Self-describing: schema and statistics travel with the data.
- Supports nested data and rich logical types.
- Open, stable and supported by almost every data tool.

**Limitations:**

- Not designed for frequent single-row updates or deletes; files are written once.
- Slower than row formats for writing small records one at a time.
- Many tiny files hurt performance; files need batching or compaction.
- Not human-readable, unlike CSV or JSON.
- Benefits depend on good layout choices such as sorting and row group size.

## How it compares with alternatives

| Format | Layout | Best for |
|---|---|---|
| Apache Parquet | Columnar, compressed, on disk | Analytics, lakes and ML data sets |
| ORC | Columnar, on disk, close ties to Hive | Hive-centric warehouses |
| Avro | Row-based, with schema | Streaming records and message payloads, often with a [schema registry](/tools/schema-registry) |
| CSV and JSON | Row-based text | Simple exchange and human inspection |
| [Apache Arrow](/tools/apache-arrow) | Columnar, in memory | Fast processing and exchange between tools |

## Practical guidance and best practices

1. Convert analytical data from CSV or JSON to Parquet early in the pipeline.
2. Aim for files of at least tens of megabytes, and preferably hundreds; compact small files.
3. Sort or cluster data by columns that queries often filter on, so statistics become selective.
4. Choose a codec deliberately: Snappy for speed, Zstandard for a good balance of size and speed.
5. Keep schemas consistent across files and evolve them carefully; use a table format to manage changes.
6. Use proper logical types for dates, timestamps and decimals rather than strings.
7. Enable bloom filters on high-cardinality columns you look up by exact value, if your reader uses them.
8. Use a table format on top of Parquet when you need updates, deletes or transactions.

## Common mistakes

- Writing thousands of tiny files from a streaming job or a highly parallel write.
- Storing numbers and dates as strings, which defeats encodings and statistics.
- Never sorting data, so every row group spans every value and nothing can be skipped.
- Partitioning folders by a high-cardinality column, multiplying small files.
- Assuming two Parquet files with the same column names have compatible types.
- Rewriting Parquet files in place to change single rows instead of using a table format.

## Frequently asked questions

### Why is Parquet faster than CSV?

It reads only the columns you need, its data is compressed and encoded by type, and its statistics let engines skip blocks that cannot match a filter. CSV must be read and parsed in full, as text, for every query.

### What is the difference between Parquet and Arrow?

Parquet is a storage format, optimised to be small on disk and cheap to scan. Arrow is an in-memory format, optimised for fast processing and zero-copy sharing. A common pattern is to store data in Parquet and read it into Arrow for computation.

### Can I update a row in a Parquet file?

Not directly. Parquet files are immutable once written. To change data you write new files. Table formats such as Iceberg, Delta Lake and Hudi manage this, tracking which files are current and handling deletes.

### Which compression codec should I use?

Snappy is fast and widely used. Zstandard usually gives smaller files at a modest CPU cost and is a good default for many workloads. Gzip compresses well but is slower. Check what your readers support.

### How big should a Parquet file be?

Avoid very small files. The format's own documentation suggests large row groups, around 512 MB to 1 GB, and many teams target files in the range of 100 MB to 1 GB. The right size depends on your engine and query patterns.

### Is Parquet good for machine learning data?

Yes. It stores large feature tables compactly, keeps types and supports reading selected columns, and libraries such as pandas, Polars, PyArrow, Spark and many training data loaders read it.

## Glossary

- Codec: a compression algorithm such as Snappy or Zstandard.
- Dremel: Google's interactive query system whose paper inspired Parquet's nested encoding.
- Magic bytes: the `PAR1` marker at the start and end of a file.
- Physical type: the storage type, such as INT32, INT64, DOUBLE or BYTE_ARRAY.
- Projection: selecting only certain columns.
- Schema evolution: changing a schema over time while keeping old files readable.
- Trevni: an earlier columnar format that Parquet improved on.
- Small-file problem: the overhead of opening and planning very many tiny files.

## Summary

Apache Parquet stores tables column by column, encodes and compresses each column, and records statistics that let readers skip what they do not need. Its file structure of row groups, column chunks, pages and a footer makes scans cheap and files compact. It does not handle updates or transactions, which is why table formats sit on top of it, and it pairs naturally with [Apache Arrow](/tools/apache-arrow) for in-memory work. Choosing sensible file sizes, sorting by filter columns and using real types are the habits that unlock most of its benefits.