# Apache Arrow

Apache Arrow defines a standard in-memory columnar format and tools such as Arrow Flight and IPC for fast, zero-copy data exchange between systems and languages.

## Overview

Apache Arrow is an open standard for laying out tabular data in memory, column by column, together with libraries that implement it in many programming languages. When two tools agree on exactly how a column of integers or strings sits in memory, one can hand data to the other without converting it. No parsing, no copying, no re-encoding.

Before Arrow, every pair of systems needed its own converter. Moving data from a database to Python, from Python to Spark or from Spark to a machine learning library often spent more time serialising and deserialising than computing. Arrow replaced many of those converters with one shared format. Around that format the project also offers an inter-process format (IPC), a network protocol for bulk data (Arrow Flight), a database API (ADBC) and compute libraries.

Think of shipping containers. Once every port, ship, train and lorry agreed on one container size, goods could move from one to another without being unpacked and repacked. Arrow is that container for data in memory. Tools such as pandas, [Polars](/tools/polars), [DuckDB](/tools/duckdb) and [Apache Spark](/tools/apache-spark) can pass Arrow data between themselves the way a crane lifts a container from ship to train.

## What it is

| Component | Purpose |
|---|---|
| Columnar format | Specification of the memory layout for each data type |
| Schema and record batch | A schema describes columns; a record batch is a set of equal-length columns |
| C Data Interface | Small, stable C structs for sharing Arrow data inside one process, across languages |
| IPC format | Streaming and file formats for sending Arrow data between processes or saving it |
| Arrow Flight | High-performance RPC framework for moving Arrow data over a network |
| Flight SQL | Protocol on top of Flight for talking to SQL databases |
| ADBC | Arrow Database Connectivity, a database API that returns results in Arrow form |
| Compute functions | Vectorised operations such as filters, arithmetic and aggregates |
| Language libraries | Official implementations in C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby and Rust |

Arrow is not a database or a query engine on its own, and it is not mainly a storage format. It is the common memory layer that engines, libraries and services build on.

## Where it came from

The Apache Arrow project was established in February 2016, with its first commit on 5 February. It was a joint effort by developers from several open-source data projects, including founders of [Apache Parquet](/tools/apache-parquet), and Wes McKinney, the creator of pandas, was among its early leaders. The first release, 0.1.0, was tagged in October 2016.

Version 1.0.0, released in July 2020, marked the point where the columnar format received formal stability and compatibility promises. Since then the project has grown subprojects and specifications, including Flight, Flight SQL, ADBC and nanoarrow, a small C implementation for embedding. DataFusion, a Rust query engine that started inside Arrow, became its own top-level Apache project in 2024. The project celebrated its tenth anniversary in February 2026.

## How it works

### The memory layout

Each column is an array. An array is made of a few contiguous buffers.

- A validity bitmap marks which values are null, one bit per value.
- For fixed-width types such as 64-bit integers, a data buffer holds the values side by side.
- For variable-width types such as strings, an offsets buffer says where each value starts and a data buffer holds all the bytes back to back.
- Nested types such as lists and structs are built from child arrays.

Because values of one column are contiguous, modern CPUs can process them with vectorised (SIMD) instructions and good cache use. Because the layout is fixed by the specification, any library that understands Arrow can read these buffers as they are.

### Sharing within a process

Inside one process, tools exchange pointers, not data. The C Data Interface defines two small structs that describe a schema and an array. A C++ library can hand an array to Python or Rust by passing these structs. The receiver reads the same memory. This is what "zero-copy" means.

### Sharing between processes and machines

The IPC format writes a schema message followed by record batch messages. The body of each message is the Arrow buffers themselves, so writing and reading costs little beyond moving bytes. The IPC file format (also known as Feather version 2) adds a footer for random access. Data can even be memory-mapped from disk.

Arrow Flight sends these messages over gRPC. A client asks a server for a data set, and the server streams record batches back, possibly from several endpoints in parallel. Flight SQL adds standard calls for running queries and fetching metadata. ADBC gives applications a database API, like JDBC or ODBC, that returns Arrow data directly instead of row-by-row values.

```python
import pyarrow as pa
t = pa.table({"id": [1, 2, 3], "city": ["Leeds", None, "York"]})
print(t.schema)          # id: int64, city: string
print(t.column("city").null_count)   # 1
```

## Key concepts and terms

| Term | Meaning |
|---|---|
| Array | One column of values with its buffers |
| Buffer | A contiguous block of memory |
| Validity bitmap | One bit per value marking null or not null |
| Offsets buffer | Start positions of variable-length values |
| Record batch | A set of equal-length arrays sharing a schema |
| Table | A logical collection of record batches |
| Zero-copy | Sharing data by reference instead of copying it |
| IPC | Arrow's message format for moving data between processes |
| Flight | Arrow's RPC framework for bulk data transfer |
| ADBC | Arrow-native database connectivity API |
| Dictionary encoding | Storing repeated values once and referring to them by index |

## A worked example

This scenario is illustrative. A team runs a feature service that sends batches of 200,000 rows and 60 numeric columns to a model-scoring service.

1. In the first version, the feature service returns JSON. Each batch is about 180 MB of text. Encoding and parsing dominate the request time, and the scoring service briefly holds both the text and the parsed objects in memory.
2. The team switches to Arrow IPC. The feature service builds a record batch and writes it as an IPC stream. The payload is about the size of the raw numbers, roughly 96 MB for 64-bit values, and it is written with little more than a memory copy.
3. The scoring service reads the stream into Arrow arrays and hands them to its model library through the C Data Interface without another copy.
4. Later they expose the feature store through Arrow Flight, so offline jobs in Spark and notebooks in Polars fetch the same data at high throughput without bespoke exporters.

## Strengths and limitations

**Strengths:**

- Zero-copy sharing between libraries and languages.
- Layout designed for vectorised CPU processing.
- Broad support across languages and data tools.
- One format for in-process, inter-process and network exchange.
- Stable format with compatibility guarantees since 1.0.

**Limitations:**

- Uncompressed in-memory data can be large; it is not a substitute for compact storage.
- Benefits appear only when both sides understand Arrow.
- Arrays are designed to be immutable; frequent small appends or updates do not fit well.
- Library APIs and feature coverage vary between languages.
- Teams new to it face a learning curve around buffers, schemas and type mapping.

## How it compares with alternatives

| Option | Type | Typical role |
|---|---|---|
| Apache Arrow | In-memory columnar format and libraries | Processing and exchange between tools |
| [Apache Parquet](/tools/apache-parquet) | On-disk columnar file format | Compact long-term storage |
| JSON and CSV | Text, row-oriented | Human-readable, simple exchange |
| Protocol Buffers over [gRPC](/tools/grpc) | Row or message serialisation | Service APIs with small structured messages |
| JDBC and ODBC | Row-oriented database APIs | General database access |

## Practical guidance and best practices

1. Store data in Parquet and process it in Arrow; they complement each other.
2. Use Arrow between services or processes that move large tables, instead of JSON.
3. Prefer libraries that accept and return Arrow, so data stays in one format end to end.
4. Use ADBC or Flight SQL drivers when you fetch large result sets from databases that support them.
5. Use dictionary encoding for repeated strings to save memory.
6. Keep record batches reasonably sized, often tens of thousands to a few million rows, to balance overhead and memory.
7. Check type mappings, especially timestamps, time zones and decimals, when moving between tools.
8. Memory-map IPC files for fast local reuse of large intermediate results.

## Common mistakes

- Using Arrow IPC files as the long-term archive when compressed Parquet would be far smaller.
- Converting Arrow to row objects and back between steps, losing the benefit.
- Assuming "zero-copy" applies across a network; network transfers still move bytes.
- Ignoring null handling and time zone differences when converting to other libraries.
- Building tables one row at a time instead of in columnar batches.

## Frequently asked questions

### What is the difference between Arrow and Parquet?

Arrow is designed for data in memory: fast to compute on and cheap to share. Parquet is designed for data on disk: compressed, encoded and with statistics to skip data. Most pipelines read Parquet into Arrow, work on it and write Parquet again.

### What does zero-copy mean?

It means a consumer reads the producer's memory directly instead of receiving a converted copy. Within a process, Arrow makes this possible because both sides understand the same layout. Between processes, Arrow keeps serialisation cost very low even though bytes must still be moved.

### Does pandas use Arrow?

pandas can use Arrow-backed data types and reads and writes Parquet through PyArrow. Polars is built on Arrow's memory model. DuckDB can scan and return Arrow data. The level of integration differs by library and version.

### What is Arrow Flight used for?

Flight is a framework for moving large amounts of Arrow data over the network efficiently, using gRPC. It suits data services, query engines and feature stores that need to send many rows quickly, and Flight SQL extends it for SQL databases.

### Is Arrow a database?

No. Arrow is a format and a set of libraries. Databases and engines use it internally or for exchange, and some subprojects, such as the C++ compute functions, provide building blocks for query processing.

### Is Arrow useful for machine learning?

Yes, especially for moving training and inference data between data tools and ML libraries. Many data loaders and frameworks can consume Arrow or Parquet, which reduces conversion overhead.

## Glossary

- C Data Interface: stable C structures for passing Arrow data across language boundaries.
- Feather: an earlier name for the Arrow IPC file format.
- gRPC: an RPC framework used by Arrow Flight.
- Memory mapping: treating a file as memory so data is read on demand.
- nanoarrow: a small C library for producing and consuming Arrow data.
- SIMD: CPU instructions that process several values at once.
- Schema: names and types of the columns in a record batch.
- Serialisation: turning in-memory data into bytes for transfer or storage.

## Summary

Apache Arrow standardises how columnar data sits in memory, so tools and languages can share it without conversion. Its simple buffers suit fast vectorised processing, and its IPC format, Flight protocol and ADBC API extend the same idea across processes, networks and databases. Arrow does not replace storage formats: data usually lives in [Apache Parquet](/tools/apache-parquet) and is processed in Arrow. Used end to end, it removes much of the hidden cost of moving data between engines such as [DuckDB](/tools/duckdb), [Polars](/tools/polars) and [Apache Spark](/tools/apache-spark).