Glossary · Big data
What is Apache Arrow? Definition and meaning
A language-independent in-memory columnar format with libraries for zero-copy exchange, IPC and RPC (Flight).
Definition
Apache Arrow: A language-independent in-memory columnar format with libraries for zero-copy exchange, IPC and RPC (Flight).
In short: One memory format for every tool.
When it is used
Fast data exchange between engines, languages and services.
Where it fits
Apache Arrow belongs to Storage and formats in the Big data topic. These are the other terms in the same group:
- Hadoop HDFS: An open-source framework for distributed storage (HDFS) and processing (MapReduce, YARN) on clusters of commodity servers.
- Apache Hive: A data warehouse system that runs SQL over files in distributed storage and keeps table metadata in the Hive Metastore.
- Apache Hudi: An open table format and data platform that supports record-level upserts, deletes and incremental queries on lake storage.
- Apache Parquet: An open, compressed columnar file format that stores data by column for efficient analytical reads.
Learn more
The full guide to Apache Arrow explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Apache Airflow · Next term: Apache Beam