# Big data

How large-scale data is stored, processed, streamed and organised for analytics and AI, in one map.

## Overview

Big data refers to data too large, fast or varied for a single traditional database: clickstreams, sensor feeds, transactions, logs and documents at the scale of terabytes to petabytes. Big data engineering is how such data is stored, processed, streamed and made reliable for analytics and AI.

A small shop keeps its accounts in one ledger. A national retailer needs warehouses, conveyor belts, sorting centres and a catalogue system. Big data tools are that logistics network for information.

## Storage and formats

Data lands in object storage or [Hadoop HDFS](/tools/hadoop-hdfs) as columnar files such as [Apache Parquet](/tools/apache-parquet), processed in memory with [Apache Arrow](/tools/apache-arrow), and organised into tables by formats such as [Apache Iceberg](/tools/apache-iceberg), [Delta Lake](/tools/delta-lake) and [Apache Hudi](/tools/apache-hudi). This is the foundation of the [Lakehouse](/tools/lakehouse), often organised in [bronze](/tools/bronze-layer), [silver](/tools/silver-layer) and [gold](/tools/gold-layer) layers.

## Processing and querying

[Apache Spark](/tools/apache-spark) remains the workhorse for distributed processing; [Trino](/tools/trino), [ClickHouse](/tools/clickhouse), [BigQuery](/tools/bigquery), [Amazon Redshift](/tools/amazon-redshift) and [Snowflake](/tools/snowflake) serve SQL analytics; [DuckDB](/tools/duckdb) and [Polars](/tools/polars) make single-machine analytics remarkably fast.

## Streaming

Real-time data flows through [Apache Kafka](/tools/apache-kafka), [Apache Pulsar](/tools/apache-pulsar) or [Amazon Kinesis](/tools/amazon-kinesis) and is processed by [Apache Flink](/tools/apache-flink), [Spark Streaming](/tools/spark-streaming) or [Apache Beam](/tools/apache-beam). [Change data capture](/tools/change-data-capture) keeps systems in sync. The article [Real-time AI: from event streams to decisions](/articles/real-time-ai-from-streams-to-decisions) shows how these feed AI.

## Orchestration and organisation

Pipelines are scheduled by [Apache Airflow](/tools/apache-airflow), [Dagster](/tools/dagster) or [Prefect](/tools/prefect), transformed with [dbt](/tools/dbt), and governed through catalogues, lineage and contracts. [Data mesh](/tools/data-mesh) distributes ownership to domain teams.

## The full map

### Storage and formats

- [Hadoop HDFS](https://www.ozercemkelahmet.com/tools/hadoop-hdfs.md)
- [Apache Hive](https://www.ozercemkelahmet.com/tools/apache-hive.md)
- [Apache Hudi](https://www.ozercemkelahmet.com/tools/apache-hudi.md)
- [Apache Parquet](https://www.ozercemkelahmet.com/tools/apache-parquet.md)
- [Apache Arrow](https://www.ozercemkelahmet.com/tools/apache-arrow.md)

### Query engines and warehouses

- [DuckDB](https://www.ozercemkelahmet.com/tools/duckdb.md)
- [Polars](https://www.ozercemkelahmet.com/tools/polars.md)
- [BigQuery](https://www.ozercemkelahmet.com/tools/bigquery.md)
- [Amazon Redshift](https://www.ozercemkelahmet.com/tools/amazon-redshift.md)

### Streaming

- [Amazon Kinesis](https://www.ozercemkelahmet.com/tools/amazon-kinesis.md)
- [Apache Pulsar](https://www.ozercemkelahmet.com/tools/apache-pulsar.md)
- [Apache Beam](https://www.ozercemkelahmet.com/tools/apache-beam.md)

### Orchestration

- [Dagster](https://www.ozercemkelahmet.com/tools/dagster.md)
- [Prefect](https://www.ozercemkelahmet.com/tools/prefect.md)

### Architecture

- [Lakehouse](https://www.ozercemkelahmet.com/tools/lakehouse.md)
- [Data mesh](https://www.ozercemkelahmet.com/tools/data-mesh.md)

## Related topics

- [Machine learning](/topics/machine-learning)
- [MLOps and AI infrastructure](/topics/mlops-ai-infrastructure)
- [Generative AI and LLMs](/topics/generative-ai-llms)

## Frequently asked questions

### What is a lakehouse?

An architecture that adds warehouse features such as transactions and schemas to open data lake storage, serving BI and ML from one copy of data.

### Do I need big data tools?

Only when data outgrows a single machine or arrives continuously; tools like DuckDB handle surprisingly large data on one computer.

## Summary

Big data engineering stores data in open columnar formats, processes it with distributed and embedded engines, streams it in real time and organises it as governed lakehouse tables for analytics and AI.