Big data
How large-scale data is stored, processed, streamed and organised for analytics and AI, in one map.
Overview
Big data refers to data too large, fast or varied for a single traditional database: clickstreams, sensor feeds, transactions, logs and documents at the scale of terabytes to petabytes. Big data engineering is how such data is stored, processed, streamed and made reliable for analytics and AI.
A small shop keeps its accounts in one ledger. A national retailer needs warehouses, conveyor belts, sorting centres and a catalogue system. Big data tools are that logistics network for information.
Storage and formats
Data lands in object storage or Hadoop HDFS as columnar files such as Apache Parquet, processed in memory with Apache Arrow, and organised into tables by formats such as Apache Iceberg, Delta Lake and Apache Hudi. This is the foundation of the Lakehouse, often organised in bronze, silver and gold layers.
Processing and querying
Apache Spark remains the workhorse for distributed processing; Trino, ClickHouse, BigQuery, Amazon Redshift and Snowflake serve SQL analytics; DuckDB and Polars make single-machine analytics remarkably fast.
Streaming
Real-time data flows through Apache Kafka, Apache Pulsar or Amazon Kinesis and is processed by Apache Flink, Spark Streaming or Apache Beam. Change data capture keeps systems in sync. The article Real-time AI: from event streams to decisions shows how these feed AI.
Orchestration and organisation
Pipelines are scheduled by Apache Airflow, Dagster or Prefect, transformed with dbt, and governed through catalogues, lineage and contracts. Data mesh distributes ownership to domain teams.
The full map
Storage and formats
Query engines and warehouses
Streaming
Orchestration
Architecture
Related topics
Frequently asked questions
What is a lakehouse?
An architecture that adds warehouse features such as transactions and schemas to open data lake storage, serving BI and ML from one copy of data.
Do I need big data tools?
Only when data outgrows a single machine or arrives continuously; tools like DuckDB handle surprisingly large data on one computer.
Summary
Big data engineering stores data in open columnar formats, processes it with distributed and embedded engines, streams it in real time and organises it as governed lakehouse tables for analytics and AI.