# Query and processing engines: Spark, DuckDB, Trino, ClickHouse and warehouses

With data in open formats, the engine becomes a choice per job. This page compares the main processing and query engines by data size, latency and team skills.

## Overview

Separating storage from compute means several engines can work on the same tables. The right engine depends on how much data a job touches, how fast an answer is needed, and whether the work is a pipeline, an interactive query or a dashboard.

## The engines compared

| Engine | Runs on | Best for | Note |
|---|---|---|---|
| [Apache Spark](/tools/apache-spark) / [PySpark](/tools/pyspark) | Cluster | Large batch pipelines, ML preprocessing | Most widely used for big pipelines |
| [DuckDB](/tools/duckdb) | One machine | Analytics up to hundreds of GB on a laptop or server | Version 2.0 (October 2026) adds a server mode |
| [Polars](/tools/polars) | One machine | Fast DataFrame work in Python | Arrow-based, multi-threaded |
| [Trino](/tools/trino) | Cluster | Interactive SQL across many sources | Federated queries |
| [ClickHouse](/tools/clickhouse) | Server or cluster | Real-time analytics, logs, traces | Very fast aggregations |
| Cloud warehouses | Managed | Governed SQL for the whole company | [Snowflake](/tools/snowflake), [BigQuery](/tools/bigquery), [Redshift](/tools/amazon-redshift) |
| Serverless SQL | Managed | Occasional queries on lake files | [Amazon Athena](/tools/amazon-athena) |

## Choosing

1. **Start on one machine.** DuckDB or Polars handle far more data than most teams expect, with no cluster to run.
2. **Move to Spark when data or teams grow.** Pipelines that touch terabytes, or many teams sharing jobs, benefit from a cluster; managed options include [Databricks](/tools/databricks) and [Amazon EMR](/tools/amazon-emr).
3. **Use Trino for exploration across sources.** It queries lakes, databases and warehouses with one SQL dialect.
4. **Use ClickHouse for high-volume event analytics.** Dashboards over billions of rows answer in under a second.
5. **Keep a warehouse where governance and many SQL users matter.**

## Orchestration

Engines run jobs; orchestrators decide when. [Apache Airflow](/tools/apache-airflow), [Dagster](/tools/dagster) and [Prefect](/tools/prefect) schedule pipelines, track runs and retry failures; [dbt](/tools/dbt) manages SQL transformations with tests.

## Frequently asked questions

### Is Spark still necessary?

For multi-terabyte pipelines and shared cluster workloads, yes. For smaller data, single-node engines are simpler and often faster.

### Can these engines read the same table?

Yes. With Iceberg or Delta Lake tables and a shared catalog, Spark, Trino, DuckDB and warehouses can all read the same data.

## Summary

Pick engines per job: one machine first, Spark for big pipelines, Trino for federated SQL, ClickHouse for real-time analytics, and a warehouse for governed company-wide SQL.

## Back to the topic

This page is part of **[Big data](/topics/big-data)**, which maps every concept, tool, simulation and article on the subject.
