Polars

Overview

Polars is an open-source DataFrame library written in Rust, with its most popular interface in Python. A DataFrame is a table in memory with named, typed columns, the everyday tool of data analysis. Polars is designed to make DataFrame work fast and memory-efficient on a single machine. It uses a columnar memory model based on Apache Arrow, runs work in parallel across all CPU cores and includes a query optimiser.

Polars offers two ways of working. In eager mode, each operation runs immediately, as in pandas. In lazy mode, you describe a whole chain of operations first and Polars builds a plan, optimises it and only then runs it. The optimiser can push filters into file scans, read only the columns you use and avoid unneeded intermediate results. Polars also has a streaming engine that processes data in batches, so some queries can run on data larger than memory.

An analogy: eager mode is like sending a shopper to the supermarket once for every item on your list, one trip at a time. Lazy mode hands the full list over first. The shopper groups items by aisle, skips things you already have and makes one efficient trip.

What it is

Polars has no index, unlike pandas. Rows are addressed by position and data is selected through expressions, which keeps the semantics simpler and easier to parallelise.

Where it came from

Polars was started by Ritchie Vink in 2020 as a Rust project, and its Python bindings soon became the main way people used it. A company was founded around the project in 2023 to fund development and build commercial offerings, including a managed cloud service. Python Polars 1.0 was announced on 1 July 2024, declaring the in-memory engine and API production-ready. The same announcement described a full redesign of the streaming engine around morsel-driven parallelism. In 2024 Polars also gained an optional GPU engine built with NVIDIA's RAPIDS cuDF library. Polars is released under the MIT licence.

How it works

Expressions

The core idea is the expression. An expression describes a computation on columns, such as "take the price column, multiply by quantity, sum it per customer". Expressions are evaluated inside contexts: select, with_columns, filter and group_by().agg(). Because Polars sees the whole expression, it can run independent expressions in parallel and plan efficient execution.

Building and optimising a lazy plan

1. You start from a lazy source, such as pl.scan_parquet(...) or df.lazy().
2. Each method call adds a node to a logical plan. Nothing runs yet.
3. When you call collect(), the optimiser rewrites the plan. It moves filters as close to the scan as possible (predicate pushdown), reads only referenced columns (projection pushdown), simplifies expressions and reorders some operations.
4. The physical engine executes the plan in parallel. With the streaming engine selected, it processes data in batches rather than loading everything first.
5. You can call explain() to see the optimised plan before running it.

import polars as pl
out = (
    pl.scan_parquet("trips/*.parquet")
      .filter(pl.col("distance_km") > 0)
      .group_by("city")
      .agg(pl.col("fare").mean().alias("avg_fare"))
      .collect()
)

Memory and parallelism

Each column is stored as a contiguous Arrow-style array with a validity bitmap for nulls. Operations run in compiled Rust code. Large operations such as group-bys and joins are split across threads. Strings use a compact representation, and categorical and enum types reduce memory for repeated labels.

Key concepts and terms

A worked example

This scenario is illustrative. A data engineer maintains a nightly pandas script that reads 120 CSV files of sensor readings, 30 GB in total, on a server with 64 GB of memory. The script loads everything, filters to one region, joins a device table and computes hourly averages. It often runs out of memory and takes about 50 minutes.

1. She first converts the CSVs to Parquet once, which also fixes column types.
2. She rewrites the script with pl.scan_parquet, filtering on region and selecting six of 25 columns before the join.
3. explain() shows the filter and column selection pushed into the scan, so Polars reads only about a fifth of the data.
4. The join and hourly group_by run in parallel on the server's 16 cores. Peak memory falls well below the machine's limit.
5. The job now finishes in a few minutes. Because the business logic is expressed as expressions, she writes unit tests on small DataFrames and runs the same code on the full data.

The exact speed-up depends on the data and the queries; the gains here come mainly from reading less data, avoiding copies and using all cores.

Strengths and limitations

Strengths:

• Fast, multi-threaded execution in compiled Rust.
• Lazy mode with a real query optimiser.
• Lower memory use thanks to columnar storage and pushdown.
• A consistent, composable expression API.
• Streaming execution for larger-than-memory work.
• Good interoperability through Arrow with DuckDB, PyArrow and others.

Limitations:

• Single-machine by design; it does not distribute work across a cluster in its open-source form.
• A different API from pandas, so existing code needs rewriting, not just renaming.
• Some libraries expect pandas objects, requiring conversion at the boundary.
• Not every operation is supported by the streaming or GPU engines; unsupported parts fall back.
• The API has evolved quickly, so older examples online may be outdated.

How it compares with alternatives

Practical guidance and best practices

1. Prefer the lazy API with scan_* sources for pipelines; use eager mode for quick exploration.
2. Store inputs as Parquet so pushdown can skip columns and row groups.
3. Write logic as expressions rather than Python functions applied row by row, which run slowly.
4. Use explain() to check that filters and column selections are pushed down.
5. Set explicit data types for keys and categories, and use categorical or enum types for repeated strings.
6. Convert to pandas only at the edges, where another library requires it.
7. Pin the Polars version in production and read release notes when upgrading.
8. Test streaming mode on your largest queries if memory is tight.

Common mistakes

• Translating pandas code line by line, including loops and apply calls, and losing the speed advantage.
• Calling collect() after every step, which turns lazy mode back into eager mode.
• Reading CSVs repeatedly instead of converting them to Parquet once.
• Expecting a pandas-style index and writing awkward workarounds.
• Using map_elements with Python functions for simple operations that have built-in expressions.
• Assuming Polars will scale to any size; very large data may still need a distributed engine.

Frequently asked questions

Is Polars faster than pandas?

Often, for medium and large data, because it runs compiled code in parallel, uses columnar memory and optimises lazy queries. For very small data or simple operations the difference may not matter. Measure on your own workload.

Do I have to use lazy mode?

No. Eager mode works much like pandas and is fine for interactive work. Lazy mode is recommended for pipelines and larger data because it allows optimisation and streaming.

Can Polars handle data larger than memory?

Some queries can run with the streaming engine, which processes data in batches. Support depends on the operations involved. For data far beyond one machine's disk and memory, a distributed engine is more suitable.

Can I use Polars with scikit-learn and plotting libraries?

Yes, in many cases. Some libraries accept Polars objects directly; others need a conversion to NumPy or pandas, which Polars provides. Keep the heavy data preparation in Polars and convert at the boundary.

Polars or DuckDB?

Both are fast single-machine engines that share Arrow memory well. Choose Polars if your team prefers a DataFrame expression API in Python or Rust. Choose DuckDB if your team prefers SQL. Many projects use both, passing data between them cheaply.

Does Polars support SQL?

Yes. It has a SQL interface that runs queries against DataFrames and LazyFrames, translated into the same optimised plans.

Glossary

• Arrow: the columnar in-memory format Polars' memory model follows.
• Eager execution: running each operation as soon as it is called.
• Lazy execution: deferring work until the full plan is known.
• Global interpreter lock: a CPython lock that limits parallel Python code; Polars avoids it by running in Rust.
• Morsel-driven parallelism: splitting work into small chunks that threads pick up.
• Query plan: the tree of operations an engine will run.
• RAPIDS cuDF: NVIDIA's GPU DataFrame library, used by Polars' GPU engine.
• UDF: a user-defined function, usually slower than built-in expressions.

Summary

Polars is a fast DataFrame library that brings ideas from databases, such as query optimisation, pushdown and parallel execution, to everyday data wrangling in Python and Rust. Its expression API and lazy mode let it plan work before running it, and its Arrow-based memory keeps exchange with tools such as DuckDB and Apache Arrow cheap. It stays on one machine and asks you to learn a new API, but for many pipelines it replaces slow pandas scripts with shorter, clearer and much faster code. When data truly outgrows a single server, PySpark or another distributed engine takes over.

Related

It belongs to Query engines and warehouses in my stack, under Big data. Others in the same group: