Apache Spark

Overview

Apache Spark is an open-source engine for processing large datasets across a cluster of machines. It offers one programming model for batch jobs, SQL queries, streaming, graph work and machine learning. You describe what you want with DataFrames or SQL, and Spark plans the work, splits it into tasks, runs them in parallel and combines the results. It can process data far larger than the memory of any single computer.

Spark is the workhorse behind many data pipelines. It reads and writes common formats and stores, and it works well with table formats such as Delta Lake and Apache Iceberg. Most people use it from Python through PySpark, and use Spark Streaming for near-real-time jobs. For tabular AI, Spark prepares, cleans and samples the large tables that models learn from, and it can run model inference at scale.

What it is

Spark is a distributed computing framework. It is written mostly in Scala and runs on the Java virtual machine, with APIs in Scala, Java, Python, R and SQL. A Spark application has one driver that plans the job and many executors that do the work on different machines.

A helpful analogy is a big kitchen preparing a banquet. The head chef (the driver) reads the order, splits it into steps and hands out tasks. Many cooks (executors) each chop a share of the vegetables. When a step needs everything in one place, such as combining sauces, the cooks pass ingredients around. The head chef never chops everything alone, and the banquet gets done much faster than with one cook.

The main components

• Spark Core. Scheduling, memory management, fault recovery and the basic distributed data abstraction.
• Spark SQL and DataFrames. Structured data processing with an optimiser.
• Structured Streaming. Stream processing using the same DataFrame API. See Spark Streaming.
• MLlib. Distributed machine learning algorithms and pipelines.
• GraphX. Graph processing, mostly used in Scala.
• Pandas API on Spark. A pandas-like interface that runs on Spark.

Where it runs

Spark runs on its own standalone cluster manager, on Hadoop YARN, on Kubernetes, and on managed platforms such as Databricks and cloud services like AWS Glue based jobs. It can also run in local mode on a laptop for development, with threads instead of machines.

Where it came from

Spark started as a research project at a university lab in the late 2000s. Researchers wanted to speed up iterative algorithms, such as machine learning, which were slow on the earlier MapReduce model because each step wrote results to disk. Spark kept data in memory between steps where possible, and offered a richer set of operations than just map and reduce.

The project was donated to the Apache Software Foundation and became a top-level project. Its early abstraction was the Resilient Distributed Dataset, or RDD, described in a research paper in the early 2010s. Later versions added DataFrames and a query optimiser, then Datasets, then Structured Streaming. The shift from low-level RDDs to declarative DataFrames was a major step, since it let the engine optimise the plan.

Over the years, Spark added adaptive query execution, better Python support, Arrow-based data exchange, ANSI SQL modes and connect-style client protocols. Exact version features vary, so consult the documentation for the version you run.

How it works

Step 1: You write a program

You create a SparkSession, load data into a DataFrame and apply operations such as filter, select, join and groupBy.

Step 2: Transformations are lazy

Most operations are transformations. They do not run at once. Spark just records them as a plan. This is called lazy evaluation. It is like writing a shopping list before going to the store. Nothing is bought until you decide to go.

Step 3: Actions trigger work

An action, such as count, collect, show or write, tells Spark to run the plan. Only then does the real computation begin.

Step 4: The optimiser builds a plan

For DataFrames and SQL, the Catalyst optimiser rewrites the logical plan. It pushes filters down, prunes unused columns, reorders joins and chooses efficient strategies. The Tungsten engine then generates compact code and manages memory efficiently. The result is a physical plan.

Step 5: The plan is split into stages and tasks

Spark cuts the physical plan into stages at points where data must be moved between machines. Each stage is divided into tasks, one per data partition. A task is the smallest unit of work and runs on one executor core.

Step 6: Narrow and wide dependencies

Some operations are narrow: each output partition depends on one input partition, such as filter or select. Others are wide: output partitions depend on many input partitions, such as groupBy or join. Wide operations need a shuffle, which redistributes data across the network by key.

Shuffles are the most expensive part of many jobs. A good analogy is a school reunion where everybody must sit at the table for their graduation year. People must walk across the room, which takes time. Fewer or smaller shuffles mean faster jobs.

Step 7: Executors run tasks

The driver sends tasks to executors. Each executor reads its partition, applies the operations and writes results or shuffle files. Executors may cache data in memory or on disk when you ask.

Step 8: Fault tolerance through lineage

Spark remembers how each partition was built, a record called lineage. If an executor dies, Spark recomputes only the lost partitions from their source and the recorded steps. This avoids copying every intermediate result for safety.

Step 9: Adaptive execution adjusts at runtime

Adaptive query execution looks at real statistics after each shuffle. It can merge small shuffle partitions, switch a join strategy and split skewed partitions. This reduces the need for manual tuning.

Step 10: Results are returned or written

Results are either written to storage in parallel or collected to the driver. Collecting very large results to the driver is a common cause of out-of-memory errors, so write large outputs to storage instead.

Key concepts and terms

A worked example

This example uses PySpark with fake paths. It reads order data, cleans it, joins a small lookup table and writes a summary.

Setup and read

from pyspark.sql import SparkSession, functions as F

spark = (
    SparkSession.builder
    .appName("orders_summary")
    .config("spark.sql.shuffle.partitions", "200")
    .getOrCreate()
)

orders = spark.read.parquet("s3://example.internal/lake/silver/orders")
countries = spark.read.parquet("s3://example.internal/lake/ref/countries")

Nothing has run yet. Spark has only read file metadata and recorded the plan.

Transform

clean = (
    orders
    .filter(F.col("status") != "cancelled")
    .withColumn("revenue", F.col("quantity") * F.col("unit_price"))
    .withColumn("order_date", F.to_date("order_ts"))
)

summary = (
    clean
    .join(F.broadcast(countries), on="country_code", how="left")
    .groupBy("order_date", "country_name")
    .agg(
        F.countDistinct("order_id").alias("orders"),
        F.sum("revenue").alias("revenue"),
        F.avg("unit_price").alias("avg_price"),
    )
)

The broadcast hint tells Spark to send the small country table to every executor, which avoids shuffling the large orders table for the join. Only the groupBy needs a shuffle.

Look at the plan

summary.explain()

The output shows the physical plan with scans, filters, a broadcast hash join, partial and final aggregations, and an exchange for the shuffle. Reading plans is a key skill for tuning.

Action

summary.write.mode("overwrite").partitionBy("order_date").parquet(
    "s3://example.internal/lake/gold/daily_summary"
)

This write is the action. Now the job runs: Spark reads files in parallel, filters, computes revenue, joins, shuffles by group keys, aggregates and writes partitioned output.

The same job in SQL

SELECT
  CAST(o.order_ts AS DATE)         AS order_date,
  c.country_name,
  COUNT(DISTINCT o.order_id)       AS orders,
  SUM(o.quantity * o.unit_price)   AS revenue,
  AVG(o.unit_price)                AS avg_price
FROM silver.orders o
LEFT JOIN ref.countries c ON o.country_code = c.country_code
WHERE o.status <> 'cancelled'
GROUP BY 1, 2;

SQL and DataFrame code compile to the same kind of plan, so use whichever your team reads best.

A numeric picture of partitions

Say the input is 10 GB and the target partition size is about 128 MB. That gives roughly 80 input partitions, so 80 tasks in the first stage. If the cluster has 40 executor cores, the tasks run in two waves. After the shuffle, with 200 shuffle partitions, each holds about 50 MB of data if evenly distributed. If one country holds half the data, one task may receive 5 GB and run far longer than the rest. That is skew, and adaptive execution or salting a key can help.

Caching a reused dataset

clean.cache()
clean.count()          # materialises the cache
clean.groupBy("country_code").count().show()
clean.unpersist()

Cache only datasets that are reused and fit in available memory. Unneeded caching wastes memory that tasks need.

Strengths and limitations

Strengths

• Scale. Handles data from megabytes to petabytes with the same code.
• Unified engine. Batch, SQL, streaming and ML in one framework.
• Rich API. DataFrames, SQL and Python, Scala, Java and R support.
• Optimiser. Catalyst and adaptive execution improve plans automatically.
• Fault tolerance. Lineage recovers from failed tasks and machines.
• Ecosystem. Many connectors, formats and table layers.
• Portability. Runs on many cluster managers and clouds.
• Community. Large user base and plenty of documentation.

Limitations

• Overhead on small data. For data that fits on one machine, a single-node tool is faster and simpler.
• Latency. Job start-up and scheduling make it a poor fit for millisecond queries.
• Tuning. Shuffles, skew, memory and partition counts need care.
• Python overhead. Python UDFs move data between processes and can be slow without vectorised alternatives.
• Memory pressure. Large collects, wide rows and skew cause out-of-memory errors.
• Operational burden. Clusters need sizing, monitoring and upgrades unless a managed service is used.
• Streaming trade-offs. Micro-batch streaming has latency floors that true event-at-a-time engines such as Apache Flink can beat.
• Not a transactional store. It needs a table format for ACID behaviour.

How it compares with alternatives

A common modern setup uses Spark for heavy transformation, a table format for storage, an orchestrator such as Apache Airflow for scheduling, and a warehouse or BI tool for serving.

Role in tabular foundation models

Tabular foundation models learn from rows and columns of many different tables. They deal with mixed types, missing values and sometimes missing column names, and they can predict from a few example rows by in-context learning. Spark does not replace such a model, but it feeds it, evaluates it and scales it.

Preparing and shaping tables

Real tables are big and messy. Spark can read many files, fix types, drop duplicates, join sources, and produce one clean table with one row per entity. That is the usual input for models like TabPFN, TabICL and TabSTAR. Spark also builds the layered tables of the medallion architecture, such as the gold layer feature tables models use.

Row and column structure

Spark DataFrames are exactly rows with typed columns. Moving between a Spark DataFrame and a model's input format is a matter of selecting columns and converting a sample to a local array or DataFrame. Because tabular foundation models have limits on context size, Spark is used to choose the context: stratified samples, recent rows, or nearest neighbours.

Mixed types and missing values

Spark keeps nulls as nulls and has functions to count, flag and fill them. Teams use it to profile missingness across billions of cells, to add indicator columns, and to run imputation at scale, either with simple rules or by calling a model in batches. A rule of thumb is to keep original values and write imputed values to new columns or tables.

No column names

Some tabular models work when columns are anonymous. Spark makes it easy to rename columns to c0, c1, and so on to test that a model does not depend on names, or to strip names that might leak labels. It can also hold a mapping table, so results can be translated back.

Pre-training data pipelines

Pre-training on many tables needs a large collection of tables processed in a consistent way. Spark can read thousands of small and large tables, apply the same cleaning, split into train and evaluation sets by table, and write them in a columnar format. For synthetic data, Spark can generate many tables in parallel from random seeds. Further background is on the model research and model architecture pages.

In-context learning at scale

In-context learning predicts rows given labelled example rows. To score a billion rows, you need to batch queries and attach a context to each batch. Spark partitions the data, and each task loads the model once, builds or receives a context and scores its partition. In PySpark, vectorised functions such as pandas-style UDFs help pass batches efficiently.

Continual learning

When data arrives continuously, Spark Structured Streaming or scheduled batch jobs can pick up new rows, compute updated features and refresh the context or fine-tune data for a continual learning model. Tracking table versions in Delta Lake tells the job exactly what is new since the last run.

Imputation and anomaly detection

Both tasks produce a value per row or per cell. Spark runs them over large tables by applying a model within each partition. Imputed values are written to new tables. Anomaly scores can be computed in the same pass and then aggregated, for example by day or by region. See anomaly detection for approaches.

Sector and semantic classification

Classifying records into sectors or tagging column types needs consistent features and a place to store results. Spark can compute features, call a classifier in batch, and write labels with confidence values for review. It also supports evaluation by joining predictions to ground truth and computing metrics per class, as discussed on the model benchmarks page.

Honest limits

Spark is not usually where the model itself runs best. Large neural models need GPUs and specific libraries, so many teams run the model on GPU servers and use Spark to feed it. For small tables, Spark is overkill, and a single machine is better. For gradient boosting on moderate data, tools like XGBoost also have Spark integrations.

Practical guidance and best practices

Writing good Spark code

• Prefer DataFrames and SQL over raw RDDs, so the optimiser can help.
• Select only needed columns early.
• Filter as early as possible.
• Avoid Python UDFs when a built-in function exists.
• If you need Python logic, prefer vectorised pandas-style functions over row-by-row UDFs.
• Use built-in functions for dates, strings and arrays.
• Avoid collect() on large data.
• Name stages and jobs so the UI is readable.

Partitions and shuffles

• Aim for partitions of roughly 100 to 200 MB after reading.
• Tune the shuffle partition count to data size, or rely on adaptive execution.
• Broadcast small tables in joins.
• Watch for skew, and handle it with salting, adaptive skew handling or pre-aggregation.
• Use repartition before a wide write when you need a certain number of output files.
• Use coalesce to reduce partitions without a full shuffle.

Storage and formats

• Use columnar formats like Parquet, ideally through a table format such as Delta Lake.
• Avoid many tiny files, and compact regularly.
• Partition by columns used in filters, with moderate cardinality.
• Keep schema explicit when reading CSV or JSON, rather than inferring it every time.

Resources

• Size executors with a reasonable number of cores and memory, rather than tiny or huge ones.
• Leave memory headroom for overhead and shuffle.
• Use dynamic allocation if your platform supports it.
• Use spot or preemptible capacity for fault-tolerant batch jobs.
• Set timeouts and retries for tasks.

Reliability and testing

• Keep jobs idempotent so retries are safe.
• Unit test transformation functions on small DataFrames in local mode.
• Add data quality checks before publishing.
• Log row counts at key stages.
• Use an orchestrator like Apache Airflow for scheduling and dependencies.

Observability

• Use the Spark UI to find slow stages, skew and spills.
• Look for large shuffle read and write sizes.
• Track job duration trends.
• Alert on failed and unusually long jobs.

Security and governance

• Keep credentials out of code, and use secret stores or role-based access.
• Encrypt data at rest and in transit.
• Register tables in a data catalog.
• Remove or mask personal data early in the pipeline.

Window functions and ranking

Window functions compute a value for each row based on a group of related rows without collapsing them. They are common in feature engineering, such as "the previous order amount" or "rank within customer".

from pyspark.sql import Window

w = Window.partitionBy("customer_id").orderBy("order_ts")

features = (
    orders
    .withColumn("prev_amount", F.lag("amount").over(w))
    .withColumn("order_rank", F.row_number().over(w))
    .withColumn("running_total", F.sum("amount").over(w))
)

Windows need a shuffle by the partition key, so a very hot key can cause skew. Always order by a time column that has no future information if the result feeds a model.

Handling skewed joins

Suppose one customer id appears in half the rows. A normal join sends all of them to one task. Options include:

• Let adaptive execution split the skewed partition.
• Add a random salt to the hot key on the big side, replicate the small side across salt values, and join on both.
• Filter out the hot key, handle it separately and union the results.
• Pre-aggregate before the join to reduce rows.

Check the Spark UI for tasks that take many times longer than the median. That is the signature of skew.

Reading and writing common formats

Testing Spark code

Good tests run fast and need no cluster. Build a local session, create a tiny DataFrame, call your function and compare the result.

def test_net_revenue(spark):
    df = spark.createDataFrame(
        [(2, 10.0, 0.0), (1, 5.0, 5.0)],
        ["quantity", "unit_price", "refund"],
    )
    out = add_net_revenue(df).collect()
    assert [r.net_revenue for r in out] == [20.0, 0.0]

Keep transformation logic in small pure functions that take and return DataFrames. Keep reading and writing in a thin outer layer. This makes unit tests simple and keeps jobs easy to reason about.

Sizing a cluster, roughly

There is no exact formula, but a simple method helps you start:

1. Estimate input size after filters.
2. Divide by a target partition size, about 128 MB, to get the number of tasks.
3. Choose total cores so the stage finishes in a few waves.
4. Give each core a few gigabytes of memory, more for wide joins and caching.
5. Run, then read the UI for spills to disk and long tasks, and adjust.

Spills to disk mean memory is too tight for the shuffle. Long tail tasks mean skew. Many tiny tasks mean too many partitions.

Batch scoring with a model

A common pattern is to load a model once per partition and score batches of rows. The pandas-style function below receives a batch as a pandas DataFrame and returns predictions.

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def score(batch: pd.Series) -> pd.Series:
    model = get_model()          # cached per worker process
    return pd.Series(model.predict(batch.to_frame()))

scored = features.withColumn("score", score("total_spend"))

The helper get_model is a placeholder for your own loader. In practice, pass several feature columns and keep model loading outside the per-row path. For a tabular foundation model with an in-context step, the function would also receive or fetch the context rows. Test with a small partition first, since model memory per worker can limit parallelism.

Version and upgrade habits

Spark releases bring performance and SQL behaviour changes. Pin versions in production, read the migration notes before upgrading, run your test suite and a sample of real jobs on the new version, and compare output row counts and checksums. Behaviour changes in date parsing, null handling and type coercion are the usual surprises.

Common mistakes

1. Using Spark for tiny data. A single-node tool is faster and cheaper.
2. Collecting big results to the driver. This causes out-of-memory failures.
3. Row-by-row Python UDFs. They are slow because of serialisation overhead.
4. Ignoring skew. One giant task delays the whole stage.
5. Too few or too many partitions. Too few underuse the cluster, too many add scheduling overhead.
6. Caching everything. Cache steals memory from execution.
7. Letting schema inference run on huge files. It adds a full pass and may guess wrong.
8. Not broadcasting small tables. A needless shuffle slows joins.
9. Writing many small files. Downstream reads suffer.
10. Mixing time zones. Date logic produces different results on different clusters.
11. No idempotency. A retry duplicates data.
12. Tuning by guesswork. Always inspect the plan and the UI before changing settings.

Frequently asked questions

What is Apache Spark used for?

Spark is used to transform, analyse and learn from large datasets. Typical uses include ETL pipelines, SQL analytics, log processing, feature engineering, streaming aggregation and distributed machine learning. It is popular because one engine and one set of APIs cover batch, SQL and streaming work.

What is the difference between Spark and Hadoop?

Hadoop is a broader ecosystem that includes a distributed file system, a resource manager and the MapReduce engine. Spark is a processing engine that can run on Hadoop's resource manager and read from its storage, but it is much faster for many workloads because it avoids writing every intermediate result to disk. Many modern deployments use Spark with cloud storage and no Hadoop at all.

What is a DataFrame in Spark?

A Spark DataFrame is a distributed table with named columns and types. You work with it through functions like select, filter, join and groupBy, or through SQL. Because the operations are declarative, Spark can optimise the whole plan before running it. It is the main API for most Spark users.

What does lazy evaluation mean?

Lazy evaluation means Spark records transformations but does not run them until an action asks for a result. This lets the optimiser see the whole plan and improve it, for example by pushing filters closer to the data source. It also means errors in a transformation may only appear when the action runs.

What is a shuffle and why is it expensive?

A shuffle redistributes data across the cluster so that rows with the same key end up together, which is needed for joins, groupBy and some window operations. It involves writing data to disk, sending it over the network and reading it back. Reducing the amount of shuffled data is a major part of tuning.

When should I use Spark instead of pandas?

Use Spark when the data is too large for one machine, when you need to run on a cluster, or when you want one pipeline to scale from small to large. If the data fits comfortably in memory on one machine, pandas, Polars or DuckDB are usually simpler and quicker. Some teams prototype in pandas and move to Spark for production.

Can Spark run machine learning models?

Yes. MLlib provides distributed algorithms for classification, regression, clustering and feature processing, and Spark integrates with other libraries. It is also commonly used to prepare data and to run batch inference with models trained elsewhere. Large neural networks usually run on GPU servers, fed by Spark.

How do I make a Spark job faster?

Start by reading the plan and the Spark UI. Reduce data early with filters and column selection, broadcast small tables, fix skew, use efficient file formats and avoid slow Python UDFs. Check partition counts and file sizes. Only then change cluster settings. Most large gains come from changing the query, not the hardware.

Is Spark good for real-time processing?

Spark Structured Streaming handles near-real-time workloads with latencies typically from sub-second to seconds, using micro-batches or a continuous mode for some cases. For strict millisecond latency or complex event-time patterns, engines designed for event-at-a-time processing may be a better choice. See Spark Streaming for details.

Do I need to know Scala to use Spark?

No. Most users work with Python through PySpark or with SQL. Scala is useful for writing custom extensions and for the typed Dataset API, but day-to-day data work rarely requires it.

Glossary

• Action: An operation that triggers execution of the plan.
• Adaptive query execution: Runtime plan changes based on actual data statistics.
• Broadcast join: A join that sends a small table to all executors.
• Catalyst: Spark's query optimiser.
• Cluster manager: Software that allocates cluster resources.
• DataFrame: A distributed table with named, typed columns.
• Driver: The coordinating process of a Spark application.
• Executor: A worker process that runs tasks.
• Lazy evaluation: Delaying work until a result is needed.
• Lineage: The history of transformations used to rebuild data.
• MLlib: Spark's machine learning library.
• Narrow dependency: An operation where each output partition depends on one input partition.
• Partition: A slice of a dataset processed by one task.
• RDD: A low-level distributed collection of records.
• Shuffle: Data movement across the cluster to group by key.
• Skew: Uneven distribution of data across partitions.
• Stage: A set of tasks that run without a shuffle between them.
• Task: The unit of work applied to one partition.
• Transformation: A lazy operation that defines a new dataset.
• Wide dependency: An operation where output partitions depend on many inputs.

Summary

Apache Spark is a general engine for processing large tables in parallel. You write DataFrame or SQL code, Spark optimises the plan, splits it into stages and tasks, and runs them across a cluster with automatic recovery from failures. Its main costs are shuffles, skew and start-up overhead, so tuning starts with reading the plan.

For tabular AI, Spark prepares and samples clean tables, profiles missing values, scales imputation, anomaly scoring and classification, and keeps pipelines repeatable with versioned tables. Pair it with Delta Lake for reliable storage and PySpark for Python access, and use it when data is large enough to need a cluster.

Related

It belongs to Batch and streaming processing in my stack, under Data engineering and lakehouse. Others in the same group: