Apache Beam

Overview

Apache Beam is an open-source programming model for data processing pipelines. You describe what should happen to the data, such as reading, transforming, grouping, joining and writing, using a Beam SDK in Java, Python or Go. A separate component called a runner then executes that pipeline on a processing engine, such as Google Cloud Dataflow, Apache Flink or Apache Spark.

Beam's central idea is that batch and streaming are the same problem. A batch job processes a bounded data set, such as yesterday's files. A streaming job processes an unbounded one, such as an endless flow of events. Beam gives both the same abstractions. The name itself combines "batch" and "stream". To handle the hard parts of streaming, such as late data and out-of-order events, Beam provides windows, watermarks and triggers that let you say precisely when results should be computed and updated.

An analogy is a recipe written in a standard format. The recipe says what to do: chop, mix, bake for 30 minutes. It does not care whether it is cooked in a home kitchen, a restaurant or a factory. Beam pipelines are recipes, and runners are kitchens. The same recipe can be cooked in different kitchens, although each kitchen has its own strengths.

What it is

Where it came from

Beam's ideas come from Google's internal data processing work, including FlumeJava for batch pipelines and MillWheel for streaming. Google combined them into Cloud Dataflow and described the approach in the Dataflow Model paper in 2015. In 2016 Google and partners contributed the Dataflow SDKs and runners to the Apache Software Foundation, where the project became Apache Beam. It became a top-level Apache project on 10 January 2017, and Beam 2.0, the first stable release, followed in May 2017.

Since then Beam has added the Python and Go SDKs, a portability framework so that one runner can execute pipelines from several languages, cross-language transforms that let Python pipelines use Java connectors, Beam SQL, a DataFrame API for Python, a YAML way of defining pipelines and support for machine learning inference with RunInference. Releases come regularly, in the 2.x series.

How it works

Building a pipeline

1. You create a Pipeline object with options that choose the runner and its settings.
2. A read transform creates the first PCollection from a source, such as files or a Apache Kafka topic.
3. You apply transforms. Each returns a new PCollection, forming a directed acyclic graph.
4. Write transforms send results to sinks.
5. When you run the pipeline, the runner translates the graph into its engine's jobs, optimises it, for example by fusing consecutive steps, and executes it on workers.

Event time, windows and watermarks

Beam distinguishes event time, when something happened, from processing time, when the system sees it. Windows group elements by event time. Fixed windows cut time into equal slices, such as one minute. Sliding windows overlap. Session windows group activity separated by gaps of inactivity. The global window holds everything, which is the default for batch.

Because events can arrive late and out of order, the runner tracks a watermark, an estimate that all data up to a given event time has arrived. When the watermark passes the end of a window, the window is normally considered complete.

Triggers and late data

Triggers control when results are emitted. The default emits once when the watermark passes the window's end. You can add early firings, such as every 30 seconds of processing time for a live estimate, and late firings for data that arrives after the watermark, within an allowed lateness. Accumulation mode decides whether each firing reports the full result so far or only the change since the last firing.

State, timers and exactly-once results

For advanced cases, stateful DoFns keep per-key state and set timers. Runners that support it, such as Dataflow and Flink, checkpoint state and deliver exactly-once processing within the pipeline. End-to-end exactly-once output also depends on the sink.

import apache_beam as beam
from apache_beam.transforms.window import FixedWindows
with beam.Pipeline() as p:
    (p | beam.io.ReadFromText("clicks.csv")
       | beam.Map(lambda line: (line.split(",")[1], 1))
       | beam.combiners.Count.PerKey()
       | beam.io.WriteToText("counts"))
# For a stream, add: | beam.WindowInto(FixedWindows(60)) before counting.

Key concepts and terms

A worked example

This scenario is illustrative. A media site wants a count of video plays per title every minute, using event time, for a live trending list, and the same logic run over last year's logs to backfill history.

1. Plays are published to a messaging topic. Each event has a title_id and a played_at timestamp. About 5 per cent of events arrive more than a minute late from mobile devices that were offline.
2. The pipeline reads the stream, assigns event timestamps from played_at and applies one-minute fixed windows.
3. It counts plays per title_id, with an early trigger every 10 seconds of processing time for a live estimate and allowed lateness of 10 minutes.
4. Results go to a key-value store for the website, with accumulating mode so each update replaces the previous count.
5. For the backfill, the team swaps the read step for a file source over a year of logs. The windowing and counting code is unchanged. They run it in batch mode on the same runner and write the results to the warehouse.
6. They test both modes locally with the direct runner on small samples, using TestStream to simulate late data.

Strengths and limitations

Strengths:

• One model and codebase for batch and streaming.
• Precise control of event time, windows, triggers and late data.
• Portable across several runners, reducing engine lock-in.
• SDKs in Java, Python and Go, with cross-language transforms.
• Many built-in connectors and a fully managed option through Dataflow.

Limitations:

• Portability is not perfect; runners support different feature sets and performance varies.
• The model has a learning curve, especially windows, triggers and watermarks.
• Debugging distributed pipelines can be harder than with engine-native APIs.
• Some engine-specific optimisations are hidden behind the abstraction.
• Python pipelines may be slower than Java for heavy per-element work.

How it compares with alternatives

Practical guidance and best practices

1. Decide early whether results depend on event time; if so, assign timestamps at the source.
2. Choose windows that match the business question, such as fixed for reports and sessions for user activity.
3. Set allowed lateness and triggers deliberately, and document what each output firing means.
4. Prefer Combine transforms over GroupByKey followed by manual aggregation; they can pre-aggregate on workers.
5. Avoid hot keys; spread heavy keys or use combiners that support fan-out.
6. Test with the direct runner and TestStream, including late and out-of-order data.
7. Check the runner capability matrix before relying on advanced features.
8. Make sinks idempotent or transactional for reliable end-to-end results.

Common mistakes

• Using processing time for business metrics that should use event time.
• Leaving the default trigger and being surprised that late data is dropped.
• Building a huge side input that does not fit in worker memory.
• Calling external services once per element without batching, which slows pipelines and overloads services.
• Assuming a pipeline that works on one runner will behave the same on another.
• Forgetting that a streaming pipeline's global window never closes without triggers.

Frequently asked questions

Is Apache Beam an execution engine?

No. Beam is a programming model and set of SDKs. Execution happens on a runner, such as Google Cloud Dataflow, Apache Flink, Apache Spark or a local runner for testing.

Why use Beam instead of writing directly for Flink or Spark?

You get one model for batch and streaming, strong event-time semantics and the option to change runners. If you are committed to one engine and need its specialised features, its native API may be simpler.

What is a watermark?

It is the runner's estimate of how far event time has progressed, meaning it believes most data up to that time has arrived. Watermarks tell Beam when a window can be considered complete. Data arriving after the watermark is late and is handled according to allowed lateness and triggers.

What is the relationship between Beam and Dataflow?

Dataflow is Google Cloud's managed service that runs Beam pipelines. Beam grew out of Dataflow's SDKs, and Dataflow is one of the most widely used Beam runners, but Beam itself is an independent Apache project.

Can I use Python and Java together in one pipeline?

Yes, through cross-language transforms on runners that support the portability framework. A common use is calling Java I/O connectors from a Python pipeline.

Does Beam support machine learning?

Yes. The RunInference transform runs models from several frameworks inside pipelines, for batch or streaming inference, and Beam is often used for preprocessing training data.

Glossary

• Accumulation mode: whether each trigger firing reports totals so far or only new changes.
• DAG: directed acyclic graph of pipeline steps.
• Direct runner: a local runner for development and testing.
• Fixed window: non-overlapping time interval of equal length.
• Session window: window that closes after a gap of inactivity.
• Sliding window: overlapping windows of fixed length.
• TestStream: a testing source that simulates timing, watermarks and late data.
• Portability framework: Beam's language-neutral protocol between SDKs and runners.

Summary

Apache Beam lets you write a data pipeline once and run it as batch or streaming on several engines. Its PCollections and transforms describe the processing, while windows, watermarks and triggers give precise control over event time and late data. Runners such as Dataflow, Apache Flink and Apache Spark execute the work. Portability has limits and the model takes effort to learn, but for teams that need consistent batch and streaming logic it is a strong choice. See real-time systems for where it fits in a wider architecture.

Related

It belongs to Streaming in my stack, under Big data. Others in the same group: