Apache Flink

Overview

Apache Flink is a distributed engine for processing streams of events with low latency and reliable state. It reads data as it arrives, keeps running totals and other memory in a fault-tolerant store, and writes results out continuously. It can also run the same logic over finite files, which makes it a single engine for both live and historical data.

Teams pick Flink when answers must be fresh and correct at the same time. Fraud checks, live dashboards, sensor monitoring and feature pipelines are typical uses. Flink treats time carefully, so a late or out-of-order event still lands in the right window. Its checkpoints let a job recover after a crash without losing or double counting data. Most Flink jobs read from Apache Kafka and write to a lake, a warehouse or another topic.

What it is

Flink is an open source framework and runtime from the Apache Software Foundation. You write a program, or a SQL query, that describes how events are transformed. Flink splits the work into parallel tasks and spreads them over many machines.

The core idea is that a stream is the basic data model. A batch file is just a stream that happens to end. This view is often called "streams first".

The main building blocks

• A source reads events from a system such as a log, a message queue or files.
• An operator transforms events. Examples are map, filter, join, aggregate and window.
• A sink writes the results to a log, a database, a lake table or a file.
• A job is the whole graph of sources, operators and sinks.
• A cluster runs jobs. It has a coordinator that plans work and workers that execute it.

The APIs

Flink offers several layers. Pick the one that fits the task.

• SQL and the Table API let you write declarative queries over streams. They are the easiest way to start.
• The DataStream API gives full control over state, timers and custom logic in Java or Scala. A Python version also exists.
• Process functions are the low-level building block. You get access to state, timers and event time directly.

What "stateful" means

Many questions need memory. How many logins did this user make in the last ten minutes? What was the previous reading from this sensor? Flink keeps that memory next to the computation as managed state.

State is partitioned by key. Each key, such as a user id or a device id, has its own private slice. This is how Flink scales: more machines hold more keys.

Think of a restaurant with many waiters. Each waiter keeps a notepad for the tables they serve. The notepad is the state. The kitchen does not need to know about it.

Where it came from

Flink grew out of a research project at universities in Berlin. The project was called Stratosphere and it studied parallel data processing. It joined the Apache incubator in the mid 2010s and became a top level project soon after.

The early 2010s were the era of batch engines such as Hadoop MapReduce. Streaming engines of that time were fast but weak on guarantees. Many offered "at least once" delivery, which could double count events. Some systems paired a fast streaming layer with a slow batch layer to correct errors. That pairing was known as the lambda architecture, and it meant writing every piece of logic twice.

Flink aimed to remove that duplication. Its design used ideas from the Chandy-Lamport algorithm for taking consistent snapshots of a distributed system. It also drew on the Dataflow model, a paper from the Google research community that described event time, windows and triggers in a unified way. The result was a streaming engine that gives exactly-once state consistency and treats event time as a first-class concept.

A short timeline in words

• Early research phase: Stratosphere explores flexible parallel dataflows.
• Incubation: the project is renamed Flink and joins the Apache incubator.
• Top level project: the community grows and adopts the DataStream API.
• Streaming SQL: the Table API and SQL layer matures and unifies batch and stream.
• Later years: better state backends, unaligned checkpoints, a Kubernetes operator and wider connector support.

Exact release numbers change often, so check the project documentation for the current version.

How it works

This section walks through a Flink job from submission to result.

Step 1: Describe the dataflow

You write a program or a SQL query. Flink turns it into a logical graph of operators. Each operator is a box and each arrow is a stream of events.

Step 2: Plan and parallelize

Flink turns the logical graph into a physical plan. Every operator gets a parallelism, which is the number of copies running side by side. Operators that do not need a shuffle are chained into one task to avoid network hops.

Imagine an assembly line. Short steps that can be done by the same worker are merged into one station. When items must be regrouped by customer, they travel to a new station. That regrouping is a shuffle, or a keyBy in Flink.

Step 3: Distribute to workers

A coordinator process, called the JobManager, schedules the tasks. Worker processes, called TaskManagers, run them in slots. A slot is a share of a worker's resources.

Step 4: Process events with keyed state

After keyBy, all events with the same key go to the same task. That task owns the state for the key. It reads state, updates it and emits results.

State lives in a state backend. Two common choices exist.

• A heap backend keeps state as objects in memory. It is fast but limited by memory.
• An embedded key-value store backend keeps state on local disk. It handles very large state at a modest speed cost.

Step 5: Track time

Flink separates three kinds of time.

• Event time is when the thing happened, as recorded in the event itself.
• Ingestion time is when Flink first saw the event.
• Processing time is the wall clock of the machine doing the work.

Event time gives correct answers when data arrives late or out of order. Processing time is simpler and lower latency but depends on timing luck.

Step 6: Use watermarks

How does an engine know that all events up to 10:05 have arrived? It cannot know for sure. Flink uses watermarks for this. A watermark is a special marker that says "I do not expect events older than this timestamp anymore".

A common strategy is bounded out-of-orderness. You tell Flink that events may be up to five seconds late. The watermark then trails the largest seen timestamp by five seconds. When the watermark passes the end of a window, the window fires.

Picture a train timetable. The station master waits a few minutes after the scheduled departure for late passengers, then closes the doors. The waiting rule is the watermark strategy.

Step 7: Group events into windows

A window cuts an endless stream into finite chunks.

• Tumbling windows are fixed size and do not overlap, such as every minute.
• Sliding windows have a fixed size but start more often, such as a ten minute window every minute.
• Session windows close after a gap of inactivity, such as a user going quiet.
• Global windows put everything in one bucket and rely on custom triggers.

Step 8: Take checkpoints

Flink periodically takes a checkpoint, which is a consistent snapshot of all operator state plus the read positions of sources. The method inserts special barriers into the stream. When an operator sees a barrier on all its inputs, it saves its state and passes the barrier on.

If a machine fails, Flink restarts the job from the last completed checkpoint. Sources rewind to the saved positions and state is restored. The result looks as if the failure never happened.

It works like saving a video game. You do not save every second. You save at safe points, and after a crash you reload the latest save and replay.

Step 9: Write results safely

Exactly-once state consistency means each event affects state once. For outputs, Flink needs sink support. Two techniques are common.

• Idempotent writes make a repeated write harmless, for example an upsert by key.
• Two-phase commit writes data in a pending state and commits it only when the checkpoint completes.

With a transactional sink and a replayable source, you get end-to-end exactly-once results.

Step 10: Evolve safely with savepoints

A savepoint is a manually triggered snapshot meant for planned changes. You can stop a job, upgrade the code or change parallelism, and resume from the savepoint. Giving operators stable ids helps Flink map old state to new code.

Key concepts and terms

A worked example

This example counts clicks per user in one minute windows, using Flink SQL. It assumes a Kafka topic named clicks on a fake broker.

Define the source

CREATE TABLE clicks (
  user_id     STRING,
  page        STRING,
  event_ts    TIMESTAMP(3),
  WATERMARK FOR event_ts AS event_ts - INTERVAL '5' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'clicks',
  'properties.bootstrap.servers' = 'broker.example.internal:9092',
  'properties.group.id' = 'click-counter',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'json'
);

The WATERMARK line tells Flink that events may be five seconds late. Everything else is plain column definitions.

Define the sink

CREATE TABLE clicks_per_minute (
  user_id      STRING,
  window_start TIMESTAMP(3),
  window_end   TIMESTAMP(3),
  click_count  BIGINT
) WITH (
  'connector' = 'kafka',
  'topic' = 'clicks_per_minute',
  'properties.bootstrap.servers' = 'broker.example.internal:9092',
  'format' = 'json'
);

For a sink that is an append-only Kafka topic, the query must emit only final window results. That is true for window aggregation.

Run the query

INSERT INTO clicks_per_minute
SELECT user_id, window_start, window_end, COUNT(*) AS click_count
FROM TABLE(
  TUMBLE(TABLE clicks, DESCRIPTOR(event_ts), INTERVAL '1' MINUTE)
)
GROUP BY user_id, window_start, window_end;

What happens inside

1. The source reads JSON records and assigns a watermark to each partition.
2. The group by user turns into a keyBy, so every user lives on one task.
3. Each task keeps a counter per user and per open window.
4. When the watermark passes a window end, the counter is emitted and its state is dropped.
5. Every few seconds, a checkpoint saves the counters and the Kafka offsets.

A numeric walk-through

Suppose these events arrive for one user. The watermark delay is five seconds.

After event 3 the largest timestamp is 10:01:05, so the watermark is 10:01:00. That equals the end of the first window, so the window fires with two events. Event 4 then arrives. It belongs to the closed window and is dropped, unless you set allowedLateness or a side output for late data.

This shows the trade-off. A bigger delay captures more late events but delays results. A smaller delay is faster but drops more late data.

The same job in Python

from pyflink.table import EnvironmentSettings, TableEnvironment

env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
env.get_config().set("execution.checkpointing.interval", "10 s")

env.execute_sql("""
  CREATE TABLE clicks (user_id STRING, event_ts TIMESTAMP(3),
    WATERMARK FOR event_ts AS event_ts - INTERVAL '5' SECOND)
  WITH ('connector'='kafka','topic'='clicks',
        'properties.bootstrap.servers'='broker.example.internal:9092',
        'properties.group.id'='click-counter','format'='json')
""")

result = env.sql_query("""
  SELECT user_id, window_start, COUNT(*) AS n
  FROM TABLE(TUMBLE(TABLE clicks, DESCRIPTOR(event_ts), INTERVAL '1' MINUTE))
  GROUP BY user_id, window_start, window_end
""")
result.execute().print()

Strengths and limitations

Strengths

• True streaming. Events are processed one at a time, not in micro-batches, so latency can be very low.
• Strong state. Large keyed state with incremental checkpoints supports complex logic.
• Correct time handling. Event time, watermarks and late data policies are built in.
• Exactly-once state. Recovery is consistent without custom code.
• One engine for batch and stream. The same SQL can run over a file or a topic.
• Rich connectors. Many sources and sinks, including logs, files, lake table formats and databases.
• Flexible deployment. Runs standalone, on Kubernetes or on managed cloud services.

Limitations

• Operational weight. State, checkpoints and parallelism need tuning. A new team can find it hard.
• Skewed keys. One very hot key can overload a single task.
• Large state costs. Huge state means slow checkpoints and long recovery unless tuned.
• Schema and code changes. Changing state types needs care. Savepoint compatibility is a real project concern.
• Watermark design is a judgment call. Idle partitions can stall watermarks. You must configure idleness handling.
• Not an ad hoc query engine. Flink runs long-lived jobs. For interactive exploration use an engine such as Trino.

How it compares with alternatives

A rule of thumb helps. If you need per-event logic, complex timers or very low latency, look at Flink. If you already run Spark and a few seconds is fine, Structured Streaming may be simpler. If you only need Kafka-to-Kafka transforms inside an application, a library may be enough.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns, handle mixed types such as numbers, categories, dates and text, tolerate missing values and sometimes work even when column names are absent. They are often pre-trained on large collections of synthetic or real tables. At inference time they can use in-context learning: you pass a few labelled rows and the model predicts for new rows without gradient updates. Flink sits upstream and downstream of such models. It does not train them, but it shapes the tables they see.

Turning events into rows

Most business data begins as events. A click, a payment or a sensor reading is a record. A tabular model wants a table where each row is an entity or a moment and each column is a feature.

Flink can build that table continuously. For each user it can maintain counts, sums, last values and time since last event. The output is a row per entity that always reflects the latest state. This is often called an online feature table.

Windowed features with correct time

Training and serving must see the same features. If a training set is built from batch jobs and serving uses a different code path, the numbers disagree, which is known as training-serving skew. Flink's ability to run the same SQL on a bounded history and an unbounded stream helps reduce that skew. Event-time windows also prevent leakage, because a feature for time T uses only events before T.

In-context learning with fresh context rows

A tabular foundation model that uses in-context learning needs a context set of labelled rows. A Flink job can keep that set fresh. For example, it can maintain the most recent labelled examples per segment and publish them to a topic or a store. At prediction time the service pulls the context and the query rows, and the model answers in one forward pass. This links to ideas in TabPFN and TabICL, which rely on context rows.

Missing values and imputation

Streams are messy. Sensors drop out and fields arrive null. Flink can apply simple rules in flight, such as carrying forward the last known value from keyed state or flagging a missing field with a companion indicator column. These flags are useful inputs for foundation models, which often learn their own imputation. For learned approaches see imputation networks and masked cell modeling.

Anomaly detection in motion

Streaming anomaly detection compares each event to a running baseline. Flink can hold rolling statistics per key, such as mean and variance, and flag outliers. It can also call a model service for each row or each small batch. A foundation model can score rows zero-shot, and Flink handles the routing, retries and time alignment. See anomaly detection and isolation forest for the model side.

Continual learning and drift

Data changes over time. Flink can compute drift statistics on live windows, for example the share of nulls or the mean of a column, and compare them to a reference. A drift alert can trigger fine-tuning or a refresh of the context set. This supports continual learning pipelines without retraining blindly.

Sector and semantic classification

Some pipelines classify rows or columns by meaning, for instance assigning an industry sector to a business record. Flink can enrich each event with a model-predicted label by calling an inference endpoint asynchronously. The async I/O operator exists for exactly this: it keeps many requests in flight so the stream does not block on slow calls. Related ideas appear in semantic sector heads.

Schema safety

Tabular models are sensitive to column changes. A renamed or retyped column can silently break predictions. Flink SQL has typed schemas, and combined with a registry such as Schema Registry it can reject or route records that do not match. See also schema evolution and data contract validation.

What Flink does not do

Flink is not a model trainer and not a model server. It does not replace a feature store, although it can fill one. Treat it as the real-time data plane that cleans, joins and shapes tables before and after a model. For the model side, see the architecture and research pages, and the benchmarks page for how tabular models are measured.

Practical guidance and best practices

Start with SQL

Write the first version in Flink SQL. It is shorter, easier to review and often fast enough. Move to the DataStream API only when you need custom timers or state structures.

Design time first

• Decide which timestamp is the event time and make sure sources provide it.
• Choose the watermark delay from real data: measure how late events typically are.
• Handle idle partitions with an idleness timeout, or watermarks may stall.
• Send late events to a side output and count them. Do not silently drop them.

Choose keys with care

• Pick keys with many distinct values to spread load.
• For a hot key, add a salt, compute partial results, then merge them.
• Avoid keying by a field that can change during an entity's life.

Control state size

• Set a time-to-live on state that does not need to live forever.
• Prefer incremental checkpoints with a disk-based backend for big state.
• Avoid unbounded joins. Use interval joins or temporal joins with a bounded range.
• Watch state size per operator in the metrics.

Configure checkpoints

execution.checkpointing.interval: 30 s
execution.checkpointing.mode: EXACTLY_ONCE
execution.checkpointing.min-pause: 10 s
execution.checkpointing.timeout: 10 min
state.backend.type: rocksdb
state.backend.incremental: true
state.checkpoints.dir: s3://example-bucket/flink/checkpoints
state.savepoints.dir: s3://example-bucket/flink/savepoints
restart-strategy.type: exponential-delay

The interval is a balance. Short intervals reduce replay after a crash but add overhead. Start around thirty seconds to a few minutes and measure.

Give operators stable ids

When you use the DataStream API, assign a uid to stateful operators. Without it, a code change can alter generated ids and block restoring from a savepoint.

Plan for deployment

• Use a Kubernetes operator or a managed service for lifecycle handling.
• Run a job in application mode for isolation between jobs.
• Store checkpoints in durable object storage such as Amazon S3.
• Keep high availability enabled for the coordinator.

Observe everything

Track these metrics.

• Consumer lag on the source topics.
• Checkpoint duration, size and failure count.
• Backpressure and busy time per task.
• Watermark lag compared to wall clock.
• Records dropped as late.

Export metrics to Prometheus and chart them in Grafana.

Test before production

• Unit test operators with the test harnesses provided by the project.
• Replay a day of recorded events through the job and compare with a batch result.
• Test restart from a savepoint after every release.
• Inject failures to confirm exactly-once behaviour at the sink.

Writing to lake tables

Flink can write to table formats such as Apache Iceberg and Delta Lake. Streaming writes produce many small files, so schedule compaction. See Iceberg compaction. Commit on checkpoints so files appear atomically.

Common mistakes

1. Using processing time by default. Results change with load and replays. Use event time for business logic.
2. Forgetting watermarks. Without them, event-time windows never fire.
3. Setting the watermark delay by guess. Measure real lateness first.
4. Ignoring idle sources. One quiet partition holds back the whole watermark.
5. Unbounded state. A group by on an ever-growing key with no time-to-live will eventually fill the disk.
6. Hot keys. Most of the traffic goes to one key, so one task is busy while others sit idle.
7. Skipping operator ids. Savepoint restore fails after a code change.
8. Checkpoint interval too short. The job spends its time snapshotting.
9. Non-idempotent sinks without transactions. Retries cause duplicates despite exactly-once state.
10. Blocking calls in operators. A synchronous HTTP call stalls the pipeline. Use async I/O.
11. Treating Flink like a database. Long-lived jobs are not ad hoc query tools.
12. No plan for schema change. Adding a field upstream breaks deserialization.
13. Testing only the happy path. Late data, restarts and duplicates are where bugs live.
14. Setting parallelism higher than the source partitions. Extra source tasks sit idle.

Frequently asked questions

What is Apache Flink used for?

Apache Flink is used for processing events continuously with low latency and reliable state. Typical uses include real-time analytics, fraud and anomaly detection, monitoring of sensors, streaming ETL into a lake or warehouse and building fresh features for machine learning models.

How is Flink different from Spark Streaming?

Flink processes each event as it arrives, while classic Spark streaming groups events into small batches. Flink therefore tends to reach lower latency and has finer control over state and timers. Spark is often chosen when a team already uses it for batch work and can accept latency of a few seconds.

What does exactly-once mean in Flink?

It means that after a failure and recovery, each event is reflected in the managed state exactly once. Flink achieves this with checkpoints and replayable sources. For external outputs you also need an idempotent or transactional sink, otherwise a replay could write a result twice.

What is a watermark?

A watermark is a marker in the stream that states that no events older than a certain timestamp are expected. It lets Flink decide when an event-time window is complete. The delay you configure sets the balance between result freshness and how many late events are still counted.

Can Flink do batch processing too?

Yes. Flink treats a batch as a bounded stream, and the same SQL or program can run in batch execution mode over files or tables. Batch mode can use different scheduling and shuffle strategies that suit finite data.

Do I need Kafka to use Flink?

No. Flink reads from many systems, including files, databases through change data capture, and other message brokers. Kafka is simply the most common source and sink because its replayable log fits Flink's recovery model very well.

How does Flink handle failures?

Flink restarts failed tasks from the most recent completed checkpoint. Sources rewind to the stored positions and operators reload their saved state. A restart strategy controls how often and how quickly it retries, and high availability settings protect the coordinator itself.

What is the difference between a checkpoint and a savepoint?

A checkpoint is taken automatically and is mainly for crash recovery. A savepoint is triggered by an operator on purpose and is meant for planned actions such as upgrades, rescaling and migrations. Savepoints use a more portable format and are kept until you delete them.

Is Flink useful for machine learning pipelines?

It is very useful for the data part. Flink can compute windowed features, join streams, clean values and call model endpoints in flight. It is not a training framework, so training and model serving are usually handled by other tools.

How do I choose between Flink SQL and the DataStream API?

Start with SQL because it is concise and covers joins, windows and aggregations. Choose the DataStream API when you need custom state, complex timers or logic that SQL cannot express cleanly. You can mix both in the same application.

Glossary

• Async I/O: An operator pattern that issues many external requests in parallel without blocking the stream.
• Backpressure: A signal that downstream tasks are slower than upstream tasks, so data must wait.
• Barrier: A marker that flows with the data to define a checkpoint boundary.
• Changelog: A stream of inserts, updates and deletes that describes how a table changes.
• Checkpoint: An automatic snapshot of state and source positions used for recovery.
• Chaining: Running adjacent operators in one task to avoid network transfer.
• DataStream API: The programmatic API for streams with full access to state and time.
• Event time: The time an event occurred, taken from the event data.
• Exactly-once: A guarantee that each event changes state once despite failures.
• JobManager: The coordinator that schedules tasks and manages checkpoints.
• Keyed state: State partitioned by a key.
• Parallelism: The number of parallel instances of an operator.
• Processing time: The local clock time of the machine that handles the event.
• Savepoint: A manual, portable snapshot used for upgrades.
• Side output: A secondary output of an operator, often used for late or invalid events.
• Slot: A unit of resources on a worker where a task runs.
• State backend: The component that stores and checkpoints state.
• TaskManager: A worker process that runs tasks.
• Time-to-live (TTL): A setting that removes state after a given age.
• Watermark: A progress marker for event time.
• Window: A finite group of events defined by time or count.

Summary

Apache Flink is a distributed, stateful stream processing engine. Its main strengths are low latency, event-time correctness, large managed state and exactly-once recovery through checkpoints. Users write SQL or code, and Flink runs it in parallel over many machines.

For tabular foundation models, Flink acts as the real-time data plane. It turns raw events into clean, time-correct rows, keeps context sets and drift statistics fresh, handles missing values in flight and routes rows to models for scoring or labelling. It pairs naturally with Apache Kafka for transport and with lake formats for storage. Learn its time model and checkpoint model first, keep state bounded, and test restarts early. Those habits matter more than any single setting.

Related

It belongs to Batch and streaming processing in my stack, under Data engineering and lakehouse. Others in the same group: