Streaming data: Kafka, Flink, change data capture and real-time pipelines
Streaming processes each event as it arrives instead of waiting for a nightly batch. This page explains the event log, the stream processor and change data capture, and when the extra complexity pays off.
Overview
A streaming platform has two parts. An event log stores every event in order and lets many consumers read it at their own pace. A stream processor reads the log, transforms, joins and aggregates events, and writes results to tables, caches or other topics. Change data capture feeds the log with every change made in operational databases.
The building blocks
| Part | Role | Main options |
|---|---|---|
| Event log | Durable, ordered, replayable stream of events | Apache Kafka, Amazon Kinesis, Apache Pulsar, Amazon MSK |
| Connectors | Move data into and out of the log | Kafka Connect, single message transforms |
| Change data capture | Stream every insert, update and delete from a database | CDC |
| Stream processor | Stateful transforms, joins, windows | Apache Flink, Spark Structured Streaming, Apache Beam |
| Schemas | Keep producers and consumers compatible | Schema registry |
Ideas that matter
The log is the source of truth. Because events are kept, a new consumer can replay history, and a bug can be fixed by reprocessing. Time has two meanings. Event time is when something happened; processing time is when the system saw it. Windows over event time with watermarks give correct results even when events arrive late. Delivery guarantees. At-least-once is the default; exactly-once needs transactional writes on both ends and costs throughput. State lives in the processor. Flink keeps running totals and join state, checkpointed so it survives failures.
When streaming is worth it
Use streaming when a decision loses value in minutes: fraud checks, stock levels, recommendations, monitoring, or keeping a search index in sync. Keep batch when a daily answer is enough; it is simpler and cheaper. Many platforms write streams into lakehouse tables so both styles share the same data. See real-time systems and event-driven architecture.
Learn more
Read real-time AI, from streams to decisions, then the Kafka and Flink guides. For replication between clusters see MirrorMaker 2.
Frequently asked questions
Is Kafka a database?
It is a durable log, not a query engine. It stores events reliably and in order; queries usually run on tables built from it.
Flink or Spark for streaming?
Flink is built for low-latency, stateful streaming; Spark Structured Streaming suits teams already on Spark that can accept micro-batch latency.
Summary
An event log keeps every event, connectors and CDC fill it, a stream processor turns it into results, and a schema registry keeps everyone compatible. Stream where minutes matter; batch everywhere else.
Back to the topic
This page is part of Big data, which maps every concept, tool, simulation and article on the subject.