Özer Cem KELAHMETKELAHMET
Topics · Big data

Streaming data: Kafka, Flink, change data capture and real-time pipelines

Streaming processes each event as it arrives instead of waiting for a nightly batch. This page explains the event log, the stream processor and change data capture, and when the extra complexity pays off.

Overview

A streaming platform has two parts. An event log stores every event in order and lets many consumers read it at their own pace. A stream processor reads the log, transforms, joins and aggregates events, and writes results to tables, caches or other topics. Change data capture feeds the log with every change made in operational databases.

The building blocks

Part Role Main options
Event log Durable, ordered, replayable stream of events Apache Kafka, Amazon Kinesis, Apache Pulsar, Amazon MSK
Connectors Move data into and out of the log Kafka Connect, single message transforms
Change data capture Stream every insert, update and delete from a database CDC
Stream processor Stateful transforms, joins, windows Apache Flink, Spark Structured Streaming, Apache Beam
Schemas Keep producers and consumers compatible Schema registry

Ideas that matter

The log is the source of truth. Because events are kept, a new consumer can replay history, and a bug can be fixed by reprocessing. Time has two meanings. Event time is when something happened; processing time is when the system saw it. Windows over event time with watermarks give correct results even when events arrive late. Delivery guarantees. At-least-once is the default; exactly-once needs transactional writes on both ends and costs throughput. State lives in the processor. Flink keeps running totals and join state, checkpointed so it survives failures.

When streaming is worth it

Use streaming when a decision loses value in minutes: fraud checks, stock levels, recommendations, monitoring, or keeping a search index in sync. Keep batch when a daily answer is enough; it is simpler and cheaper. Many platforms write streams into lakehouse tables so both styles share the same data. See real-time systems and event-driven architecture.

Learn more

Read real-time AI, from streams to decisions, then the Kafka and Flink guides. For replication between clusters see MirrorMaker 2.

Frequently asked questions

Is Kafka a database?

It is a durable log, not a query engine. It stores events reliably and in order; queries usually run on tables built from it.

Flink is built for low-latency, stateful streaming; Spark Structured Streaming suits teams already on Spark that can accept micro-batch latency.

Summary

An event log keeps every event, connectors and CDC fill it, a stream processor turns it into results, and a schema registry keeps everyone compatible. Stream where minutes matter; batch everywhere else.

Back to the topic

This page is part of Big data, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open