Apache Kafka

Overview

Apache Kafka is a distributed event streaming platform built around a durable, partitioned log. Producers append records to the end of a log, and consumers read from it at their own pace. Because records are stored on disk and replicated, they can be read again later, by one consumer or by many, and they survive machine failures.

Kafka became the backbone of many data platforms because it decouples the systems that create data from the systems that use it. A payment service does not need to know whether a fraud engine, a warehouse loader or a dashboard will read its events. It writes once to a topic, and any number of readers can pick the data up. Kafka is often paired with Apache Flink for processing and Kafka Connect for moving data in and out.

What it is

Kafka is an open source project of the Apache Software Foundation. It runs as a cluster of servers called brokers. Clients connect to the brokers to write and read records, which are the basic unit of data.

A record has a key, a value, a timestamp and optional headers. The value is usually a small message such as a JSON document or a compact binary format. The key is used to decide placement and to group related events.

Core building blocks

• Topic: a named stream of records, like a folder for one kind of event.
• Partition: a topic is split into partitions, and each partition is an ordered, append-only log.
• Offset: the position of a record within a partition, counted from zero.
• Producer: a client that writes records to topics.
• Consumer: a client that reads records from topics.
• Consumer group: a set of consumers that share the work of reading a topic.
• Broker: a server that stores partitions and serves clients.
• Cluster: the group of brokers working together.
• Replica: a copy of a partition kept on another broker.

A log, not a queue

Many message systems delete a message once it is read. Kafka does not. It keeps records for a configured time or size, or forever, and the reader decides where to start. Reading does not remove anything.

Picture a village notice board where messages are pinned in order. Anyone can walk up and read from any point. Reading a notice does not take it down. Each reader remembers their own place with a bookmark. In Kafka, that bookmark is the offset.

What people use it for

• Moving events between microservices.
• Collecting logs and metrics.
• Capturing database changes for downstream systems.
• Feeding real-time analytics and stream processors.
• Building event sourcing and audit trails.
• Buffering bursts between fast producers and slow consumers.

Where it came from

Kafka was created at a professional networking company around 2010 to solve a data plumbing problem. The company had many systems, such as activity tracking, search, recommendations and monitoring. Each pair of systems needed its own pipe, and the result was a tangle of point-to-point links.

The engineers designed a single central log that every system could write to and read from. They open sourced it, and it became a top level Apache project in the early 2010s. Several of its creators later founded a company that offers commercial services around it, and a large community contributes to the open project.

The design drew on older ideas: write-ahead logs in databases, publish and subscribe messaging, and the log as a unifying abstraction for distributed systems. The idea that a replicated log is the heart of data infrastructure has influenced many later systems.

Evolution in brief

• Early years: a fast log with consumer groups and replication.
• Connect and Streams: a framework for integrating systems and a library for stream processing.
• Delivery guarantees: idempotent producers and transactions, which enable exactly-once processing in many cases.
• Metadata without a separate coordinator: a built-in consensus mode, called KRaft, replaced the external coordination service that older clusters needed.
• Tiered storage and more: moving older data to cheaper storage while keeping the log abstraction.

Check the project documentation for what is current in your version.

How it works

This section follows a record from producer to consumer.

Step 1: A producer creates a record

The application builds a record with a topic name, an optional key, a value and optionally headers. It passes the record to a producer client.

Step 2: Serialization

Kafka brokers store bytes. The client must convert the key and value into bytes. This is serialization. Common choices are JSON, Avro and Protobuf. Binary formats with schemas are smaller and safer. They are usually used with a Schema Registry.

Step 3: Choose a partition

The producer decides which partition receives the record.

• If the record has a key, the producer hashes the key. The same key always goes to the same partition.
• If there is no key, the producer spreads records across partitions, often in batches.
• A custom partitioner can implement other rules.

Same key, same partition, matters a lot. It guarantees that all events for one customer or one device are stored in order.

Think of a post office with numbered sorting bins. The postcode decides the bin. All letters for one postcode land in one bin, in the order they arrived.

Step 4: Batch and send

The producer groups records into batches per partition. It waits a short time to fill a batch, which is controlled by linger.ms and batch.size. Batches can be compressed to save network and disk. This makes Kafka efficient at high throughput.

Step 5: The leader appends to its log

Each partition has one leader replica and zero or more followers. The producer sends the batch to the leader. The leader appends it to the end of its log on disk and assigns offsets.

Kafka relies on the operating system page cache and on sequential disk writes. Sequential I/O is fast even on ordinary disks. This is part of why Kafka can sustain very high rates.

Step 6: Replication

Followers fetch new records from the leader and append them to their own logs. The set of followers that are caught up is called the in-sync replicas, or ISR.

The producer chooses how much acknowledgement it wants through the acks setting.

• acks=0: do not wait. Fastest, but data can be lost.
• acks=1: wait for the leader only.
• acks=all: wait for all in-sync replicas.

Combined with min.insync.replicas, acks=all ensures a record is on several machines before it is confirmed.

Step 7: High watermark and visibility

A record becomes visible to consumers only after all in-sync replicas have it. The point up to which records are safe is the high watermark. If the leader fails, one of the in-sync followers becomes the new leader and no acknowledged record is lost.

Step 8: Consumers pull

Consumers ask brokers for records starting at an offset. This is a pull model, so consumers control their own pace. Slow consumers do not slow down producers. They simply fall behind, and the gap is called lag.

Step 9: Consumer groups share the work

Consumers with the same group.id form a consumer group. Kafka assigns each partition to exactly one consumer in the group. If a topic has six partitions and three consumers in a group, each reads two partitions.

Different groups are independent. A warehouse loader and a fraud service can each read the full topic in their own groups without affecting each other.

The number of partitions sets the maximum parallelism of a group. Extra consumers beyond the partition count sit idle.

Step 10: Commit offsets

A consumer periodically commits its offset, which stores "I have processed up to here" in an internal topic. After a restart, the consumer resumes from the last commit.

The order of processing and committing defines the delivery guarantee.

• At most once: commit first, then process. A crash can lose records.
• At least once: process first, then commit. A crash can cause repeats.
• Exactly once: use transactions or idempotent sinks so repeats have no effect.

Step 11: Retention and compaction

Records are removed by policy.

• Time or size retention: delete segments older than a limit.
• Log compaction: keep only the latest record per key. This turns a topic into a table-like changelog.

Compacted topics are common for reference data, such as the current profile of every user.

Step 12: Rebalancing and failure handling

When a consumer joins or leaves a group, partitions are reassigned in a rebalance. Newer protocols make rebalances less disruptive. Brokers also detect failures and elect new leaders automatically.

Key concepts and terms

A worked example

This example creates a topic, writes events and reads them with a consumer group. All hosts are fake.

Create a topic

kafka-topics.sh --bootstrap-server broker1.example.internal:9092 \
  --create --topic payments \
  --partitions 6 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

This topic has six partitions, three copies of each and a seven day retention. With min.insync.replicas=2 and acks=all, a write needs two copies before it is confirmed.

Produce records in Python

import json
from confluent_kafka import Producer

p = Producer({
    "bootstrap.servers": "broker1.example.internal:9092",
    "acks": "all",
    "enable.idempotence": True,
    "compression.type": "lz4",
    "linger.ms": 20,
})

def on_delivery(err, msg):
    if err:
        print("failed:", err)
    else:
        print(f"ok partition={msg.partition()} offset={msg.offset()}")

for i in range(5):
    event = {"payment_id": i, "account": f"acct-{i % 2}", "amount": 10.5 * (i + 1)}
    p.produce("payments", key=event["account"], value=json.dumps(event), callback=on_delivery)

p.flush()

The key is the account. All payments for acct-0 go to one partition and keep their order.

Consume with a group

from confluent_kafka import Consumer

c = Consumer({
    "bootstrap.servers": "broker1.example.internal:9092",
    "group.id": "fraud-checker",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
})
c.subscribe(["payments"])

try:
    while True:
        msg = c.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print("error:", msg.error())
            continue
        # process the record here
        print(msg.key(), msg.value())
        c.commit(message=msg, asynchronous=False)
finally:
    c.close()

Committing after processing gives at-least-once behaviour. If the process dies between handling and committing, the record is read again, so the handler should be safe to repeat.

Inspect the group

kafka-consumer-groups.sh --bootstrap-server broker1.example.internal:9092 \
  --describe --group fraud-checker

The output lists each partition, the current offset, the log end offset and the lag.

A numeric walk-through

Suppose six partitions each receive 5,000 records per second, so 30,000 per second in total. Each record is 1 kilobyte. That is about 30 megabytes per second of incoming data. With a replication factor of three, the cluster writes roughly 90 megabytes per second across brokers. Over a day the topic receives about 2.6 terabytes of raw data before compression. With a retention of seven days and a compression ratio of three, the topic would need roughly 6 terabytes of disk in total, counting replicas. Numbers like these drive broker count and disk planning.

Ordering example

Events for key acct-0 arrive in order: deposit, withdrawal, withdrawal. All three go to one partition. A consumer sees them in the same order. Events for acct-1 live in another partition and may be read earlier or later relative to acct-0. Kafka guarantees order within a partition, not across partitions.

Strengths and limitations

Strengths

• High throughput. Batching, sequential I/O and compression allow very large rates.
• Durability. Replicated logs on disk survive broker failures.
• Replay. Consumers can rewind to reprocess history after a bug or for a new use case.
• Decoupling. Producers and consumers do not need to know each other.
• Scalability. Add partitions and brokers to scale out.
• Ordering per key. Related events stay in sequence.
• Large ecosystem. Connectors, stream processors, schema tools and client libraries in many languages.
• Multiple readers. Many independent groups can read the same data.

Limitations

• Operational complexity. Brokers, storage, networking and upgrades need real expertise, though managed services such as Amazon MSK reduce it.
• Not a database. Querying by arbitrary field is not supported. You read by topic, partition and offset.
• No global order. Order exists only within a partition.
• Partition count is hard to change well. Increasing partitions changes key placement for new data.
• Large messages are awkward. The default limits are around one megabyte. Store big payloads elsewhere and send a reference.
• Rebalance pauses. Group changes can pause consumption briefly.
• Not ideal for low volume. For a handful of messages a day, a simple queue may be easier.
• Complex semantics. Exactly-once needs care across the whole pipeline.

How it compares with alternatives

Use Kafka when you need a durable, replayable stream consumed by many systems. Use a classic queue when you need work distribution with per-message acknowledgement and flexible routing. See event-driven architecture for how these pieces fit together.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They process rows and columns, mixed types such as numbers, categories, dates and text, missing values and sometimes columns with no useful names. They are often pre-trained on large numbers of synthetic or real tables, and many use in-context learning with labelled example rows passed at prediction time. Kafka does not train or run such models. Its role is to carry the data that becomes their tables, in a way that is ordered, durable and replayable.

From events to rows

Almost every business table starts as a stream of events. A record in a Kafka topic can be one row, or a change to a row. A downstream job, such as one built with Apache Flink, joins and aggregates topics into one row per entity. That row is the unit a tabular model reads.

Change data capture as a table feed

Databases can publish row-level changes to Kafka. See change data capture. Compacted topics then hold the latest version of every row, which behaves like a live copy of the source table. Models and feature jobs can read the compacted topic and rebuild the table at any time.

Replay for reproducible training sets

Because Kafka keeps history, you can rebuild a training table as of a given time by replaying from a timestamp or offset. That helps avoid leakage, since features use only events that existed at the time. It also lets you test a changed feature definition against the same history.

Fresh context for in-context learning

A model that uses in-context examples needs a representative set of labelled rows. A stream job can maintain the latest labelled rows per segment and publish them to a compacted topic. A prediction service reads the topic to refresh its context, then combines it with query rows. Compare this idea with TabPFN and TabICL.

Missing values and late data

Streams carry nulls, delayed fields and corrections. Kafka keeps the original facts, so a later correction becomes a new record rather than a silent overwrite. Downstream jobs can emit missing-value indicators next to values. Tabular models often learn their own imputation, so preserving the null and adding a flag usually works better than inventing values early. See imputation networks and masked cell modeling.

Schemas and column names

Tabular models are sensitive to schema changes. A Schema Registry enforces compatibility rules, such as allowing new optional fields but blocking type changes. This keeps columns stable for model inputs. Records carry field names, which can be kept for models that use names as hints or stripped for models that ignore them. See schema evolution.

Scoring streams in real time

A common pattern is: consume events, build a row, call a model service, publish the prediction to another topic. This supports real-time anomaly detection and live classification, including labelling business records by sector or meaning. See semantic sector heads. Predictions are published with the original key so they can be joined to the source.

Feedback and continual learning

Predictions and later true outcomes can both be written to topics. A job joins them to create fresh labelled data. This feeds evaluation, drift detection and continual learning loops, where context sets or fine-tuned models are refreshed on a schedule.

Dead letter topics

Rows that fail validation, such as wrong types or impossible values, can be routed to a separate topic. Data teams review them and fix the producer. This prevents bad rows from polluting a training table or a context set.

Limits of the role

Kafka stores and moves records. It does not compute joins by itself, hold feature definitions or serve models. Pair it with a processor, a store and a serving layer. For the model side, see the architecture, research and benchmarks pages.

Practical guidance and best practices

Design topics deliberately

• Use clear names, such as domain.entity.event.
• Keep one kind of event per topic, unless order across kinds matters.
• Choose keys that match how consumers need ordering.
• Avoid keys with a few very hot values, which cause skewed partitions.

Size partitions

• Choose enough partitions for the peak parallelism you expect, with some headroom.
• Do not create thousands of partitions for tiny topics. Each partition has overhead.
• Remember that adding partitions later changes the key to partition mapping for new records.

Configure producers for safety

acks=all
enable.idempotence=true
retries=2147483647
delivery.timeout.ms=120000
max.in.flight.requests.per.connection=5
compression.type=lz4
linger.ms=10

These settings favour durability and avoid duplicates from retries while keeping per-partition order.

Configure topics for safety

replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

This combination tolerates the loss of one broker without losing acknowledged data.

Configure consumers carefully

• Disable auto commit if you need control, and commit after processing.
• Make handlers idempotent so repeats are harmless.
• Tune max.poll.records and processing time so the consumer is not kicked out of the group.
• Handle poison messages with retries and a dead letter topic.

Use schemas

Register Avro or Protobuf schemas and enforce compatibility. Plain JSON without a contract leads to silent breaking changes. See Schema Registry.

Secure the cluster

• Encrypt traffic with TLS.
• Authenticate clients with SASL or mutual TLS.
• Use access control lists per topic and per group.
• Keep secrets in a secrets manager and rotate them.

Monitor the essentials

• Consumer lag per group and partition.
• Under-replicated partitions.
• Request latency and throughput.
• Disk usage and log segment growth.
• Controller changes and leader elections.

Export metrics to Prometheus and use Grafana for dashboards and alerts.

Move data in and out without custom code

Use Kafka Connect for databases, object storage and other systems. Use MirrorMaker 2 to replicate between clusters. Land data in Amazon S3 or lake tables for long-term analytics.

Plan for disaster recovery

Decide whether you need a second cluster in another region. Replicate topics and consumer offsets. Test failover, since untested plans tend to fail.

Common mistakes

1. Assuming global ordering. Kafka orders only within a partition.
2. Poor key choice. A constant or low-variety key sends all traffic to one partition.
3. Too few partitions. The group cannot scale beyond the partition count.
4. Too many partitions. Overhead and recovery times grow.
5. Replication factor of one. A single disk failure loses data.
6. Using acks=1 for important data. A leader crash right after acknowledging can lose records.
7. Auto commit with slow processing. Offsets are committed before work is done, which loses data on a crash.
8. Non-idempotent consumers. At-least-once delivery then creates duplicate effects.
9. No schema discipline. One producer change breaks every consumer.
10. Sending huge messages. Large payloads hurt throughput and hit size limits.
11. Ignoring lag. Slow consumers drift further behind until retention deletes records they still need.
12. Treating Kafka as a query database. Use a store designed for lookups.
13. Retention shorter than recovery time. If a consumer is down longer than retention, data is gone.
14. No dead letter handling. One bad record blocks a partition forever.
15. Skipping capacity tests. Surprises appear at peak traffic.

Frequently asked questions

What is Apache Kafka used for?

Kafka is used to move and store streams of events between systems. Typical uses are connecting microservices, collecting logs and metrics, capturing database changes, feeding real-time analytics and building pipelines into data lakes and warehouses.

Is Kafka a message queue?

Not exactly. A traditional queue removes a message after it is consumed, while Kafka keeps records in a log for a retention period and lets many consumers read them independently. You can use it like a queue with a single consumer group, but its design is closer to a distributed commit log.

What is a partition in Kafka?

A partition is an ordered, append-only slice of a topic. Partitions let Kafka spread a topic over many brokers and let many consumers read in parallel. Order is guaranteed within a partition, and records with the same key land in the same partition.

What is an offset?

An offset is the sequence number of a record within a partition. Consumers use offsets to track their progress. Committing an offset tells Kafka where to resume after a restart.

What is a consumer group?

A consumer group is a set of consumers that cooperate to read a topic. Kafka gives each partition to only one consumer in the group, so the work is shared. Different groups read the same data independently of each other.

Does Kafka guarantee exactly-once delivery?

Kafka supports exactly-once processing in many pipelines through idempotent producers and transactions. The guarantee covers reading from and writing to Kafka topics. When the pipeline touches an external system, that system must also be idempotent or transactional to avoid duplicates.

How long does Kafka keep data?

For as long as you configure it. Retention can be based on time or size, and a compacted topic keeps the latest record for each key indefinitely. Some deployments also tier older data to cheaper storage while keeping it readable.

Do I still need ZooKeeper?

Newer Kafka versions use a built-in consensus mode called KRaft to manage metadata, so an external coordination service is no longer required. Older clusters still run with the external service, and migration paths exist. Check the documentation for your version.

How is Kafka different from Flink?

Kafka stores and transports streams of records. Flink processes them: it joins, aggregates, windows and transforms events with state. They are often used together, with Kafka as the durable source and sink and Flink as the compute layer.

How can Kafka support machine learning on tabular data?

It delivers fresh, ordered, replayable events that processing jobs turn into rows and features. Teams use it to rebuild training tables at a past time, keep context sets current, score events in real time and collect predictions and outcomes for monitoring and retraining.

Glossary

• Acknowledgement (acks): A producer setting for how many replicas must confirm a write.
• Broker: A Kafka server that stores partitions and serves clients.
• Commit: Saving a consumer's progress as an offset.
• Compaction: Keeping only the latest record for each key.
• Consumer group: Consumers that share the partitions of a topic.
• Controller: The component that manages metadata and leader election.
• Dead letter topic: A topic that holds records that could not be processed.
• Follower: A replica that copies a partition leader.
• High watermark: The offset up to which records are fully replicated and visible.
• Idempotence: The property that repeating an operation has the same effect as doing it once.
• ISR: The in-sync replicas of a partition.
• KRaft: Kafka's built-in metadata consensus mode.
• Lag: The distance between a consumer's position and the end of the log.
• Leader: The replica that handles client traffic for a partition.
• Offset: A record's position in a partition.
• Partition: An ordered log that makes up part of a topic.
• Producer: A client that writes records.
• Rebalance: Reassigning partitions among consumers in a group.
• Replication factor: The number of copies of each partition.
• Retention: The rule for how long records are kept.
• Segment: A file chunk of a partition log.
• Topic: A named category of records.

Summary

Apache Kafka is a distributed, replicated log that decouples producers from consumers. Records are appended to partitions, ordered within each partition, stored durably and read by consumer groups at their own pace. Replication, retention and replay make it a reliable backbone for event-driven systems and data pipelines.

For tabular foundation models, Kafka is the transport and memory of the data plane. It turns business events into the ordered streams from which rows and features are built, supports change data capture as a live copy of source tables, lets you rebuild training sets as of any time, keeps context sets and predictions flowing, and isolates bad rows in dead letter topics. Start with careful keys, enough partitions, safe producer and topic settings and schema discipline, then add processing, connectors and monitoring around it.

Related

It belongs to Event streaming in my stack, under Data engineering and lakehouse. Others in the same group: