Amazon MSK

Overview

Amazon MSK, short for Managed Streaming for Apache Kafka, is an AWS service that runs Apache Kafka clusters for you. Kafka is a distributed log used to move streams of events between systems. MSK creates the brokers, spreads them across availability zones, patches them, replaces failed nodes, and connects them to AWS security and monitoring. You still use the normal Kafka clients and tools.

The value is simple: you get event streaming without running the brokers yourself. Applications publish records to topics, and other applications read them, at their own pace, with ordering kept inside each partition. For teams working with tabular data, a stream is a table that never stops growing. Rows arrive as events, and downstream jobs build features, scores and alerts from them. See /model/architecture for how tabular foundation models read rows and columns, and see the role section below for how streams feed them.

What it is

MSK is a managed control plane and data plane for Apache Kafka. It offers different ways to run Kafka and a few companion services.

Main flavours

• Provisioned clusters: you choose the broker instance type, number of brokers and storage. You have the most control over capacity and configuration.
• Serverless clusters: you create a cluster and the service scales capacity with your traffic, within limits. It trades some configuration freedom for simplicity.
• MSK Connect: a managed way to run Kafka Connect workers and connectors. See Kafka Connect.
• MSK Replicator: a feature for copying topics and consumer offsets between MSK clusters, such as across regions. See MSK Replicator.

Pieces you work with

• Broker: a Kafka server that stores partitions and serves reads and writes.
• Topic: a named stream of records, divided into partitions.
• Producer: a client that writes records.
• Consumer: a client that reads records, usually as part of a consumer group.
• Schema registry: a service that stores record schemas. See Schema Registry.
• Metrics and logs: broker logs and metrics sent to CloudWatch, S3 or other places.

An everyday analogy

Picture a busy postal sorting office with long conveyor belts. Each belt is a partition, and each labelled group of belts is a topic. Senders drop parcels on the belt. Parcels stay on the belt for a set time, and any number of readers can walk beside the belt and look at them in order, each with a bookmark showing how far they have read. MSK is a company that builds and maintains the sorting office, replaces broken belts, and puts locks on the doors. You bring the parcels and the readers.

Where it came from

Apache Kafka was created at a large professional network company in the early 2010s to handle activity data and log pipelines, and was then open sourced and given to the Apache Software Foundation. Its core idea is a replicated, append-only commit log that many consumers can read independently. It spread widely because it unified messaging, log collection and stream processing in one system. See Apache Kafka.

Running Kafka well takes skill. Operators manage broker sizing, disk, replication, upgrades, coordination services, security and monitoring. Amazon MSK was introduced in the late 2010s to take over that work on AWS. Since then the service has added options such as serverless clusters, tiered storage, more authentication modes, managed connectors, a replicator and support for newer Kafka versions, including the replacement of the older ZooKeeper coordination layer with Kafka's own consensus mode in newer versions. Exact versions and dates change often, so check the official documentation.

How it works

Step 1: Create a cluster

You choose a cluster type and a Kafka version. For provisioned clusters, you pick the broker instance size, the number of brokers per zone and the storage per broker. The service places brokers in subnets across several availability zones inside your virtual network.

Analogy: you tell the builder how big a sorting office you want and in which neighbourhoods.

Step 2: Secure access

Clients connect to broker endpoints inside your network. You choose how clients prove who they are and what they may do:

• IAM-based access control, using AWS identities.
• SASL with SCRAM, with credentials stored in a secrets service.
• Mutual TLS, with client certificates.
• Unauthenticated access, only for tests inside a locked network.

Traffic is encrypted in transit with TLS, and data at rest is encrypted with a key you can manage.

Step 3: Create topics

A topic has a number of partitions and a replication factor. Partitions let a topic spread over brokers and let consumers read in parallel. Replication keeps copies of each partition on several brokers so that a broker failure does not lose data.

Step 4: Produce records

A producer sends records, each with an optional key, a value, a timestamp and headers. If a key is given, records with the same key go to the same partition, which keeps their order. Producers choose acknowledgement levels that trade latency for safety.

Step 5: Consume records

A consumer group is a set of consumers that share a topic. Each partition is read by one consumer in the group at a time. The group stores its offsets, which mark how far it has read. If a consumer fails, others take over its partitions. A new group can start from the beginning and replay history while retained.

Step 6: Retention and storage

Kafka keeps records for a configured time or size, not until they are read. With tiered storage, older data can move to cheaper storage while recent data stays on broker disks. Compaction can keep only the latest record per key for topics that act like tables of current state.

Step 7: Monitor and scale

MSK publishes metrics for brokers, topics and consumer lag. Lag is the gap between the newest record and the group's position. Rising lag signals slow consumers. You can scale storage, change broker size or add brokers, and rebalance partitions. Serverless clusters scale within their limits automatically.

Delivery guarantees in simple terms

Key concepts and terms

A worked example

A retail team wants to stream point-of-sale events into Kafka, keep them for seven days, and write them to S3 as Parquet for analysis. All names are fake.

Step 1: Create a topic

kafka-topics.sh --bootstrap-server b-1.example.internal:9098 \
  --command-config client.properties \
  --create --topic pos-events \
  --partitions 12 --replication-factor 3 \
  --config retention.ms=604800000

The client properties file holds the connection and authentication settings. With IAM authentication it looks like this:

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Step 2: Produce events

import json, time
from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers=["b-1.example.internal:9098"],
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
    key_serializer=lambda k: k.encode("utf-8"),
    acks="all",
    linger_ms=20,
)

event = {
    "event_id": "e-0001",
    "store_id": "s-17",
    "customer_id": "c-4821",
    "amount": 23.40,
    "ts": int(time.time() * 1000),
}
producer.send("pos-events", key=event["customer_id"], value=event)
producer.flush()

The key is the customer id, so all events for one customer land in one partition and stay ordered.

Step 3: Consume and aggregate

from kafka import KafkaConsumer
import json
from collections import defaultdict

consumer = KafkaConsumer(
    "pos-events",
    bootstrap_servers=["b-1.example.internal:9098"],
    group_id="spend-aggregator",
    auto_offset_reset="earliest",
    enable_auto_commit=False,
    value_deserializer=lambda b: json.loads(b.decode("utf-8")),
)

spend = defaultdict(float)
for msg in consumer:
    ev = msg.value
    spend[ev["customer_id"]] += ev["amount"]
    consumer.commit()

Committing offsets after processing gives at-least-once behaviour. A production job would batch commits and handle duplicates by event id.

Step 4: Sink to S3 with a connector

A managed connector can write topic data to S3 in files. A typical configuration, shown with placeholder values:

connector.class=io.example.connect.s3.S3SinkConnector
tasks.max=4
topics=pos-events
s3.bucket.name=example-stream-landing
s3.region=us-east-1
format.class=io.example.connect.formats.ParquetFormat
flush.size=100000
rotate.interval.ms=600000
partitioner.class=io.example.connect.partitioner.TimeBasedPartitioner
path.format='dt'=YYYY-MM-dd

The class names above are placeholders. Use the real class of the connector you choose.

A numeric walk-through of sizing

Suppose the topic receives 5,000 events per second, each about 400 bytes. That is 2 MB per second, or about 172 GB per day. With seven days of retention that is about 1.2 TB, and with a replication factor of three it is about 3.6 TB across brokers. If each consumer instance handles 1,000 events per second, the group needs at least five consumers, so 12 partitions give room to grow to 12 consumers. These figures show why retention, replication and partition count are chosen together.

Stream processing on top of Kafka

Kafka stores and moves records. Computation happens in clients or engines.

• Plain consumers are enough for simple filters and counters.
• Kafka Streams is a library for stateful processing inside your own application.
• Apache Flink runs large stateful jobs with event-time windows and exactly-once state.
• Spark streaming reads topics in micro-batches and shares code with batch jobs.

A typical windowed aggregation, in Spark style:

from pyspark.sql import functions as F

events = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "b-1.example.internal:9098")
    .option("subscribe", "pos-events")
    .load())

parsed = events.select(F.from_json(F.col("value").cast("string"), schema).alias("e")).select("e.*")

agg = (parsed
    .withWatermark("ts_ts", "10 minutes")
    .groupBy(F.window("ts_ts", "5 minutes"), "store_id")
    .agg(F.sum("amount").alias("sales")))

A watermark tells the engine how late an event may arrive before a window is closed. Late data is a normal fact of streams.

Event time and processing time

Event time is when something happened. Processing time is when your code saw it. They differ because of network delays, retries and outages. For tables built from streams, always keep the event time as a column. Models and features that depend on time must use event time, or they will mix up the order of events.

Common topic design patterns

Strengths and limitations

Strengths

• Real Kafka compatibility, so the ecosystem of clients and tools works.
• Brokers, patching and failure replacement are handled for you.
• Multi-zone placement for availability.
• Integrates with AWS identity, networking, encryption and monitoring.
• Choice between provisioned control and serverless simplicity.
• Managed connectors and a replicator for common tasks.
• Replayable log: new consumers can reread history within retention.

Limitations

• You still design topics, partitions, keys and retention. The service does not do that for you.
• Not a database. Querying by arbitrary column is not what Kafka is for.
• Partition count is hard to reduce, and changing keys changes ordering.
• Consumers must handle duplicates, schema changes and reprocessing.
• Clusters live in your private network, so clients need network access.
• Costs depend on broker size, storage and data transfer. This article gives no prices.
• Some Kafka settings and plugins are restricted compared with self-managed clusters.

How it compares with alternatives

Rule of thumb: choose MSK when you want Kafka's protocol, tools and ecosystem and do not want to run brokers. Choose a simpler queue when you only need to pass tasks between services.

Cluster layout, in words

A production provisioned cluster often has three brokers per zone-set, spread across three availability zones, with a topic replication factor of three and a minimum of two in-sync replicas. A producer using acks=all then needs two copies to confirm a write. If one zone fails, writes and reads continue. If two fail, writes stop to protect data. This design chooses safety over availability in the worst case.

Upgrades and maintenance

The service applies patches and supports version upgrades. Prepare by checking client compatibility, testing in a non-production cluster, and doing upgrades in a quiet window. Clients should tolerate brief leader changes by retrying. A rolling process means brokers restart one at a time, so a well-configured client sees a short pause at most.

Data transfer and network cost awareness

Traffic between zones and out of the region can add cost. Rack-aware consumer settings let consumers read from a replica in their own zone when supported, which reduces cross-zone traffic. Keep consumers near the cluster and compress records with a format such as lz4 or zstd to cut network and storage use.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They work with rows and columns of mixed types, cope with missing values, may see columns without helpful names, and predict from example rows given in context. They are pre-trained on synthetic or real tables and can support imputation, anomaly detection, continual learning and semantic or sector classification. See /model/research and /model/benchmarks.

A Kafka stream is the live form of a table. Each record is a row, and the fields of the record are the columns. MSK does not run such models, but it moves, stores and replays the rows they need.

Streaming rows into context

In-context learning needs a recent set of labelled example rows. A consumer can maintain a rolling window of the latest labelled events per entity, write it to a store, and hand it to a model call. As new events arrive, the context updates without any change to model weights. This is a practical form of keeping predictions fresh.

Scoring events as they arrive

For anomaly detection, a service can read each event, send a small batch to a model, and publish an anomaly score to another topic. Alerts then flow from that topic. Batching rows into groups before calling a model is important, because tabular models are more efficient on batches than on single rows.

Imputation and missing values in streams

Events often arrive with fields missing. A stream processor can attach a missing-value mask, send the row to a model for imputation, and publish both the original and the filled version. Keeping the original lets you audit the fill.

Schemas and column meaning

A model that sees columns without clear names depends on stable positions and types. Schema evolution in a stream can break that. Use Schema Registry and compatibility rules so that adding or renaming fields does not silently shift what the model reads. See also Schema evolution and Schema drift detection.

Building training and evaluation tables

Streams can be landed to S3 with Kafka Connect, then turned into daily tables for evaluation and pre-training collections. The log's timestamps make it easy to build point-in-time correct tables that avoid leakage: for each prediction time, only include events before it. Replaying a topic from an offset reproduces the exact input of a past run.

Continual learning

When a model is updated over time, a stream provides the new data in order and with timestamps. A job can pull the latest slice, check it against a data contract with Data contract validation, and pass it to an update or evaluation step. Offsets and snapshots give a clean way to record which data a version saw.

Governance and privacy

Sensitive fields can be tokenized in a stream processor before they reach a model, using ideas from Data tokenization. Topic-level access controls and encryption limit who can read raw events. Lineage records help show which topics fed which tables, see Data lineage.

Example: a model scoring loop

The sketch below reads events in batches, calls a scoring function, and publishes results. The scoring call stands for any tabular model service. Batching keeps the call efficient.

from kafka import KafkaConsumer, KafkaProducer
import json

consumer = KafkaConsumer("pos-events", bootstrap_servers=["b-1.example.internal:9098"],
                         group_id="anomaly-scorer", enable_auto_commit=False,
                         value_deserializer=lambda b: json.loads(b.decode()))
producer = KafkaProducer(bootstrap_servers=["b-1.example.internal:9098"],
                         value_serializer=lambda v: json.dumps(v).encode())

def score_batch(rows):
    # placeholder for a call to a model endpoint at https://models.example.internal/score
    return [0.0 for _ in rows]

while True:
    batch = consumer.poll(timeout_ms=1000, max_records=500)
    rows = [m.value for msgs in batch.values() for m in msgs]
    if not rows:
        continue
    scores = score_batch(rows)
    for row, s in zip(rows, scores):
        producer.send("pos-anomaly-scores", {"event_id": row["event_id"], "score": s})
    producer.flush()
    consumer.commit()

Notice three habits: batch before scoring, flush output before committing input offsets, and carry the event id so results can be joined back to rows.

Backfills and replays

If a model version changes, you may want to rescore history. Start a new consumer group at the earliest retained offset, or read the landed files from the lake. Keep the old and new scores in different columns or topics so that comparisons are possible.

Practical guidance and best practices

Design topics with care

• Choose keys that match the ordering you need, such as customer id or device id.
• Pick a partition count with growth in mind, because it is hard to reduce.
• Name topics with a clear convention, such as domain.entity.event.
• Keep one event type per topic, or use a schema that makes types clear.
• Use compaction for topics that hold the latest state per key.

Size the cluster

• Estimate throughput, retention and replication to get storage needs.
• Use at least three brokers across three zones for production availability.
• Keep an eye on disk use and set alarms before it fills.
• Use tiered storage when long retention is needed but compute is not.
• Start with serverless for spiky or unknown loads if its limits fit.

Secure it

• Prefer IAM or mutual TLS over open access.
• Give each application only the topics it needs, with read or write rights.
• Encrypt in transit and at rest.
• Keep clusters in private subnets and control inbound rules.
• Rotate credentials stored in the secrets service.

Write robust producers and consumers

• Use acknowledgements from all in-sync replicas for important data.
• Enable idempotent producers to avoid duplicate writes on retry.
• Commit offsets after processing, and make handlers idempotent.
• Handle poison records with a dead letter topic.
• Use a schema registry and a compatibility mode.

Monitor

• Watch consumer lag per group and partition.
• Watch under-replicated partitions, broker CPU, disk and network.
• Alert on produce and fetch error rates.
• Track end-to-end latency with a heartbeat topic.

Integrate with the lake

Land streams to S3 as Parquet for batch work, and read them with Amazon Athena or EMR Serverless. Consider Apache Iceberg tables for streaming ingestion with reliable commits. Bring in bulk historical files through AWS DataSync.

Quick troubleshooting guide

Common mistakes

• Too few partitions. Parallelism is capped by the partition count. Growing later changes key placement.
• Too many partitions. Thousands of tiny partitions add overhead and slow recovery.
• Poor keys. A hot key sends most traffic to one partition. A missing key loses ordering.
• Using Kafka as a database. Long retention with ad hoc queries is better served by a lake or a database fed from the stream.
• Ignoring duplicates. At-least-once delivery means consumers must be idempotent.
• No schema discipline. A producer that renames a field can break every consumer.
• Running with replication factor one. A broker failure then loses data.
• Ignoring consumer lag. Lag grows silently until retention deletes unread data.
• Large messages. Very large records strain brokers. Store big payloads in object storage and send a reference.
• Open network access. Brokers should not be reachable from the public internet.

• Skipping tests for rebalances. Consumers that cannot handle partition moves cause pauses and duplicates.

• Committing offsets before processing. A crash then loses records silently.

• Reusing one consumer group for unrelated jobs. They steal partitions from each other.
• Skipping event time. Windows and features then depend on arrival order, which varies.
• Sending a model call per record. Latency and cost grow. Batch the rows.

Frequently asked questions

What is Amazon MSK?

Amazon MSK is a managed service that runs Apache Kafka clusters on AWS. It provisions brokers, spreads them across zones, patches and replaces them, and connects to AWS security and monitoring. You use the usual Kafka clients and tools to publish and read event streams.

Is MSK the same as Apache Kafka?

MSK runs Apache Kafka, so applications that speak the Kafka protocol work with it. The difference is who operates the brokers. MSK handles the infrastructure, while you still design topics, partitions, schemas and consumer behaviour. Some low-level settings are managed by the service.

What is the difference between MSK provisioned and MSK Serverless?

Provisioned clusters let you choose broker size, count and storage, giving you control over capacity and tuning. Serverless clusters scale with your traffic within set limits and hide most capacity choices. Provisioned suits steady, high volume loads. Serverless suits variable or small workloads with simpler operations.

How does MSK differ from Amazon Kinesis?

Both stream events, but MSK exposes the Kafka protocol and its ecosystem, while Kinesis offers an AWS-native API and different scaling concepts. If you already use Kafka clients, connectors or stream processors, MSK is a closer fit. If you want a simple AWS-only stream, Kinesis may be easier.

How do I secure an MSK cluster?

Place it in private subnets, restrict network access with security groups, encrypt traffic with TLS, and encrypt storage with a managed key. Use IAM, SCRAM or mutual TLS for authentication, and grant each client only the topics and actions it needs. Avoid unauthenticated access outside tests.

Can I send data from MSK to S3?

Yes. A common pattern is to use a sink connector in MSK Connect or another Kafka Connect deployment that writes topic records to S3 as Parquet or JSON files, partitioned by time. These files can then be queried by Athena or processed by Spark.

How does MSK help with machine learning on tabular data?

It moves and stores rows as they happen. You can build rolling context tables, score events with a model, publish predictions to other topics, and replay history to rebuild training or evaluation tables. The model itself runs in a separate service, with MSK acting as the data backbone.

How long can MSK keep data?

Retention is set per topic by time or size, and is limited by broker storage unless tiered storage is used. Tiered storage lets older data move to cheaper storage so you can keep more history. Even so, Kafka is not a permanent archive, so land important data in object storage as well.

What are consumer groups?

A consumer group is a set of consumers that cooperate to read a topic. Each partition is assigned to one consumer in the group at a time, so the group reads in parallel without reading the same record twice. Different groups read the topic independently, each with its own offsets.

How do I handle schema changes in a stream?

Use a schema registry and a compatibility mode, such as backward compatible, so producers can add optional fields without breaking consumers. Test changes before release, and avoid renaming or retyping existing fields. For downstream models that depend on column positions or types, validate each batch against a contract.

Glossary

• Acknowledgement: a broker reply that confirms a write.
• Availability zone: an isolated location within a cloud region.
• Broker: a Kafka server.
• Commit log: an append-only record of events in order.
• Compaction: keeping the newest record for each key.
• Consumer group: consumers sharing a topic's partitions.
• Consumer lag: distance between latest record and a group's position.
• Dead letter topic: a topic that holds records that could not be processed.
• Idempotent: safe to apply more than once with the same result.
• Kafka Connect: a framework for source and sink connectors.
• Offset: a record's position in a partition.
• Partition: an ordered slice of a topic.
• Producer: a client that writes records.
• Replication factor: number of copies of each partition.
• Retention: how long records are kept.
• Topic: a named stream of records.

Summary

Amazon MSK runs Apache Kafka for you on AWS. You get the Kafka protocol and ecosystem, with brokers placed across zones, patched, monitored and replaced by the service. You choose provisioned clusters for control or serverless clusters for simplicity, secure access with IAM, SCRAM or mutual TLS, and extend with managed connectors and a replicator.

For tabular foundation models, a Kafka stream is a live table. MSK moves and stores the rows that feed rolling context windows, event scoring, imputation and anomaly detection, and it provides ordered, replayable history for building leak-free training and evaluation tables. Success depends on good keys, partition counts, schema discipline, idempotent consumers and careful security.

Related

It belongs to AWS data analytics in my stack, under Data engineering and lakehouse. Others in the same group: