MSK Replicator

Overview

MSK Replicator is a feature of Amazon MSK, the managed Kafka service on AWS. It copies topics and consumer group offsets from one MSK cluster to another, in the same region or in a different one. AWS runs the replication for you, so there are no connector clusters to build, patch or scale. You create a replicator, name the source and target clusters, choose which topics to copy, and the service does the rest.

The main reasons to use it are cross-region resilience and data sharing. If a region has a problem, applications can switch to the copy in another region and carry on from close to where they stopped. If you want a central cluster to hold data from several places, a replicator can feed it. It does the job that MirrorMaker 2 does for self-managed Kafka, but as a managed capability. Like all asynchronous replication, it has lag, so it reduces risk rather than removing it.

What it is

MSK Replicator is a managed replication resource in the MSK family. You define it with a few parts:

• A source cluster and a target cluster. Both are MSK clusters. Clusters must be set up with supported authentication, typically IAM-based access control.
• A replication configuration with patterns for topics to include and exclude.
• A setting for how consumer group offsets are handled.
• A setting for how topic names are treated in the target: a prefixed name or the same name.
• A service execution role that lets the service read from the source and write to the target.

The service copies records, topic configuration such as partition count, and, if enabled, consumer group offsets. Because it is a managed service, scaling, patching and failure recovery of the replication workers belong to the provider.

Topic names

Two approaches exist. With a prefixed name, topics in the target are named with the source cluster's name as a prefix, similar to remote topics in MirrorMaker 2. This avoids loops in two-way setups. With identical names, topics keep the same name in the target, which makes failover easier for applications because they do not have to change what they subscribe to. Identical naming needs care in active-active designs, because without protection two replicators could copy records back and forth. The service offers settings meant to handle this, and you should read the current documentation for the exact rules in your setup.

What it does not do

• It does not transform records. It copies them.
• It does not make replication synchronous.
• It does not replicate a schema registry. If you use one, plan schema availability in the target.
• It does not remove the need for tested failover procedures.

Where it came from

Teams that ran Kafka across regions first used the original MirrorMaker, then MirrorMaker 2 on their own compute. On AWS this usually meant building a Connect cluster on virtual machines or containers, tuning it, monitoring it and handling upgrades. That work is not the core business of most teams, and failures in the replication layer showed up at the worst moment, during an outage.

Cloud providers answered with managed options. MSK Replicator was introduced as a service feature of MSK to offer cross-cluster and cross-region replication without operating the replication infrastructure. It followed the same logic as earlier managed features: take a common pattern that many customers build themselves, and run it as a service with the usual cloud promises of scaling, monitoring and integration with identity and logging. Details of supported cluster types, regions and limits change over time, so check the current service documentation before planning.

How it works

Step by step

1. You create or choose two MSK clusters, a source and a target. They may be in different regions, or the same region.
2. You allow access, usually with IAM policies and a service role, so the replicator can read from the source and write to the target.
3. You create a replicator and supply the cluster references, topic patterns, and offset settings.
4. The service provisions the replication capacity it needs and starts reading from the source topics.
5. As records arrive in the source, the service copies them to the target, keeping their order within each partition.
6. The service also creates missing topics in the target, with matching partition counts, and keeps certain topic settings in step.
7. If consumer group offset sync is enabled, the service translates the offsets for selected groups and writes them in the target, so consumers there can resume.
8. The service publishes metrics to the cloud monitoring system, such as replication latency and throughput.

An everyday analogy: imagine a bank that keeps its ledgers in two buildings in different cities. A courier service (the replicator) carries a copy of each new page to the second building soon after it is written. The service also copies the bookmarks that show where each clerk had stopped reading. If the first building closes, clerks open the second building's ledgers and continue near their bookmarks. The courier is fast but not instant, so the last few pages may still be on the road.

Configuration example

Below is an example using a command line style. Names, accounts and addresses are placeholders, and exact option names may differ across tool versions, so treat it as a sketch.

aws kafka create-replicator \
  --replicator-name orders-dr \
  --service-execution-role-arn arn:aws:iam::111122223333:role/example-replicator-role \
  --kafka-clusters '[
    {"AmazonMskCluster": {"MskClusterArn": "arn:aws:kafka:region-a:111122223333:cluster/source-example/uuid-1"},
     "VpcConfig": {"SubnetIds": ["subnet-example1","subnet-example2"], "SecurityGroupIds": ["sg-example"]}},
    {"AmazonMskCluster": {"MskClusterArn": "arn:aws:kafka:region-b:111122223333:cluster/target-example/uuid-2"},
     "VpcConfig": {"SubnetIds": ["subnet-example3","subnet-example4"], "SecurityGroupIds": ["sg-example2"]}}
  ]' \
  --replication-info-list '[
    {"SourceKafkaClusterArn": "arn:aws:kafka:region-a:111122223333:cluster/source-example/uuid-1",
     "TargetKafkaClusterArn": "arn:aws:kafka:region-b:111122223333:cluster/target-example/uuid-2",
     "TargetCompressionType": "NONE",
     "TopicReplication": {"TopicsToReplicate": ["orders.*","payments.*"],
                          "TopicsToExclude": ["__.*"],
                          "CopyTopicConfigurations": true,
                          "CopyAccessControlListsForTopics": false,
                          "DetectAndCopyNewTopics": true},
     "ConsumerGroupReplication": {"ConsumerGroupsToReplicate": [".*"],
                                  "SynchroniseConsumerGroupOffsets": true,
                                  "DetectAndCopyNewConsumerGroups": true}}
  ]'

Infrastructure as code

Teams commonly manage replicators with templates. A sketch in a YAML style:

Resources:
  OrdersReplicator:
    Type: AWS::MSK::Replicator
    Properties:
      ReplicatorName: orders-dr
      ServiceExecutionRoleArn: arn:aws:iam::111122223333:role/example-replicator-role
      KafkaClusters:
        - AmazonMskCluster: { MskClusterArn: arn:aws:kafka:region-a:111122223333:cluster/source-example/uuid-1 }
          VpcConfig: { SubnetIds: [subnet-example1, subnet-example2], SecurityGroupIds: [sg-example] }
        - AmazonMskCluster: { MskClusterArn: arn:aws:kafka:region-b:111122223333:cluster/target-example/uuid-2 }
          VpcConfig: { SubnetIds: [subnet-example3, subnet-example4], SecurityGroupIds: [sg-example2] }
      ReplicationInfoList:
        - SourceKafkaClusterArn: arn:aws:kafka:region-a:111122223333:cluster/source-example/uuid-1
          TargetKafkaClusterArn: arn:aws:kafka:region-b:111122223333:cluster/target-example/uuid-2
          TargetCompressionType: NONE
          TopicReplication:
            TopicsToReplicate: ["orders.*"]
          ConsumerGroupReplication:
            ConsumerGroupsToReplicate: [".*"]

Check the current resource definition before use, since property names are part of a service interface that can change.

Permissions

The service execution role needs permission to read from the source cluster and to write, create topics and manage groups in the target. In IAM-based access control this is expressed as actions on cluster, topic and group resources. Keep these policies narrow, and limit them to the topics that you replicate.

Offsets

With offset sync on, the service writes translated consumer group offsets in the target. As in other replication tools, the translation is approximate and conservative. After failover, consumers may reprocess some records. The consumer group must not be actively consuming in the target while syncing, or the sync may skip it, so the usual order is: stop syncing or confirm the group is idle there, start consumers in the target, and proceed.

Failover and failback

The service helps you copy the data. It does not decide when to fail over. Your runbook must cover:

• How you detect a regional problem.
• Who may call a failover.
• How applications find the new cluster, for instance through configuration or DNS.
• How to bring data back to the original cluster. That usually means creating a replicator in the reverse direction, then moving applications back at a calm time.

Monitoring

The service reports metrics such as replication latency, messages per second and bytes per second, per replicator and per topic. Set alarms on latency and on any drop to zero throughput while the source is active. Add an independent check for critical topics, such as comparing end offsets or running a probe message through the path.

Key concepts and terms

A worked example

A retail platform runs an order stream on an MSK cluster in one region. Business rules say that if the region fails, order processing must resume in another region within fifteen minutes, with at most a few seconds of lost events.

Design

• A source cluster orders-primary in region A.
• A target cluster orders-standby in region B, sized for production load.
• A replicator copies orders.* and payments.*, with identical topic names in the target for easy failover, and consumer group offset sync on.
• An alarm fires if replication latency passes 30 seconds, or if throughput falls to zero for five minutes while the source is active.
• Producers and consumers read the bootstrap address from configuration, so changing regions needs a config change only.

Day to day

Orders flow into region A. The replicator copies them. A dashboard shows typical latency of a couple of seconds. The standby cluster has no active consumers, apart from a small test group that reads a sample to prove the data is decodable.

Incident

Region A becomes unavailable.

1. The on-call engineer checks the last replication latency and throughput. The last data point shows about four seconds of delay, so the expected data gap is about four seconds of orders.
2. The team confirms the failure and calls the failover.
3. They update configuration so consumers and producers use region B's bootstrap address.
4. Consumers start in region B. The synced offsets place them a little before where they stopped, so several recent orders are processed twice. The order service is idempotent, using the order id, so the effect is harmless.
5. Producers write to region B. Orders are now accepted there.
6. Finance gets a note about the four second gap, and a reconciliation job compares downstream totals with the payment provider.

Recovery

When region A returns, the team creates a replicator from B to A to catch up the old cluster, with care to avoid mixing the two directions on the same topic names in a way that could loop. After catch-up and checks, they plan a quiet-hour switch back, or decide to keep B as primary and make A the standby.

A small estimate of data at risk

If the source produces 5,000 records per second and replication is 4 seconds behind at failure, then roughly 5,000 x 4 = 20,000 records may be missing in the target. Whether that is acceptable depends on the business. For an order stream with a source of truth elsewhere, it may be fine. For payment authorisations, it may need extra protection, such as writing to two places at the application level.

def records_at_risk(rate_per_sec, lag_sec):
    return int(rate_per_sec * lag_sec)

print(records_at_risk(5000, 4))   # 20000
print(records_at_risk(5000, 30))  # 150000

Checking a copy after the fact

A simple verification compares counts per partition, using end offsets, for a range of event time. Because offsets can differ between clusters, compare by event time windows or by business keys, not by raw offset.

-- Compare hourly counts landed from each cluster's topic (placeholder tables)
SELECT 'primary' AS src, date_trunc('hour', event_time) AS hr, count(*) AS n
FROM lake.orders_primary
GROUP BY 1, 2
UNION ALL
SELECT 'standby' AS src, date_trunc('hour', event_time) AS hr, count(*) AS n
FROM lake.orders_standby
GROUP BY 1, 2
ORDER BY hr, src;

Networking and access

Both clusters sit inside private networks. For the service to reach them, the replicator is given subnets and security groups for each cluster. Cross-region traffic travels over the provider's network. Key points:

• The security groups must allow the service to connect to the broker ports of each cluster.
• Authentication on the clusters should match what the service supports. IAM-based access control is the usual choice.
• Applications that fail over need a network path to the target cluster. Check this early, because a standby that applications cannot reach is of no use.
• Keep the two environments consistent in encryption settings and client versions.

Thinking about cost

Cost has several parts, and it helps to list them before you start.

To reduce cost without losing safety, replicate only business-critical topics, use compression, trim retention in the standby if rules allow, and review unused topics regularly. Do not trim the standby so far that it cannot hold the data you need after an incident.

A drill plan you can repeat

Run a failover drill at least a few times a year. A good drill has a fixed shape.

1. Pick a low-risk window and tell all affected teams.
2. Record the replication latency and the throughput before starting.
3. Switch a test consumer group and a test producer to the target cluster.
4. Check that translated offsets place the consumer near the right spot.
5. Compare a sample of records read in the target with the source by business key.
6. Switch back, and confirm that no records were lost or doubled beyond what the design allows.
7. Write down the elapsed time for each step and compare it with your RTO.
8. Fix anything that was slow, unclear or manual.

Drills turn a theoretical plan into a practised routine, and they often reveal a missing permission or an outdated address.

Use as a data sharing tool

Replication is not only about disaster recovery. A team can publish a set of topics from one cluster and replicate them to another team's cluster, which then owns its own consumers and retention. This keeps teams loosely coupled, as in event-driven architecture. The producers do not need to know who reads, and the consumers do not load the producers' brokers. Agree on the contract for each shared topic, including schema, fields that are safe to share, and expected freshness.

Mapping the data path to a model pipeline

The following table shows a typical flow from events to a model-ready table, with the place of the replicator.

Strengths and limitations

Strengths

• No replication servers to operate. Scaling, patching and recovery of the workers are handled for you.
• Copies data, topic configuration and consumer group offsets.
• Works across regions and within a region.
• Integrates with cloud identity, networking and monitoring.
• Quick to set up compared with building a connector cluster.
• Preserves partition and record order within a partition.
• Fits infrastructure-as-code workflows.

Limitations

• Tied to Amazon MSK clusters. It is not a general tool for any Kafka cluster.
• Asynchronous, so the target may lag and a sudden failure can lose recent records.
• No data transformation.
• Offsets are translated, not identical. Expect duplicates after failover.
• Cross-region data transfer has its own cost, and latency across regions varies.
• Less flexible than a self-run tool for unusual topologies or special filters.
• Features, limits and supported configurations evolve, so verify them before design.
• Schema registry content is not copied for you.
• Two-way replication needs careful naming to avoid loops, and conflicts between regions remain an application problem.

How it compares with alternatives

A balanced plan often uses several layers: multi-zone clusters for zone failures, a replicator for regional failures, and object storage for history and rebuilds.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns, handle mixed numeric and categorical types and missing values, can work with uninformative column names, and are pre-trained on large numbers of synthetic or real tables. They learn from supplied rows through in-context learning and can be refreshed as data changes. MSK Replicator does none of the modelling. It decides where copies of event streams exist, how fresh they are, and how safely they can be used.

Getting rows to the right place

Many companies hold operational data in one region and run analytics or machine learning in another. A replicator can bring selected topics to the region where training or scoring runs, with no change to producers. The rows then land in tables, for example through Kafka Connect sinks into Apache Iceberg or Delta Lake on Amazon S3. A tabular model can read those tables for training, for context examples, or for batch scoring.

Freshness and in-context learning

In-context learning uses recent labelled rows as examples. If the context is built from a replicated copy, its freshness is the replication lag plus pipeline delay. Latency is normally small, but during incidents it can rise. Make the context builder aware of the replication watermark: only use rows older than the point up to which data is known to be complete, and record that watermark with each context table.

Duplicates and leakage

After a failover or a replicator restart, consumers may see some records twice. Duplicate rows inflate counts and can leak between train and test sets if copies fall on both sides. Deduplicate on a stable event id when building training tables. Keep the id as a column during the build, then drop it before the model sees the table if it should not be a feature.

Mixed types, missing values and no column names

Replication moves bytes unchanged, so column types and values arrive exactly as written. That protects type fidelity, which matters for mixed numeric and categorical columns. Missing values stay missing. A gap caused by an outage is different from a missing cell, and a model should not read an outage as a quiet period. Mark outage windows in metadata so imputation and anomaly detection do not learn from false gaps. If you intend to test name-free modelling, remove names in the table-building step, and keep the stream itself faithful.

Schemas travel separately

If messages use a Schema Registry, the schema ids inside messages refer to the source registry. In the target region, the model pipeline needs a way to decode them. Plan this at design time. A perfect copy of undecodable bytes is not useful for building tables. Use schema drift detection in the target to confirm the shape of replicated data stays the same as the source.

Pre-training data and continual learning

For pre-training corpora and periodic refreshes, a second cluster with longer retention can serve as a replay source, which protects production brokers from heavy read load. For continual learning, track which data is included by recording topic, time range and replication watermark. This supports data lineage and reproducible retraining. For background on these learning styles, see /model/research, and for how tables are encoded, see /model/architecture.

Anomaly detection across regions

A common pattern is to score anomalies near the source for speed, and replicate events to a central region for model training and review. Remember that the central view lags slightly, so central dashboards should show freshness. For models that compare behaviours across regions, keep the origin as a column, because replication can hide where a record started if names are identical.

Sector and semantic classification

When classifying entities or columns using data from several regional clusters, the origin and time range are useful context. Access control matters: replicated topics may contain personal or regulated data that must not leave a region. Mask or drop sensitive fields before replication using single message transforms on the source side or in a stream job, and verify rules with data contract validation. For measured results on public suites of tables, see /model/benchmarks.

Practical guidance and best practices

Planning

• Write down RPO and RTO before choosing tools. If the RPO is zero, asynchronous replication alone will not meet it.
• Size the target cluster for production load, not for a copy only. A standby that cannot take traffic is not a standby.
• Decide naming early: prefixed for safe two-way flows, identical for easy failover in one-way flows.
• Check regional rules for where data is allowed to live.

Setup

• Use least-privilege roles for the service, scoped to the replicated topics and groups.
• Replicate only what you need. Exclude internal and test topics.
• Enable consumer group offset sync for groups that need to fail over.
• Create the replicator through templates, and keep them in version control.
• Match important topic settings, such as retention, in the target, and review copied settings.

Operations

• Alarm on latency, throughput and replicator state.
• Run a canary: a small producer and consumer that measure end-to-end delay across clusters.
• Record the watermark of complete data in a table, for use by downstream jobs.
• Review costs for data transfer between regions, and for the target cluster.
• Plan upgrades of the clusters so that source and target stay compatible.

Application design

• Make consumers idempotent and use stable event ids.
• Read cluster addresses from configuration, so a switch needs no code change.
• Do not depend on exact offsets across clusters.
• Test with real failover drills, including the return path.

Data hygiene

• Keep schemas in a place that both regions can read.
• Mask sensitive fields before data leaves its region, where rules require it.
• Track which tables were built from which cluster and time range.

A pre-launch checklist

Before you rely on a replicator for production, walk through this list.

• The source and target clusters exist, with matching authentication.
• The service role has only the permissions it needs.
• Topic patterns include every topic you need and exclude internal ones.
• Offset sync is on for the groups that must fail over.
• Latency and throughput alarms are set and routed to a person.
• A canary measures end-to-end delay.
• Schemas can be read in the target region.
• Consumers are idempotent.
• A written runbook covers failover and failback.
• A drill has been done, and times recorded.
• Regional data rules were checked for the replicated topics.
• Downstream table builds use a watermark and deduplicate.

Reading metrics: a quick guide

Key ordering and partitions

Records with the same key land on the same partition in the source, and replication keeps them on the same partition number in the target. That preserves per-key order. This only holds if the target topic has the same partition count. If someone creates a topic by hand in the target with a different count, new records may be spread differently and per-key order can break. Let the service create target topics, and avoid changing partition counts on either side without a plan.

What to tell downstream users

Teams that read from the standby or from replicated tables should know:

• How fresh the data is, and how that is measured.
• That duplicates may appear after a failover.
• That a short gap may exist after a sudden failure.
• Where to find the watermark of complete data.
• Who to contact when the numbers look wrong.

Clear messages reduce confusion in a stressful moment and help analysts and model builders avoid drawing the wrong conclusion from incomplete data.

Common mistakes

• Assuming a replicator gives zero data loss.
• Creating the standby cluster too small to carry production traffic.
• Using identical topic names in a two-way setup without protection against loops.
• Forgetting to make consumers idempotent.
• Not planning how schemas become available in the target.
• Granting broad permissions to the service role.
• Testing only the copy and not the application failover.
• Never testing the return path after failover.
• Treating the target copy as complete during or just after an outage.
• Replicating everything, including noisy test topics, and paying for needless transfer.
• Training models on both the primary and standby tables, and so counting records twice.
• Relying on the service to transform or filter fields inside records.

Frequently asked questions

What is MSK Replicator?

It is a managed feature of Amazon MSK that copies topics, topic configuration and consumer group offsets from one MSK cluster to another. The clusters can be in the same region or in different regions. AWS operates the replication workers, so you do not run a connector cluster yourself.

Is MSK Replicator the same as MirrorMaker 2?

They do similar work, but they are different products. MirrorMaker 2 is open source and runs on Kafka Connect that you operate. MSK Replicator is a managed service for MSK clusters. The managed option reduces operations work, while the open source tool gives more flexibility and works with any Kafka cluster.

Does it guarantee that no data is lost?

No. Replication is asynchronous, so records written just before a failure may not have reached the target. The potential loss is roughly the replication lag multiplied by the write rate. Choose an architecture that fits your recovery point objective, and monitor lag.

Can I use it within a single region?

Yes. It supports same-region replication as well as cross-region replication. Same-region use can be helpful for copying data between environments or accounts, or for isolating workloads, though it does not protect against a region-wide failure.

What happens to consumer offsets?

If offset sync is enabled, the service writes translated offsets for selected consumer groups into the target cluster. Consumers there can start near where they stopped. Translation is conservative, so some records may be processed again, and applications should handle duplicates.

Does it copy my schemas?

No. It copies the messages and, if enabled, topic settings and offsets. Schema registry content has to be made available in the target by another mechanism. Without it, consumers in the target region may not be able to decode messages that carry schema ids.

Can it transform or filter fields?

No. It copies records as they are. Filtering of topics is by name pattern only. If you need to mask or drop fields before data leaves a region, do that earlier, using a transform in a connector or a stream processing job.

How should I handle topic names?

Use prefixed names if you need two-way replication and want to avoid loops. Use identical names when you want simple failover for one-way flows, because applications keep the same subscriptions. Read the service documentation for the current rules on avoiding loops with identical names.

How does this help teams that train models on tables?

It places copies of event streams where models are trained or scored, keeps a second source for replay, and supports resilient pipelines. To get value, teams should deduplicate after failover, record the replication watermark, make schemas decodable in the target region, and mask sensitive fields before data crosses a boundary.

When should I choose something else?

Choose another tool if your clusters are not on MSK, if you need record-level changes during copy, or if you need an unusual topology. In those cases MirrorMaker 2 or a stream processing job may fit better. If you need zero data loss, look at synchronous designs or application-level protection.

Glossary

• Asynchronous replication: Copying data after it is committed at the source, without waiting.
• Consumer group offset sync: Writing translated consumer positions to the target.
• Cross-region replication: Replication between clusters in different regions.
• Failback: Returning to the original cluster after recovery.
• Failover: Switching applications to the standby cluster.
• IAM: Identity and access management, used to control access.
• Idempotent consumer: A consumer whose repeated processing of a record has no extra effect.
• Replication latency: The delay between a source write and its copy.
• Replicator: The managed resource that performs replication.
• RPO: Recovery point objective, the data loss you can tolerate.
• RTO: Recovery time objective, the downtime you can tolerate.
• Service execution role: The identity used by the service to access clusters.
• Standby cluster: A cluster that holds a copy and takes over in an incident.
• Watermark: The time up to which data is known to be complete.

Summary

MSK Replicator is the managed way to copy topics, configuration and consumer group offsets between Amazon MSK clusters, within or across regions. It removes the work of running a replication cluster, fits cloud identity and monitoring, and supports disaster recovery and data sharing. It is asynchronous, limited to MSK, and does not transform data or copy schemas.

For tabular foundation model work, it positions rows where they are needed, offers a replay source, and supports resilient pipelines. To use it well, deduplicate after failover, track replication watermarks, plan schema availability, mask sensitive fields before replication, and rehearse the whole failover and failback path. Related reading: Amazon MSK, Apache Kafka, MirrorMaker 2, Schema Registry and event-driven architecture.

Related

It belongs to Event streaming in my stack, under Data engineering and lakehouse. Others in the same group: