MirrorMaker 2

Overview

MirrorMaker 2, often written MM2, is the tool that ships with Apache Kafka for copying data between Kafka clusters. It mirrors topics, keeps topic settings in step, and translates consumer group offsets so applications can fail over from one cluster to another and carry on close to where they stopped. It is built on Kafka Connect, which means it runs as a set of connectors on a Connect cluster.

Teams use MirrorMaker 2 for disaster recovery, for moving data from many edge or regional clusters into a central one, for migrating from one cluster to another, and for sharing selected topics across organisational or geographic boundaries. It is a copy tool, not a magic switch: replication is asynchronous, and every design that uses it must plan for lag, naming and failover. This guide covers how it works, how to configure it, and how it fits with data pipelines that feed tabular models.

What it is

MirrorMaker 2 is a set of three Kafka Connect source connectors and a driver that wires them together.

Ways to run it

• Dedicated MirrorMaker 2 cluster: A purpose-built driver process (connect-mirror-maker) reads one properties file and starts all connectors. This is the simplest option.
• On an existing Connect cluster: You deploy the three connectors yourself through the Connect REST interface. This gives finer control and easier fit with existing operations.
• Standalone Connect mode: Useful for testing.

Naming of remote topics

By default, a topic named orders in a cluster with the alias primary appears in the target cluster as primary.orders. The prefix is the source cluster alias. This prevents loops in bidirectional setups and makes the origin of data clear. The default replication policy handles this; an alternative identity policy keeps the original name, which is handy for migrations but dangerous in two-way replication.

Where it came from

The first MirrorMaker was a simple consumer and producer pair that copied bytes from one cluster to another. It worked for basic cases but had well-known weaknesses. It did not replicate topic configuration, it created topics with default settings, it could not keep consumer offsets aligned, and it rebalanced badly. Operators wrote scripts around it to fill the gaps.

MirrorMaker 2 was designed in a Kafka improvement proposal to fix these issues by using the Kafka Connect framework. That gave it tasks that scale out, a standard way to run and monitor connectors, dynamic configuration, and support for exactly the pieces that were missing: configuration sync, offset translation and heartbeats. It arrived in Apache Kafka in a release after Connect had matured, and it became the standard open source approach to cross-cluster replication. Some vendors offer their own replication tools, and some cloud services provide managed alternatives, such as MSK Replicator for Amazon MSK.

How it works

Step by step

1. You define clusters with aliases, such as primary and backup, and their bootstrap servers.
2. You define flows, for example primary->backup, and choose which topics and consumer groups to include with patterns.
3. The source connector spawns tasks. Each task takes a share of the partitions from the source topics.
4. A task consumes records from the source cluster and produces them to the remote topic in the target cluster, keeping the same partition number by default.
5. The connector also creates missing remote topics, with the same partition count, and syncs configurations on a schedule.
6. An internal topic in the target cluster, usually called mm2-offset-syncs, records pairs of source and target offsets over time.
7. The checkpoint connector uses those pairs to translate each consumer group's committed offset into a matching offset on the remote topic, and writes the results to a checkpoints topic.
8. The heartbeat connector writes heartbeats that flow through the same path, so you can see delay.

Think of it as a mail forwarding service. A letter arriving at office A gets a copy sent to office B, stamped "from A". The service also keeps a note of how far each reader had got in their pile at office A, and translates it into a page number for the copy at office B. If office A burns down, readers can pick up the copy at B and start close to where they were. They may re-read a few letters, since the note is a little behind.

Offsets and why translation is needed

Offsets are per-cluster positions. Message number 1,000 in the source topic may be message number 1,000 in the target if the topic was mirrored from the start, but not if mirroring began later, if records were compacted, or if transactions leave gaps. So a consumer's committed offset cannot be reused as is. The offset-sync records form a map between source and target positions, and the checkpoint connector uses this map to compute the equivalent offset. Translation is conservative: it picks a position at or before the true one, so consumers may see some duplicates but should not skip records.

Replication is asynchronous

MirrorMaker 2 copies data after it has been committed in the source. If the source cluster fails, some recent records may not have been copied yet. The recovery point objective, the amount of data you can lose, is therefore the replication lag at the moment of failure. Monitor lag and set alerts.

Active-passive and active-active

In active-active setups, consumers that need a global view subscribe to both the local topic and the remote-prefixed topic, for example with a pattern like .*orders. This avoids loops because the replication policy will not mirror a topic back to its origin.

A configuration example

For the dedicated driver, one properties file is enough. Hosts and names are placeholders.

# Cluster aliases
clusters = primary, backup

primary.bootstrap.servers = kafka-a.example.internal:9092
backup.bootstrap.servers  = kafka-b.example.internal:9092

# Replicate primary -> backup
primary->backup.enabled = true
primary->backup.topics = orders.*, payments.*
primary->backup.topics.exclude = .*[\-\.]internal, .*\.replica, __.*
primary->backup.groups = .*
primary->backup.groups.exclude = console-consumer-.*

# Replication settings
replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3

# Sync behaviour
sync.topic.configs.enabled = true
sync.topic.acls.enabled = false
emit.checkpoints.enabled = true
emit.heartbeats.enabled = true
sync.group.offsets.enabled = true
refresh.topics.interval.seconds = 60

# Throughput
tasks.max = 8

Start the driver with:

connect-mirror-maker.sh mm2.properties

Running it on an existing Connect cluster

When you already operate Connect, you create the connectors yourself. A source connector definition looks like this:

{
  "name": "mm2-source-primary-to-backup",
  "config": {
    "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "source.cluster.alias": "primary",
    "target.cluster.alias": "backup",
    "source.cluster.bootstrap.servers": "kafka-a.example.internal:9092",
    "target.cluster.bootstrap.servers": "kafka-b.example.internal:9092",
    "topics": "orders.*",
    "replication.factor": "3",
    "tasks.max": "8",
    "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
    "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter"
  }
}

Note the byte array converter. MirrorMaker 2 should move bytes untouched, so it does not interpret or alter payloads. This is important with schema-aware formats; see the registry section below.

Reading from the backup after failover

Consumers in the target cluster can use a helper to translate offsets from the checkpoints. In Java there is a utility for this, and operators can also use scripts. A rough outline:

# Pseudocode: translate and seek after failover
from mm2_utils import translate_offsets  # placeholder helper

offsets = translate_offsets(
    remote_cluster="backup",
    source_alias="primary",
    group_id="orders-service",
)
consumer.assign(list(offsets.keys()))
for tp, offset in offsets.items():
    consumer.seek(tp, offset)

With sync.group.offsets.enabled turned on in recent versions, the connector can write translated offsets directly into the target cluster's consumer group state, as long as the group is not active there. That simplifies failover.

Handling schemas

If messages use a Schema Registry, each message carries a schema id from the source registry. MirrorMaker 2 copies the bytes, so the id travels as is. The target cluster's consumers must be able to resolve that id. Common solutions include replicating the registry's own storage, running registries that share ids, or using a registry feature designed for cross-cluster use. Verify this before you trust a failover plan. A topic that is copied perfectly but cannot be decoded is not a working backup.

Key concepts and terms

A worked example

A company runs two clusters, primary in one region and backup in another. It wants a disaster recovery copy of three topics: orders, payments and customers. The consumer group orders-service should be able to fail over.

Set up

The team writes the properties file shown earlier, sets replication factor 3 on the internal topics, and starts the driver on two machines for redundancy. After a minute the backup cluster holds:

• primary.orders, primary.payments, primary.customers
• primary.checkpoints.internal
• heartbeats
• mm2-offset-syncs.primary.internal

Normal operation

The group orders-service reads orders in the primary cluster. It has committed offset 5,000 on partition 0. MirrorMaker 2 has copied the first 5,200 records of that partition. The offset-sync topic holds pairs such as (source 4,990, target 4,990) from earlier in the run. The checkpoint connector translates the committed offset to target offset 4,990 or slightly higher, depending on the latest sync.

Disaster

The primary region becomes unavailable. At that moment, the heartbeat shows the last 3 seconds of data had not been copied. This is the actual data loss, the RPO in practice.

Failover steps

1. Confirm the primary is truly down, not just unreachable from one site.
2. Stop MirrorMaker 2 for this flow to avoid confusion.
3. Start the orders-service consumers against the backup cluster, subscribed to primary.orders.
4. Let the group resume from translated offsets. A few records between 4,990 and 5,000 will be processed again, so the service must be idempotent.
5. Point producers to the backup cluster. New records go to local topics such as orders in the backup cluster, and consumers that need the full history subscribe to both orders and primary.orders.

Failback

When the primary returns, the team sets up the reverse flow backup->primary to bring the primary up to date. Records that only exist in the backup are copied back as backup.orders. They then move consumers and producers over in a planned window and re-establish the original flow. This is the most delicate step, and it should be rehearsed.

Measuring lag with a script

A simple check compares the end offset of a source partition with the end offset of its remote copy.

def lag(source_end, target_end):
    # Both numbers are the log end offsets for the same partition.
    return max(0, source_end - target_end)

source_end = 105_400
target_end = 105_210
print("records behind:", lag(source_end, target_end))

This approximates lag if the topic was mirrored from the start and has no gaps. In production, use the metrics that MM2 exposes, including replication latency and records age, and the heartbeat topic for end-to-end measurement.

Sizing a flow with simple arithmetic

Before turning on a flow, estimate the load. Suppose the selected topics receive 40 MB per second at peak and the link between clusters can carry 100 MB per second after other traffic. The flow needs 40 MB per second plus headroom for catch-up. If the flow falls behind for ten minutes, it must copy 40 x 600 = 24,000 MB of backlog on top of live traffic. With 60 MB per second spare, catching up takes 24,000 / 60 = 400 seconds, which is under seven minutes. If the spare capacity were only 10 MB per second, catch-up would take 2,400 seconds, or forty minutes. Doing this sum in advance tells you whether the link is large enough and what your real recovery point is during incidents.

Troubleshooting guide

Choosing a topology

Use this short guide when you are unsure which pattern to pick.

1. If the only goal is a standby copy, use active-passive. It is the easiest to reason about.
2. If several regions each own their data and a central team needs all of it, use hub and spoke with one-way flows into the hub.
3. If users in two regions must write at low latency, consider active-active, but plan how conflicting writes are settled by the application.
4. If you are moving to a new cluster, mirror first, switch consumers, then switch producers, then stop the flow.

Exactly what is copied

MirrorMaker 2 copies record keys, values, headers, timestamps and the partition assignment. It can copy topic configuration, such as retention and compaction settings, and the partition count. It does not copy broker settings, quotas or the transaction state of a producer. Because of this, a restored application should be checked for any behaviour that relied on those missing pieces.

Strengths and limitations

Strengths

• Included with Apache Kafka and based on Kafka Connect, so it scales out and is monitored like other connectors.
• Replicates topic configuration, partition counts and, optionally, ACLs.
• Translates consumer offsets, which makes failover practical.
• Supports many topologies: one-way, two-way, hub and spoke, fan-out.
• Works across regions, data centres and cloud providers.
• No change to producers or consumers when used for plain copying.

Limitations

• Asynchronous, so data loss on sudden failure is possible. No guarantee of zero RPO.
• Offset translation is approximate. Expect duplicates after failover, and design for idempotency.
• Delivery semantics have historically been at least once. Newer versions have improved on this, but check what your version supports.
• Remote topic prefixes change topic names. Consumers need to subscribe to patterns or to the prefixed names.
• Schema ids need a separate plan.
• Tuning and operations need care: task counts, producer batching, network limits and cross-region latency all matter.
• Compacted topics and transactional topics need attention. Transaction markers are not mirrored as transactions.
• Two-way setups do not resolve conflicts. If two regions update the same key, you need an application-level rule.
• Large partition counts can mean many tasks and slow rebalances.

How it compares with alternatives

A common architecture combines approaches. Use MirrorMaker 2 or a managed replicator for the live copy, and archive topics to Amazon S3 for long-term history.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns, handle mixed numeric and categorical types and missing values, may work without informative column names, and are pre-trained on large collections of synthetic or real tables. They learn from provided rows through in-context learning and may be updated over time. MirrorMaker 2 does not train or serve such models. Its role is to move and protect the streams of rows that become their tables.

Moving rows to where the model lives

Data is often produced in one place and used in another. A model team may run training or scoring in a central or different cloud region. MirrorMaker 2 can bring selected topics there without changing the producers. In a hub-and-spoke design, many regional clusters send event topics to a central cluster, where a pipeline lands them as tables in Apache Iceberg or Delta Lake. The model then reads a combined table with rows from all regions.

Row order and structure

A table of events from many partitions has no single global order, and mirroring keeps partition order but not a global one. Because MM2 preserves the partition number and order within a partition, records for the same key stay in order. Use event time columns, not arrival order, when building training rows. Remember that the remote topic prefix is metadata: include the origin cluster as a column if the region matters to the model.

Mixed types and schema fidelity

MM2 copies bytes unchanged, so types are preserved exactly as written. This is good for fidelity: numeric and categorical columns are not altered in transit. The catch is that decoding requires the schema, so the registry plan from earlier is part of the data path. If a model relies on stable types, test decoding in the target cluster regularly.

Missing values and duplicates

Offset translation can cause duplicates after failover. Duplicate rows in a training table change class balance and can leak information between train and test splits if copies land on both sides. Deduplicate by a stable event id when building tables. Missing data is a separate issue: replication lag at the time of a failure can mean the latest records are absent in the backup. A model trained on a backup copy may have a short blind spot at the end of the period. Record the replication watermark so you know where complete data ends.

No column names

If a pipeline strips names to test name-free modelling, do it after replication, in the table-building step, not in the stream. Keeping the stream faithful makes replay possible.

Pre-training data and continual learning

For pre-training and refreshes, you want a durable, reproducible history. A mirrored cluster with long retention is a second copy that can be replayed into a new training table, and it can spare the production cluster from heavy read load. For continual learning, new data should arrive at a steady cadence. Heartbeats and lag metrics tell you whether a refresh is based on complete data. See /model/research for how continual and in-context approaches are studied, and /model/architecture for how tabular inputs are represented.

Imputation and anomaly detection

Anomaly detection on a stream benefits from a clean, regional split. You can run anomaly scoring close to the source and still mirror the events to a central place for model training. For imputation, be careful with gaps caused by outages. A gap in replicated data is not the same as a missing cell in a row. Mark outage windows, so a model does not treat a missing hour as a quiet hour or learn from it as a pattern.

Sector and semantic classification

When rows from several organisations or regions are combined for classification, the origin prefix helps keep sources separate for audits and for fairness checks. Keep access controls aligned with the source: mirrored topics can be exposed to people who should not see them if ACLs are not synced or reviewed.

Governance in the data path

Replication copies data, including personal data, to new places. Check regional rules about where data may live. Use single message transforms at the source connector side, or process before replication, if fields must be removed or masked before leaving a region. Track movement with data lineage, and verify that the shape of replicated data still matches its contract with data contract validation and schema drift detection. For benchmarking context, see /model/benchmarks.

Practical guidance and best practices

Planning

• Define RPO and RTO first. They decide whether you need MM2 at all, or something with tighter guarantees.
• Decide on active-passive unless you truly need writes in both places. Active-active is harder.
• Choose the replication policy early, since renaming later is painful.
• Document failover and failback as runbooks, and rehearse them.

Sizing and tuning

• Run MM2 close to the target cluster in most cases, since consumers in the source region reading across the network is usually less harmful than producing across it. Test both in your setup.
• Increase tasks.max until the network or the brokers become the limit.
• Tune producer and consumer settings for batching and compression, such as batch.size, linger.ms and compression.type, using the producer.override and consumer.override prefixes where needed.
• Mirror only what you need. Use topic patterns and exclusions.
• Set replication factor on internal topics to match your durability needs.

Monitoring

• Track replication latency, records lag and task health through Connect metrics.
• Alert on heartbeat gaps, which show a broken flow.
• Compare end offsets for critical topics as an independent check.
• Watch the size of the offset-sync and checkpoint topics.

Security

• Use separate credentials for MM2 with least privilege: read on source topics and groups, write and create on target topics.
• Use encrypted connections between clusters.
• Decide whether ACL sync is wanted. Automatic ACL copying can be convenient or can open access unintentionally.

Consumers

• Make consumers idempotent. Expect some duplicates after failover.
• Use group offset sync, or translate offsets in a controlled step.
• Subscribe with patterns in active-active designs, and guard against double counting.

Testing

• Run a game day: stop the primary, fail over, check results, fail back.
• Validate that decoded messages in the backup match the source for a sample.
• Test with a topic that has compaction, headers and null values.

A failover runbook outline

A written runbook is worth more than a perfect design that nobody remembers. A good outline has these parts.

Before an incident

• A list of the topics and groups in scope, with owners.
• The agreed RPO and RTO for each.
• The current replication lag dashboard and where to find it.
• Credentials and bootstrap addresses for the backup cluster, stored securely.
• A decision rule for who may declare a failover.

During failover

1. Declare the incident and note the time.
2. Record the last heartbeat time and the lag, so you know the data gap.
3. Stop the flow into the backup, to keep its state still while you work.
4. Start consumers against the backup using translated offsets.
5. Redirect producers using configuration or DNS.
6. Check consumer lag and error rates in the backup.
7. Tell downstream teams about the data gap and the duplicates.

After the incident

1. Plan the reverse flow to bring the original cluster up to date.
2. Reconcile any data written to both sides.
3. Schedule the failback in a quiet window.
4. Write down what went well and what did not, and update the runbook.

Metrics worth watching

Common mistakes

• Assuming zero data loss. Replication is asynchronous.
• Forgetting the registry. The bytes arrive, but consumers cannot decode them.
• Using the identity naming policy in two-way replication, which can cause loops.
• Mirroring everything, including internal and test topics, and exhausting capacity.
• Not testing failback, which is often harder than failover.
• Running one MM2 instance with no redundancy for a critical flow.
• Not making consumers idempotent.
• Forgetting that topics created in the target with different partition counts break key ordering.
• Not syncing or reviewing ACLs, leaving mirrored data over-exposed or unreachable.
• Treating replicated tables as complete without checking the replication watermark.
• Ignoring clock and time zone differences when reading heartbeats across regions.
• Using MM2 to transform data. Do transformations in a separate step.

Frequently asked questions

What is MirrorMaker 2 used for?

It copies topics, topic configuration and consumer group offsets between Kafka clusters. Typical uses are disaster recovery, aggregating data from regional clusters into a central one, migrating between clusters and sharing selected topics with other teams or regions. It runs on Kafka Connect.

How is MirrorMaker 2 different from the original MirrorMaker?

The original was a plain consumer and producer pair with limited features. MirrorMaker 2 uses Kafka Connect and adds topic configuration sync, offset translation, heartbeats, better scaling and flexible topologies. It also names remote topics with a source prefix to prevent loops.

Does MirrorMaker 2 guarantee no data loss?

No. Replication is asynchronous, so records written to the source just before a failure may not have reached the target. The amount you may lose equals the replication lag at that moment. Monitor lag and set expectations through a recovery point objective.

How do consumer offsets work after failover?

The checkpoint connector uses offset-sync records to translate each group's committed offset into an equivalent offset in the target cluster. Translation is conservative, so consumers may see a few repeated records but should not skip any. Consumers should be idempotent to handle repeats.

Why are mirrored topics prefixed with the source cluster name?

The prefix shows where data came from and stops records from being copied back and forth in a loop in two-way setups. You can change the replication policy, but the default is the safest for active-active designs.

Can MirrorMaker 2 work with a schema registry?

It copies message bytes, so schema ids inside messages are unchanged. Consumers in the target cluster need to resolve those ids, which means the schemas must be available there with matching ids, or you must use a registry feature built for this. Check this before relying on failover.

Is MirrorMaker 2 better than a managed replicator?

It depends. MirrorMaker 2 gives control and works anywhere, but you operate it. A managed service such as MSK Replicator removes the operations work for supported platforms, at the cost of flexibility. Choose based on where your clusters run and how much operational effort you want.

Can I run active-active replication?

Yes, by mirroring each cluster to the other with prefixed topic names. Consumers that need a global view subscribe to local and remote topics. The tool does not resolve write conflicts, so applications must handle updates to the same key in two places.

How does MirrorMaker 2 help machine learning teams?

It moves event streams to where training and scoring run, keeps a replayable second copy of history, and supports regional collection into a central lake. Teams must deduplicate after failover, record replication watermarks, and make sure schemas can be decoded, but with those controls it gives dependable input for table-based models.

Should I use it for transformations?

No. It is built to copy bytes faithfully. Do filtering and reshaping before or after replication, using single message transforms in a separate connector or a stream processor. Keeping replication simple makes it easier to run and to trust.

Glossary

• Active-active: Both clusters take writes and mirror each other.
• Active-passive: One cluster takes writes, the other is a standby.
• Checkpoint: A translated consumer offset for the target cluster.
• Cluster alias: The short name of a cluster in MM2 config.
• Failback: Returning to the original cluster after recovery.
• Failover: Switching to the backup cluster.
• Flow: A directed replication path between two clusters.
• Heartbeat: A periodic record for monitoring a replication path.
• Offset sync: A mapping between a source and a target offset.
• Remote topic: A mirrored topic in the target cluster.
• Replication lag: The delay between source writes and target copies.
• Replication policy: The rule for naming remote topics.
• RPO: The most data you can lose, measured in time.
• RTO: The longest time you can be down.

Summary

MirrorMaker 2 is Kafka's built-in way to replicate topics, configuration and consumer offsets between clusters, built on Kafka Connect. It supports disaster recovery, aggregation, migration and sharing, with prefixed remote topics and offset translation that make failover practical. It is asynchronous, so plan for lag, duplicates and schema resolution in the target.

For teams that build tabular models, it is plumbing that brings rows to where they are used, keeps a replayable second copy, and supports regional collection. Pair it with idempotent consumers, deduplicated table building, recorded replication watermarks, and governance checks. For related reading, see Apache Kafka, Kafka Connect, MSK Replicator, Schema Registry and event-driven architecture.

Related

It belongs to Event streaming in my stack, under Data engineering and lakehouse. Others in the same group: