Kafka Connect

Overview

Kafka Connect is a framework that moves data between Apache Kafka and other systems without custom code. A source connector reads from an external system, such as a database, and writes records to Kafka topics. A sink connector reads from topics and writes to an external system, such as object storage, a search index or a warehouse. You configure connectors with JSON and run them on a scalable, fault-tolerant service.

The value of Kafka Connect is reuse. Instead of writing and maintaining a new ingestion service for every database, file store or API, you install a connector plugin and describe what to move. The framework handles parallelism, retries, offset tracking, restarts and monitoring. This makes it one of the quickest ways to land operational data in a lake and to keep downstream tables in step with source systems.

What it is

Kafka Connect is part of the Apache Kafka project. It runs as a separate process called a worker that hosts connectors. It talks to the Kafka cluster as an ordinary client. It stores its own state, such as configurations and offsets, in internal Kafka topics, so there is no extra database to manage.

The main parts

• Connector: a high-level definition of a job, such as "copy this table to a topic". It decides how to split work into tasks.
• Task: the unit that actually moves data. A connector can run many tasks in parallel.
• Worker: a process that runs connectors and tasks. Several workers form a cluster.
• Converter: code that turns data between Connect's internal format and bytes in Kafka, such as JSON, Avro or Protobuf.
• Transform: a small step that changes each record in flight. These are called single message transforms.
• Dead letter queue: a topic where records that fail processing can be sent, for sink connectors.
• REST API: the interface used to create, inspect, pause and delete connectors.

Source and sink

Everyday analogy

Think of a port with cranes. Ships arrive from many places with differently shaped containers. The cranes (connectors) know how to unload each ship type and place standard containers on the dock (Kafka). Other cranes load standard containers onto outbound trucks. The port authority (the Connect framework) schedules cranes, counts containers and restarts a crane that breaks. Nobody has to build a new crane for every ship.

Where it came from

Kafka at first offered only producer and consumer clients. Teams that wanted to ingest from a database or write to a file store each wrote their own service. These services repeated the same hard parts: tracking progress, handling failure, scaling out and monitoring. Quality varied widely.

The Kafka community introduced Kafka Connect in the middle of the 2010s as a standard framework for this job. The design goals were clear. Connectors should be simple to write, because the framework does the heavy lifting. Operations should be uniform: one REST interface and one configuration style for every connector. The service should scale horizontally and recover from failures without data loss.

Over time the ecosystem grew. Many connectors are maintained by the community or by vendors. A well known family of connectors for change data capture, named Debezium, reads the transaction logs of databases and turns every insert, update and delete into an event. Later versions of Connect added exactly-once support for some source connectors, better error handling, connector-level offsets management and cross-cluster replication tooling built on the same framework, such as MirrorMaker 2.

Specific version features change. Consult the documentation for your release.

How it works

Follow a source connector and a sink connector through their life.

Step 1: Start workers

You start one or more workers. In standalone mode a single process runs with a local file for offsets, which is good for development. In distributed mode several workers share a group id, coordinate through Kafka and keep state in internal topics. Use distributed mode in production.

Step 2: Create a connector

You send a configuration to the REST API. The configuration names the connector class, the number of tasks allowed, the topics and the connection settings. The worker stores the configuration in the config topic. All workers see it.

Step 3: Plan the tasks

The connector class inspects the configuration and decides how to divide the work. A database source might create one task per group of tables. A file sink might create one task per group of topic partitions. The connector returns task configurations to the framework, which spreads the tasks across workers.

Step 4: Source tasks read and produce

A source task polls the external system and returns a batch of source records. Each record includes a source partition and a source offset. These describe where the data came from, such as a file name and a position, or a log sequence number in a database.

The framework converts each record using the configured converters, applies transforms and writes the result to Kafka.

Step 5: Save progress

Once Kafka confirms the writes, the framework stores the source offsets in the offsets topic. If the task restarts, it asks for the last stored offset and resumes there. Because offsets are saved after writing, a crash between the write and the save can cause repeats, so the default guarantee for sources is at least once. Some source connectors can use exactly-once support in newer versions.

Picture a courier who notes the last delivered parcel number in a logbook after each trip. If the courier falls ill, a replacement reads the logbook and continues from there. A parcel might be delivered twice if the courier fell ill just before writing the note.

Step 6: Sink tasks consume and write

A sink task is built on a normal Kafka consumer. The framework reads records from assigned topic partitions, converts them to internal records, applies transforms and hands batches to the task. The task writes them to the target system.

The framework commits consumer offsets only after the task confirms that data was flushed. This yields at-least-once delivery. Connectors that write idempotently, for example by upserting on a key, make repeats harmless.

Step 7: Handle errors

Errors can occur during conversion, transformation or writing. The configuration controls behaviour.

• errors.tolerance=none stops the task at the first error. This is safe but strict.
• errors.tolerance=all skips bad records. Pair it with a dead letter topic and logging, so skipped records are not lost silently.
• Retries with backoff handle temporary failures.

Step 8: Rebalance and fail over

When a worker leaves or joins, the group rebalances and tasks move to other workers. A task that fails can be restarted through the REST API. Connect does not automatically restart a failed task by default in older versions, so monitor task status.

Step 9: Scale

Raise tasks.max to allow more parallel tasks, up to what the connector and the source can use. A sink can use at most as many tasks as there are topic partitions it reads. A source may be limited by the way the external system can be split. Add workers to spread the tasks.

Key concepts and terms

A worked example

This example captures changes from a PostgreSQL table and then writes them to object storage as files. All hosts and names are fake.

Worker settings

bootstrap.servers=broker1.example.internal:9092
group.id=connect-cluster-a
key.converter=io.confluent.connect.avro.AvroConverter
value.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://registry.example.internal:8081
value.converter.schema.registry.url=http://registry.example.internal:8081
config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-status
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3
plugin.path=/opt/connect/plugins

The internal topics need a replication factor of three in production so that the framework's own state is durable. The converters use a registry so schemas are stored once and not in every message. See Schema Registry.

A source connector for database changes

{
  "name": "orders-cdc-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "orders-db.example.internal",
    "database.port": "5432",
    "database.user": "cdc_reader",
    "database.password": "${file:/opt/connect/secrets.properties:db_password}",
    "database.dbname": "shop",
    "topic.prefix": "shop",
    "table.include.list": "public.orders,public.customers",
    "plugin.name": "pgoutput",
    "slot.name": "connect_orders",
    "snapshot.mode": "initial",
    "tasks.max": "1",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.delete.handling.mode": "rewrite"
  }
}

The password is read from a secrets file using a config provider, so it is not placed in the JSON. The unwrap transform flattens the change envelope into a plain row, with a marker when the row was deleted.

Submit it:

curl -X POST -H "Content-Type: application/json" \
  --data @orders-cdc-source.json \
  http://connect1.example.internal:8083/connectors

What the source does

1. On first start it takes a snapshot, reading all existing rows from the two tables.
2. It then follows the database change log, producing one record per insert, update or delete.
3. Records go to topics named shop.public.orders and shop.public.customers.
4. The record key is the table primary key, so all changes for one row go to the same partition in order.
5. It saves the log position as the source offset, so a restart resumes from the right place.

A sink connector for object storage

{
  "name": "orders-s3-sink",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "topics": "shop.public.orders",
    "s3.bucket.name": "example-lake-raw",
    "s3.region": "us-east-1",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "flush.size": "50000",
    "rotate.interval.ms": "600000",
    "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
    "path.format": "'dt'=YYYY-MM-dd",
    "partition.duration.ms": "86400000",
    "locale": "en-US",
    "timezone": "UTC",
    "timestamp.extractor": "Record",
    "tasks.max": "3",
    "errors.tolerance": "all",
    "errors.deadletterqueue.topic.name": "dlq.orders-s3-sink",
    "errors.deadletterqueue.context.headers.enable": "true",
    "errors.log.enable": "true"
  }
}

This writes Parquet files into daily folders in Amazon S3, starting a new file after 50,000 records or ten minutes. The dead letter topic keeps records that could not be written, with headers explaining the error.

Check status

curl http://connect1.example.internal:8083/connectors/orders-cdc-source/status
curl http://connect1.example.internal:8083/connectors?expand=status

A healthy answer lists the connector and each task as RUNNING. A task in FAILED state includes a stack trace.

Restart a failed task

curl -X POST http://connect1.example.internal:8083/connectors/orders-s3-sink/tasks/0/restart

A numeric picture

Suppose the orders table gets 400 changes per second. The sink flushes every 50,000 records or ten minutes, whichever comes first. At 400 per second, 50,000 records take about 125 seconds, so files are written about every two minutes. With three tasks reading three partitions, each task sees around 133 records per second and flushes roughly every six minutes. That produces many small files, which is why compaction later matters. See Iceberg compaction.

Using a single message transform to mask a field

"transforms": "mask,route",
"transforms.mask.type": "org.apache.kafka.connect.transforms.MaskField$Value",
"transforms.mask.fields": "email,phone",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "shop\\.public\\.(.*)",
"transforms.route.replacement": "raw_$1"

The first transform replaces sensitive fields with an empty value of the same type. The second renames the topic. See single message transforms for more.

Strengths and limitations

Strengths

• No custom code for common integrations. Install a plugin and configure it.
• Scalable. Add workers and tasks to raise throughput.
• Fault tolerant. State lives in Kafka, so workers are stateless and replaceable.
• Uniform operations. One REST interface and one config style for all connectors.
• Offset management built in. Resume after restart without extra work.
• Rich ecosystem. Connectors for databases, storage, search, queues and more.
• Transforms and converters. Light reshaping and format handling in the same service.
• Change data capture. Log-based connectors give low-impact, ordered change streams.

Limitations

• Light transformations only. Single message transforms work on one record at a time. Joins and aggregations belong in a stream processor such as Apache Flink.
• Connector quality varies. Some are well maintained, others are not. Test before relying on them.
• At-least-once by default. Duplicates are possible unless the sink is idempotent or exactly-once features apply.
• Schema and type mapping. Edge cases such as decimals, time zones and unusual types need care.
• Operational tuning. Rebalances, memory and plugin dependencies can cause trouble.
• Source load. Snapshots can put heavy read load on the source database.
• Plugin isolation issues. Conflicting library versions can break plugins if not isolated.
• Backfills can be awkward. Re-snapshotting a large table needs planning.

How it compares with alternatives

Many teams combine them: Kafka Connect brings data in and out, and a stream processor does the heavy logic between topics. For database replication specifically, see change data capture and data synchronization.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns, deal with mixed types, missing values and sometimes missing column names, and are often pre-trained on large collections of synthetic or real tables. Many support in-context learning, where labelled example rows are given at prediction time. Kafka Connect does not train these models. Its job is to move the tables that models read and the predictions they produce, reliably and without custom code.

Landing raw tables in the lake

The first challenge is simple: get the data out of the operational systems. Source connectors, especially change data capture, copy tables continuously into topics. Sink connectors write them to object storage or lake tables as files. Together they create a bronze layer of raw history that downstream jobs refine. See medallion architecture.

Keeping a table live

A change stream is a log of every insert, update and delete. A sink that upserts by primary key keeps a live copy of the table. A feature job or a model can query this table at any time for the latest rows. Tombstones propagate deletes, which matters for privacy and for removing stale context rows.

Preserving types and nulls

Tabular models treat numeric, categorical and date columns differently. Connect converters carry schemas, so a decimal stays a decimal and a timestamp stays a timestamp. Nulls stay nulls. This is helpful because many tabular models prefer to see the missingness pattern and learn their own handling, rather than receiving filled values. See imputation networks and masked cell modeling for learned approaches.

Column names and schema drift

When a column is added upstream, the source connector can emit the new field and the registry can check compatibility. For a model that expects a fixed column set, you can use transforms to drop or rename fields so the downstream table stays stable. Stable columns keep model inputs consistent. See schema evolution and schema drift detection.

Sampling and training snapshots

Because the sink writes time-partitioned files, you can build a training table as of a date by reading only the folders up to that date. This supports leakage-free experiments. The same files can be read by engines such as Trino or Apache Spark to draw samples for in-context learning with models like TabPFN and TabICL.

Writing predictions back

A sink connector can send model outputs from a topic back to a database table, a search index or a lake table. For example, anomaly scores or sector labels produced by a scoring service can be written beside the source rows, so analysts and applications can use them. See anomaly detection and semantic sector heads.

Privacy at the edge

Masking transforms can hide or hash sensitive fields before data leaves Kafka for a lake that model teams use. This reduces the risk of training on personal data that should never leave its source. Pair transforms with proper governance and, where needed, tokenization tools.

Continual learning loops

Connectors also close the loop. Source connectors ingest new outcomes, such as confirmed labels, from operational systems. These join earlier predictions to make fresh training data. Pipelines then refresh context sets or trigger fine-tuning when drift is detected.

What Connect does not do

Connect does not clean data deeply, build features that need joins, train models or serve predictions. Put those in stream processors, batch transformations and model services. For model concepts see the architecture, research and benchmarks pages.

Practical guidance and best practices

Run distributed mode in production

Use at least three workers for resilience. Give internal topics a replication factor of three. Keep worker configuration in version control.

Treat connectors as code

Store connector configurations as files in a repository. Apply them through a pipeline that calls the REST API. This gives review, history and repeatable environments. Manage them with infrastructure tooling such as Terraform where it fits.

Secure secrets

Use config providers to load passwords from files or secret stores. Never put credentials in the JSON you post. Restrict access to the REST API, since it can reveal and change configurations.

Choose converters deliberately

• Use Avro or Protobuf with a registry for production data.
• Use JSON with embedded schemas only for development, since it makes messages large.
• Keep key and value converters consistent across connectors that share topics.

Plan the snapshot

• Take the initial snapshot at a quiet time.
• Use a read replica where possible.
• Limit the snapshot to needed tables.
• Understand how the connector handles changes that occur during the snapshot.

Design for idempotent sinks

• Use upsert mode with a primary key for database sinks.
• Use deterministic file naming for object stores where supported.
• Make downstream readers deduplicate by key and version if needed.

Handle errors on purpose

• Use a dead letter topic for sinks.
• Log and alert on dead letter volume.
• Replay fixed records from the dead letter topic after repair.

Size tasks and flush settings

• Match tasks.max to partitions and source capacity.
• Tune flush size and rotation time to balance latency and file size.
• Avoid very small files. Schedule compaction in the lake.

Monitor everything

• Task state and restart counts.
• Source lag, for example database log lag.
• Sink consumer lag.
• Records per second in and out.
• Dead letter queue size.
• Worker memory and rebalance frequency.

Export metrics to Prometheus and build alerts in Grafana.

Test upgrades

Test connector and worker upgrades in a staging environment with real volumes. Check type mappings and offsets after each upgrade.

Use managed options where fitting

If you run on a cloud platform, a managed Kafka service such as Amazon MSK can host Connect workers or offer a hosted Connect feature. Check the limits and plugin support before committing.

A rollout plan in phases

1. Pilot. Pick one low-risk table. Run a source connector and a sink connector in a test environment.
2. Validate. Compare row counts and checksums between the source and the target for a full day.
3. Harden. Add a dead letter topic, alerts on task failures and a runbook for restarts.
4. Scale. Add more tables and connectors in small groups, watching worker memory and rebalances.
5. Govern. Record each connector's owner, source, target and data classification in a catalog.
6. Review. Revisit flush settings and file sizes after a month of real traffic.

Questions to ask before adding a connector

• Who owns the source and has agreed to the read load?
• What is the expected change rate and the largest burst?
• Which fields are sensitive and need masking?
• How are deletes represented, and does the target honour them?
• What happens if the connector is down for a day?
• How will schema changes be announced and tested?
• What delivery guarantee does the target need?

Answering these early prevents most production surprises.

Capacity notes

• Estimate peak records per second and average record size for every topic a connector touches.
• Multiply by the replication factor to see the load on the Kafka cluster.
• Keep worker heap large enough for the biggest batch a sink holds before flushing.
• Reserve spare worker capacity so that losing one worker does not overload the rest.

Common mistakes

1. Running standalone mode in production. A single process is a single point of failure.
2. Low replication on internal topics. Losing the offsets topic means losing progress.
3. Credentials in plain JSON. They leak through the REST API and logs.
4. Tolerating all errors with no dead letter queue. Bad records vanish unnoticed.
5. Setting tasks.max higher than useful. Extra tasks stay idle or overload the source.
6. Heavy transforms. Complex logic in transforms is hard to test and slow. Use a stream processor.
7. Ignoring type mapping. Decimals, big integers and time zones may change on the way.
8. Forgetting tombstones. Deletes never reach the target, or compacted topics keep deleted keys.
9. Mixing converters. A consumer expects Avro but the topic holds JSON, and reading fails.
10. Snapshotting a busy primary. Production queries slow down.
11. No monitoring of failed tasks. A failed task stays down until someone restarts it.
12. Not watching the database log retention. If the connector is down too long, the log it needs may be deleted.
13. Skipping schema compatibility rules. An upstream change breaks the sink.
14. Many tiny output files. Query engines slow down without compaction.
15. Assuming exactly-once everywhere. Check the guarantee of each connector and target.

Frequently asked questions

What is Kafka Connect used for?

Kafka Connect is used to stream data between Kafka and other systems without writing integration code. Typical jobs include capturing database changes into topics, writing topics to object storage or a warehouse, and indexing events into a search system.

What is the difference between a source and a sink connector?

A source connector pulls data from an external system and writes it to Kafka topics. A sink connector reads from Kafka topics and writes to an external system. Both run inside the same framework and use the same management interface.

Is Kafka Connect part of Apache Kafka?

Yes, the framework is part of the Apache Kafka project and ships with it. Individual connectors are separate plugins, and many are developed by the community or by vendors rather than by the core project.

What is the difference between a connector and a task?

A connector is the configured job, and it decides how to split the work. A task is a running unit that does the actual reading or writing. One connector can have several tasks running in parallel on different workers.

Does Kafka Connect support exactly-once delivery?

By default, delivery is at least once. Newer versions add exactly-once support for some source connectors, and sinks can reach the same effect by writing idempotently, for example with upserts on a key. Always check what your specific connector and target support.

What are single message transforms?

They are small, configurable steps that change each record as it passes through, such as renaming a topic, masking a field, adding a timestamp or flattening a nested structure. They work on one record at a time, so they are not suitable for joins or aggregations.

How does Kafka Connect handle failures?

State is stored in Kafka, so a failed worker can be replaced and its tasks move to others. Offsets let a task resume where it stopped. Error settings decide whether a bad record stops the task or goes to a dead letter topic. Failed tasks usually need a restart through the REST API or automation.

Can Kafka Connect do change data capture?

Yes, through source connectors that read a database transaction log and emit an event for each row change. These connectors take an initial snapshot and then stream changes, which keeps downstream copies of tables up to date with low load on the source.

Do I need a schema registry with Kafka Connect?

It is not required, but it is strongly recommended for production. A registry lets you use compact binary formats with enforced compatibility rules, so changes in source tables do not break downstream readers unexpectedly.

How does Kafka Connect help machine learning on tables?

It lands operational tables in a lake continuously, keeps live copies through change data capture, preserves types and nulls, and writes model predictions back to systems where people can use them. It handles movement only, so training, feature joins and serving happen in other tools.

Glossary

• Config provider: A mechanism for loading secrets or settings from external sources into connector configs.
• Connector: A configured job that moves data in or out of Kafka.
• Converter: A component that serializes and deserializes record keys and values.
• Dead letter queue: A topic that stores records a sink failed to process.
• Distributed mode: A cluster mode where workers share state through Kafka.
• Offset: A position marker, either in a Kafka partition or in a source system.
• Plugin path: The folder where connector plugins are installed.
• REST API: The HTTP interface for managing connectors.
• Schema registry: A service that stores and checks record schemas.
• Single message transform: A per-record change applied inside Connect.
• Sink connector: A connector that writes data from Kafka to another system.
• Snapshot: An initial full read of source data before change streaming starts.
• Source connector: A connector that reads from another system into Kafka.
• Standalone mode: A single-process mode for development.
• Task: A parallel worker unit inside a connector.
• Tombstone: A record with a null value that marks a deletion.
• Worker: A process that runs connectors and tasks.

Summary

Kafka Connect is the standard way to move data in and out of Kafka without writing custom services. Source connectors read from databases and other systems, sink connectors write to storage, search and warehouses, and the framework provides scaling, offset tracking, error handling and a single management interface. Light transforms and converters shape records on the way.

For tabular foundation models, Kafka Connect supplies the plumbing. It lands raw tables in the lake, keeps live copies through change data capture, preserves types and missing values, masks sensitive fields, and carries predictions back to the systems that use them. Run it in distributed mode, keep configurations in version control, protect secrets, plan snapshots, monitor failed tasks and dead letter topics, and leave heavy logic to stream processors and batch engines.

Related

It belongs to Event streaming in my stack, under Data engineering and lakehouse. Others in the same group: