Event-driven architecture

Overview

Event-driven architecture is a way of building systems in which components communicate by publishing and reacting to events, rather than by calling each other directly. An event is a fact that something happened: an order was placed, a sensor reported a reading, a payment cleared. A producer announces the fact. Any number of consumers can react to it, each in its own time and in its own way. The producer does not need to know who is listening.

This style gives loose coupling, easy scaling, natural audit trails and the ability to add new features by adding new listeners. The price is added complexity: eventual consistency, harder debugging, duplicate and out-of-order messages, and the need to manage schemas and contracts with care. Platforms such as Apache Kafka, RabbitMQ and Amazon MSK provide the transport. Around them sit tools for contracts, replication and processing. This guide explains the ideas, the patterns, the trade-offs, and how event streams become the tables that data and machine learning systems depend on.

What it is

In a request-driven design, service A calls service B and waits for an answer. A needs to know B's address, B's interface and whether B is up. In an event-driven design, service A writes an event to a broker and moves on. Service B, and C, and D, read the event when they are ready.

The main building blocks

• Event producer: A component that detects a change and publishes an event.
• Event broker or log: The infrastructure that accepts, stores and delivers events. It may be a queue, a topic-based broker or a durable log.
• Event consumer: A component that subscribes and reacts.
• Event schema: The agreed shape of an event. See Schema Registry.
• Event channel: A topic, queue or stream that groups events of one kind.

Kinds of events

The difference between an event and a command matters. An event says what has happened and cannot be rejected. A command asks for something and can fail. Mixing them leads to confusion about who owns a decision.

Delivery styles

• Publish and subscribe: Many consumers get a copy of each event.
• Queue with competing consumers: Each message goes to one worker out of many, for load sharing.
• Log-based streaming: Events are kept in an ordered log, and each consumer tracks its own position, so history can be replayed.

Where it came from

Messaging between programs is old. Message queues appeared in enterprise systems decades ago, to link applications that could not be tied together directly. Publish and subscribe models grew from the same need, and enterprise integration patterns, described in a well-known book in the early 2000s, gave the field a shared vocabulary for messages, channels, routers and translators.

Several trends pushed event-driven ideas into the mainstream. Web scale systems needed to handle huge write loads and to isolate failures. Microservices raised the question of how independent services should share data without tight coupling. Domain-driven design offered the idea of domain events. Log-based platforms made it practical to keep events for a long time and replay them. Cloud providers added managed brokers and serverless functions that react to events. Today, event-driven design is common in payments, logistics, retail, telecoms, industrial monitoring and many other fields.

How it works

Step by step

1. Something happens in a producer, such as a customer placing an order.
2. The producer builds an event with a type, an id, a time, and the data that matters.
3. The producer sends the event to a channel on the broker. The broker stores it.
4. Consumers subscribed to that channel receive the event. In a log-based system, they pull events at their own pace.
5. Each consumer processes the event: update a database, call another system, send an email, compute a metric.
6. A consumer may publish new events, for example "PaymentRequested" after "OrderCreated". Chains of events form workflows.
7. Consumers record their progress, with an acknowledgement or a committed offset, so that after a restart they continue from the right place.

An everyday analogy: a town notice board. Someone pins a note saying "bakery has fresh bread". The baker does not phone each neighbour. People who care read the board when they pass by. New neighbours can read older notes to catch up, if the board keeps them. The baker never knows or needs to know who read the note.

Choreography and orchestration

When a business process spans many services, there are two ways to organise it.

Many systems use choreography for simple reactions and orchestration for long, critical processes.

Core patterns

Event sourcing. Instead of storing only the current state, you store the full sequence of events that led to it. The current state is a fold over the events. This gives a perfect audit trail and the ability to rebuild state or build new views from history. It also adds complexity: you need snapshots, versioning of event types and careful handling of corrections.

CQRS, command query responsibility segregation. The write model and the read model are separated. Commands change state and produce events. Read models are built from events and optimised for queries. It pairs well with event sourcing, but each can be used alone.

Saga. A long business transaction is split into steps across services, each with a compensating action to undo it if a later step fails. Sagas replace distributed transactions with a chain of local ones.

Outbox. A service writes its state change and an outgoing event to its own database in one local transaction. A separate process reads the outbox and publishes events. This avoids the problem of updating a database and a broker as two separate steps where one may fail.

Change data capture. A tool reads a database's change log and publishes changes as events. This brings events out of systems that were not built to emit them. Kafka Connect with a suitable source connector is a common way to do this.

Dead letter queue. Messages that cannot be processed after some tries are set aside with error details for later review, so that one bad message does not block the stream.

Idempotent consumer. A consumer is written so that processing the same event twice has the same effect as once. This is the standard answer to duplicate delivery.

Competing consumers. Many workers read from one queue or consumer group, which shares the work.

Event enrichment and routing. A processor reads events, adds data or decides where to send them. Light changes can be done with single message transforms; heavier ones with Apache Flink or Spark Streaming.

Delivery guarantees

True exactly-once delivery across arbitrary systems is not possible in general. What you can get is exactly-once effects within a defined boundary, using transactions, idempotent producers and idempotent sinks. Be wary of claims that skip the boundary.

Ordering

Ordering is guaranteed only within a defined scope, usually a partition or a queue. If order matters for one customer or one order, use that id as the message key so all its events go to the same partition. There is no cheap global order across all events at scale. Design so you do not need one.

A small publisher and consumer

The example below uses Python with placeholder hosts. It shows an idempotent consumer using an event id.

import json
from confluent_kafka import Producer, Consumer

producer = Producer({"bootstrap.servers": "broker.example.internal:9092"})

def publish_order_created(order_id, amount):
    event = {
        "event_id": f"order-created-{order_id}",
        "type": "OrderCreated",
        "occurred_at": "2026-01-15T10:00:00Z",
        "data": {"order_id": order_id, "amount": amount},
    }
    producer.produce("orders", key=str(order_id), value=json.dumps(event))
    producer.flush()

consumer = Consumer({
    "bootstrap.servers": "broker.example.internal:9092",
    "group.id": "billing",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["orders"])

seen = set()  # in real systems this is a database table with a unique key

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    event = json.loads(msg.value())
    if event["event_id"] in seen:
        consumer.commit(msg)
        continue
    # ... apply the business effect here ...
    seen.add(event["event_id"])
    consumer.commit(msg)

In production the deduplication store must be durable and written in the same transaction as the business effect where possible.

A transactional outbox in SQL

BEGIN;

UPDATE orders SET status = 'PAID' WHERE order_id = 42;

INSERT INTO outbox (event_id, aggregate_id, type, payload, created_at)
VALUES ('evt-0001', '42', 'OrderPaid',
        '{"order_id":42,"amount":59.9}', now());

COMMIT;

A relay process then reads new rows from outbox, publishes them, and marks them as sent. If the relay crashes after publishing but before marking, the event may be published twice. That is why consumers must be idempotent.

Schema and contract management

Events are public interfaces. Changing them carelessly breaks consumers that you may not even know about. A registry with compatibility rules, versioned event types, and a clear deprecation process are standard practice. Add fields, do not repurpose them. See Schema Registry and data contract validation.

Replicating across sites

Systems that must survive the loss of a site replicate their event logs. MirrorMaker 2 does this for self-managed Kafka, and MSK Replicator does it for Amazon MSK. Applications then need idempotent consumers, location-independent configuration and a plan for schemas in the second site.

Key concepts and terms

A worked example

Consider an online store. It has an order service, a payment service, an inventory service, a shipping service and an analytics team.

The flow

1. The customer places an order. The order service saves it with status PENDING and, using an outbox, publishes OrderCreated.
2. The payment service reads OrderCreated, charges the customer, and publishes PaymentCompleted or PaymentFailed.
3. The inventory service also reads OrderCreated and reserves the items. It publishes StockReserved or StockUnavailable.
4. The order service listens to all of these. When it has both PaymentCompleted and StockReserved, it marks the order CONFIRMED and publishes OrderConfirmed.
5. The shipping service reads OrderConfirmed and schedules a delivery.
6. If PaymentFailed or StockUnavailable arrives, the order service starts compensation: release stock, refund payment if needed, and mark the order CANCELLED. This is a saga.
7. The analytics team reads every event into a lake table, without asking any service to change.

What is gained

• The order service does not know the payment or inventory services exist.
• Adding a fraud check later needs only a new consumer of OrderCreated.
• If shipping is down for an hour, events wait in the log. When it returns, it catches up.
• Analytics has the full history of what happened and when.

What must be handled

• Events can arrive twice. Each service uses the event id to avoid repeating a charge.
• PaymentCompleted may arrive before StockReserved, or the other way around. The order service tracks both and does not assume an order.
• Late or missing events need timeouts. If neither arrives within ten minutes, the order service cancels.
• Event schemas are versioned. When OrderCreated gains a coupon_code field, it comes with a default so old consumers continue to work.

A tiny state machine

class OrderState:
    def __init__(self):
        self.paid = False
        self.reserved = False
        self.status = "PENDING"

    def on_event(self, event_type):
        if event_type == "PaymentCompleted":
            self.paid = True
        elif event_type == "StockReserved":
            self.reserved = True
        elif event_type in ("PaymentFailed", "StockUnavailable"):
            self.status = "CANCELLED"
            return
        if self.paid and self.reserved and self.status == "PENDING":
            self.status = "CONFIRMED"

o = OrderState()
o.on_event("StockReserved")
o.on_event("PaymentCompleted")
print(o.status)  # CONFIRMED

The same result appears whichever event arrives first, which is the property you want.

From events to a table

The analytics team uses a sink to write each event into a table with columns event_id, type, occurred_at, order_id, amount and status. A query then answers business questions:

SELECT date_trunc('day', occurred_at) AS day,
       count(*) FILTER (WHERE type = 'OrderCreated')   AS created,
       count(*) FILTER (WHERE type = 'OrderConfirmed') AS confirmed,
       count(*) FILTER (WHERE type = 'OrderCancelled') AS cancelled
FROM lake.order_events
GROUP BY 1
ORDER BY 1;

Because events carry time and ids, the same table can also feed a model that predicts cancellation, by joining early events with later outcomes.

Strengths and limitations

Strengths

• Loose coupling: Producers and consumers evolve independently, with a shared contract only.
• Scalability: Consumers scale out by adding instances. Brokers partition load.
• Resilience: A slow or failed consumer does not stop the producer. Events wait.
• Extensibility: New consumers can be added without changing producers.
• Audit and replay: A durable log is a record of what happened, and can rebuild state or feed new features.
• Real-time reaction: Systems respond as things happen, not in nightly batches.
• Natural fit for data platforms: Events can be landed in lakes and warehouses for analytics and models.

Limitations

• Eventual consistency: Users may see old data for a short time, and designs must accept that.
• Debugging difficulty: A request spans many services and a time delay. Tracing and correlation ids are essential.
• Duplicates and disorder: Consumers must handle repeats and out-of-order delivery.
• Schema governance: Events become long-lived public contracts, and changes need discipline.
• Operational load: Brokers, registries, connectors and monitoring all need to be run.
• Hidden dependencies: In choreography, the real workflow can be spread across many services, and hard to see.
• Data privacy: Events kept for a long time may hold personal data. Deleting it from an immutable log needs a design, such as encryption keys per subject or tombstones with compaction.
• Overuse: Not every interaction should be an event. A simple query that needs an immediate answer is often better as a direct call.

How it compares with alternatives

Most real systems mix these. Use synchronous calls for queries that need an immediate answer, events for facts that others may need, and batch for large, non-urgent movement.

Choosing a broker

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns, handle mixed numeric and categorical types and missing values, may work without meaningful column names, and are pre-trained on large numbers of synthetic or real tables. They can learn from a handful of supplied rows through in-context learning, and they can be updated over time. Event-driven architecture is not a model technique. It is the way much of the world's tabular data is produced, moved and kept up to date. Understanding it helps you build better tables for such models.

Events become rows

Every event is a candidate row. An OrderCreated event with an id, a time, an amount and a country is a row with four columns. A stream of such events is a table that grows over time. Event-driven systems therefore generate exactly the kind of data that tabular models are built for: rows of mixed types, with some missing values. See /model/architecture for how such rows are represented inside a model.

Mixed types and missing values

Events from different producers differ in what they include. Optional fields are common, so missing values appear naturally. A good schema marks which fields are nullable and which have defaults. That distinction helps model builders: true missing values can be left missing for the model or imputed, while defaults that stand for "unknown" should be turned back into missing before training. Tabular models and imputation methods handle gaps well when the gaps are honest.

Time and ordering

Events carry both an event time (when it happened) and a processing time (when the system saw it). Tabular models do not understand time unless you give them time-aware columns and careful splits. Build training tables using event time, and split by time so the model does not learn from the future. Out-of-order and late events mean a table built today may change tomorrow; record the watermark up to which the table is complete.

Labels from later events

Many labels are themselves events: a payment default, a cancellation, a failure alarm. Event-driven data makes it easy to join early events with later outcomes. The risk is leakage. If a feature is created by an event that happens after the outcome, the model will seem to predict well but fail in production. Add a rule that features may only use events before a defined cut-off time.

In-context learning

In-context learning needs a small set of relevant labelled rows. An event stream, landed in a table, can supply fresh examples that match today's behaviour. A service can select the latest examples for a segment, build a context table and ask a model to predict for new rows without retraining. The freshness of the context equals the lag of the pipeline, so measure it. See /model/research for background on this style of learning.

Continual learning

Because events keep arriving, data changes: new categories, shifts in distributions, new columns. Continual learning means updating models as this happens. Event platforms help with replay: you can rebuild a table for any period. Schema drift detection and data contract validation tell you when a change in events needs a model refresh or a pipeline fix.

Imputation and anomaly detection

Streams are a natural place for anomaly detection. Events arrive continuously, and a model or a rule can flag unusual ones as they pass. Keep two ideas apart: a contract violation (a field missing or of the wrong type) is a data quality problem to fix at the door, while a statistical anomaly (a normal-looking event with unusual values) is something a model can judge. Imputation is usually done later, in the table-building step, so that the raw events stay faithful to what was observed.

No column names

Events from sensors or legacy systems sometimes have cryptic field names. Tabular models can often work with limited naming help, but your own pipeline benefits from documentation. Keep a mapping between technical names and meanings in the schema, and preserve it with lineage.

Sector and semantic classification

Event streams from many organisations or business lines may need classifying by sector or meaning, for example tagging merchants, devices or documents. Fields such as descriptions, codes and categories supply the signal. Keep these fields intact through the pipeline and avoid over-masking text that the classifier needs. Where privacy requires masking, use single message transforms or a de-identification step, and see Spark UDF de-identification for batch approaches.

Pre-training data and benchmarks

For pre-training and evaluation, events can be landed in open table formats such as Apache Iceberg or Delta Lake, then sampled into training tables. Lineage tools like data lineage record where each table came from. For public evaluation results on tabular tasks, see /model/benchmarks.

A reference flow

1. Services publish events with registered schemas.
2. A connector lands raw events into a lake table.
3. Validation checks contracts and drift.
4. A job deduplicates, aligns types and applies defaults-to-null rules.
5. A table builder creates versioned training and context tables with a time cut-off.
6. A tabular model reads them for training, in-context prediction, imputation or anomaly scoring.
7. Predictions can be published as new events, closing the loop.

Practical guidance and best practices

Design events well

• Name events in the past tense and in the language of the business: OrderPlaced, not PlaceOrder or OrderTable_Insert.
• Include a unique event id, event type, event time, schema version and a correlation id.
• Keep events small and focused. Avoid copying whole databases into each event.
• Decide whether events carry state or only a reference. State-carrying events reduce calls back to the source, but become larger and harder to change.
• Use stable keys, like customer id, for ordering where needed.

Handle failure

• Make every consumer idempotent.
• Use retries with backoff, and a dead letter queue for events that keep failing.
• Use the outbox pattern when a service must change its database and publish an event.
• Define timeouts for processes that wait for several events.
• Plan for poison messages that always fail.

Govern contracts

• Use a registry with compatibility checks.
• Treat events as public APIs: version them, document them, deprecate with notice.
• Test consumers against contract examples in build pipelines.
• Keep a catalogue of topics with owners, descriptions and retention.

Observe the system

• Pass a correlation id through every event, and log it.
• Track consumer lag, processing time, error rates and dead letter volume.
• Build dashboards by business flow, not just by service.
• Use distributed tracing where possible.

Protect data

• Decide how long events are kept, and why.
• Keep personal data out of events where possible. Use ids and look up details when needed.
• Consider encrypting sensitive fields with keys per subject so that deletion is possible by deleting the key.
• Apply access controls to topics, by team and by purpose.

Start small

• Pick one flow with clear value and try the pattern there.
• Keep synchronous calls where they are simpler.
• Grow the number of event types slowly, and review them regularly.

Common mistakes

• Treating the broker as a database for every query, rather than building read models.
• Using events as remote procedure calls in disguise, with a request event and a reply event for every call.
• Skipping idempotency and then having double charges or duplicate emails.
• Assuming a global order of events.
• Publishing the internal database structure as the event schema, which locks consumers to implementation details.
• Making events too big, or too thin to be useful.
• Changing the meaning of a field without changing the schema or the version.
• No ownership of topics, so nobody knows who to ask about a change.
• Updating a database and publishing an event as two separate steps with no outbox.
• Losing track of the real process because choreography hides it.
• Forgetting that deleting personal data from a log needs a plan.
• Building training tables from events without a time cut-off, which leaks the future into the model.
• Treating a table built from events as final, when late events can still arrive.

Frequently asked questions

What is event-driven architecture?

It is a design style in which components communicate by publishing and reacting to events, which are records of things that happened. Producers do not call consumers directly. A broker stores and delivers events, and consumers react in their own time. This gives loose coupling and scalability.

What is the difference between an event and a message?

A message is any unit sent between components. An event is a message that states a fact about the past, such as an order being placed. Commands are messages too, but they ask for an action and can be refused. The difference affects who owns a decision and how consumers should react.

When should I not use an event-driven design?

Avoid it when you need an immediate, consistent answer, such as a simple lookup, or when the system is small and tightly related. Events add moving parts, and the benefits appear with many consumers, uneven load, or a need for history. A direct call is often simpler for a single, synchronous need.

How do I deal with duplicate events?

Make consumers idempotent. Give each event a unique id and store the ids of processed events, ideally in the same transaction as the business effect. Then processing the same event twice does no harm. Platforms offer transactions and idempotent producers that reduce duplicates, but consumers should still be safe.

Is exactly-once delivery possible?

Within a defined boundary, such as inside one streaming platform with transactions, you can get exactly-once effects. Across arbitrary systems you generally cannot guarantee it, so the practical approach is at-least-once delivery with idempotent processing.

What are event sourcing and CQRS?

Event sourcing stores state as a sequence of events instead of overwriting current values, so you can rebuild state and keep a full audit trail. CQRS separates the model for writing from the model for reading, with read models built from events. They are often combined, but each can be used alone, and neither is required for event-driven design.

How do schemas fit in?

Events are long-lived contracts between teams, so their shape must be managed. A schema registry stores versions of event schemas and checks compatibility when they change. Adding optional fields with defaults is safe, while renames and type changes need a plan. Good schema discipline prevents silent breakage.

How do events help with machine learning on tables?

Each event is a natural row with an id, a time and typed fields, and streams of events produce growing tables. Because logs can be replayed, you can rebuild training tables for any period. Teams need to handle late events, duplicates, missing values and time-based leakage to produce reliable training and context tables.

How do I debug a system built on events?

Use correlation ids that follow a business flow through all events, structured logs, distributed tracing and dashboards organised by flow. Keep a catalogue of topics and owners. Being able to replay events into a test environment is also a powerful debugging tool.

How does event-driven design work across regions?

Event logs can be replicated between sites with tools such as MirrorMaker 2 or a managed replicator. Replication is asynchronous, so consumers must be idempotent and applications must tolerate a short gap after a sudden failure. Plan schema availability and failover procedures as part of the design.

Glossary

• Backpressure: A signal or mechanism that slows producers when consumers fall behind.
• Broker: Infrastructure that accepts, stores and delivers events.
• Choreography: Coordination by services reacting to each other's events.
• Command: A request for an action.
• Compensating action: A step that undoes the effect of an earlier step in a saga.
• Consumer group: Consumers that share the work of reading a topic.
• Correlation id: An id carried through related events to link them.
• CQRS: Separate models for commands and queries.
• Dead letter queue: A holding place for messages that fail repeatedly.
• Event: A record of something that happened.
• Event sourcing: Storing state as a sequence of events.
• Eventual consistency: Agreement across parts of a system after a short delay.
• Idempotency: The property that repeating an operation has the same effect as doing it once.
• Offset: A position in a log partition.
• Orchestration: A central coordinator directing a process.
• Outbox: A table used to publish events reliably from a database.
• Partition: An ordered slice of a topic.
• Saga: A multi-step process with compensating actions.
• Watermark: The time up to which events are considered complete.

Summary

Event-driven architecture builds systems around events: facts that producers publish and consumers react to, through a broker or a durable log. It brings loose coupling, scalability, resilience, audit trails and replay, and it asks for discipline in return: idempotent consumers, managed schemas, careful ordering, observability and data governance. Core patterns include event sourcing, CQRS, sagas, the outbox, change data capture and dead letter queues.

For tabular foundation models, event streams are the source of the rows. Events carry typed fields, nullable values, ids and times, and a replayable log lets teams rebuild tables for any period. To turn them into good training and context tables, deduplicate, respect event time, avoid leakage, record watermarks, and watch for schema drift. Related reading: Apache Kafka, Kafka Connect, Schema Registry, MirrorMaker 2, MSK Replicator and Apache Flink.

Related

It belongs to Event streaming in my stack, under Data engineering and lakehouse. Others in the same group: