Event sourcing

Overview

Event sourcing is a way of storing data in which every change is saved as an immutable event, and the current state is calculated from those events. Instead of keeping only the latest balance of a bank account, an event-sourced system keeps "Account opened", "Deposited 500", "Withdrew 120", "Deposited 40", and works out that the balance is 420. The events are the source of truth; any table showing current state is derived from them and can be rebuilt.

An everyday analogy is an accountant's ledger, or a chess game recorded move by move. A chess board shows only where the pieces are now. The score sheet shows every move in order. From the score sheet you can replay the game to any point, see exactly how a position arose, and settle any argument. You cannot do that from the board alone. Event sourcing keeps the score sheet.

The approach suits domains where history matters as much as the present: finance, orders, logistics, healthcare records, compliance-heavy workflows. It pairs naturally with CQRS, where events feed read models, and with domain-driven design, where domain events already describe what happens in business terms. For data and AI teams, an event log is also a rich source for features and training data, because it records what happened and when, not only the final outcome.

What it is

Event sourcing has a small set of parts.

Each event has a position in its stream. Appends usually use optimistic concurrency: a writer says "append this event if the stream is still at version 7", and the store rejects the write if someone else got there first.

Where it came from

The idea is old: accountants have kept append-only ledgers for centuries, and databases have long used transaction logs internally. As a named software pattern, event sourcing was described by Martin Fowler in an article in 2005. Greg Young developed and popularised it from the late 2000s, together with CQRS, through talks, courses and his work on a dedicated event store database.

Through the 2010s event sourcing became associated with domain-driven design and microservices. Frameworks and libraries such as Axon Framework for Java and Marten for .NET, along with dedicated event store databases, made it easier to adopt. The growth of streaming platforms such as Apache Kafka also made event logs familiar, although a streaming log and an event store serve somewhat different needs.

How it works

1. Receive a command

A user asks to withdraw 100 from account A-17. The command handler must decide whether that is allowed.

2. Load the aggregate

The handler reads every event in stream account-A-17, or the latest snapshot plus later events, and applies them in order to rebuild current state: balance 420, account open, no freeze.

3. Decide

The aggregate checks rules. Is the account open? Is the balance enough? If yes, it produces a new event, MoneyWithdrawn { amount: 100 }. If not, it rejects the command. The aggregate itself does not change the database; it only decides which events should happen.

4. Append

The handler appends the new event with an expected version. If another withdrawal was appended in the meantime, the append fails and the handler retries with fresh state. Appended events are never edited or deleted in normal operation.

5. Project

Subscribers receive the new event and update read models: an account balance table, a monthly statement, a fraud-detection feature. Each projection tracks its position so it can resume after a restart.

6. Replay and rebuild

Because the events are complete, any read model can be deleted and rebuilt by replaying the streams, and new read models can be created from historical events. You can also reconstruct state as of any past time, which supports audits and debugging.

7. Evolve

Event schemas change over time. Teams version events and use upcasters or new event types, rather than rewriting history.

Key concepts and terms

A worked example

This scenario is illustrative. A logistics company tracks parcels. Its old system stored one row per parcel with a status column that was overwritten on each scan. When a customer disputes a late delivery, support can see only the final status, not the path.

The team moves parcel tracking to event sourcing. Each parcel has a stream. Events include ParcelRegistered, ArrivedAtHub { hub: "Izmir-2", at: 08:14 }, LoadedOnVehicle, DeliveryAttemptFailed { reason: "recipient absent" } and Delivered. A typical parcel has 9 events; with 120,000 parcels a day, the store grows by just over a million events daily. Parcels are short-lived, so snapshots are unnecessary.

Three projections run: a current-status table for the customer website, a per-hub throughput table for operations, and an hourly feed of hub-to-hub transit times for a delay-prediction model. Six months later the data science team wants a new feature, "number of failed attempts in the last 30 days per address". They write a new projection and replay half a year of events to build it, without changing the write side. When a customer complains, support replays one stream and sees that the parcel sat at a hub for 19 hours after a vehicle breakdown. For privacy, recipient names and addresses are stored encrypted with a per-customer key, so a deletion request can be honoured by destroying that key.

Strengths and limitations

Strengths:

• A complete, ordered audit trail of every change.
• Ability to reconstruct state at any past moment.
• New read models and analytics can be built from history.
• Natural fit with domain events, CQRS and event-driven integration.
• Appends are simple and fast, and concurrency conflicts are explicit.

Limitations:

• A different mental model that takes time for teams to learn.
• Event schema evolution is permanent work, because history cannot simply be migrated.
• Queries need projections; you cannot easily ask ad-hoc questions of the event store.
• Eventual consistency between writes and read models.
• Deleting personal data conflicts with immutability and needs a deliberate design.
• Long streams need snapshots to keep loading fast.

How it compares with alternatives

Practical guidance and best practices

1. Use event sourcing for the parts of a system where history and auditability are central, not everywhere.
2. Name events in business language and past tense, and include enough data to be meaningful later.
3. Keep streams small and per entity; very long streams signal a modelling problem.
4. Use optimistic concurrency on every append.
5. Plan event versioning from day one, with schema definitions and upcasters.
6. Make projections idempotent and rebuildable, and test full rebuilds regularly.
7. Separate personal data, for example by encryption with per-subject keys, so deletion requests can be met.
8. Add snapshots only when measurements show loading is slow.

Common mistakes

• Storing technical events like "RowUpdated" instead of meaningful domain events.
• Querying the event store directly for reports instead of building projections.
• Editing past events in place to fix bugs, destroying the audit value.
• Using a general message broker as the only event store without per-stream concurrency control.
• Ignoring privacy law until a deletion request arrives.
• Applying event sourcing to a simple CRUD domain and paying the cost for no benefit.

Frequently asked questions

Is event sourcing the same as event-driven architecture?

No. Event-driven architecture is about services communicating through events. Event sourcing is about how one service stores its own state. A system can do either without the other.

Can I use Kafka as an event store?

Kafka is a durable log and works well for distributing events, but it lacks convenient per-entity streams with optimistic concurrency checks. Many teams use a dedicated event store or a database for the source of truth and Kafka for distribution.

How do I handle GDPR deletion requests?

Common techniques are keeping personal data out of events and referencing it, or encrypting it with a per-person key and deleting the key when required, often called crypto-shredding.

What are snapshots for?

They speed up loading aggregates with many events by storing state at a version, so only later events need replaying. They are an optimisation and can be rebuilt.

How do I change an event's structure?

Add a new version, keep old events as they are, and convert old versions to the new shape when reading using an upcaster. Avoid rewriting stored history.

Does event sourcing help machine learning?

Yes, often. Events record the sequence and timing of what happened, so they support point-in-time correct features and training data that state-only tables cannot provide.

Glossary

• Event: an immutable past-tense fact.
• Event stream: the ordered events of one entity.
• Event store: a database for appending and reading streams.
• Rehydrate: rebuild state from events.
• Snapshot: a stored state at a version.
• Projection: a builder of read models.
• Upcaster: a converter from old event versions.
• Expected version: the stream version a writer assumes when appending.
• Crypto-shredding: making data unreadable by deleting its key.

Summary

Event sourcing stores every change as an immutable event and derives current state by replaying them. Named as a pattern by Martin Fowler in 2005 and popularised by Greg Young, it is often combined with CQRS and domain-driven design. It gives a complete audit trail, time travel and the ability to build new views from history, but brings schema evolution work, eventual consistency, projection maintenance and privacy challenges. Use it where history truly matters, design meaningful events, plan versioning and deletion from the start, and keep projections rebuildable.

Related

It belongs to Architecture in my stack, under MLOps and AI infrastructure. Others in the same group: