Data mesh

Overview

Data mesh is an approach to organising analytical data in large organisations. Instead of one central data team collecting everything into a single lake or warehouse and serving everyone, responsibility moves to the business domains that create the data, such as orders, payments, logistics or marketing. Each domain team publishes its data as a product that others can find, trust and use. A shared self-serve platform makes publishing easy, and a federated governance model sets the common rules that let products work together.

Data mesh is mainly about people and ownership, not a specific technology. It responds to a familiar bottleneck. As an organisation grows, a central data team ends up between hundreds of source systems it does not understand and hundreds of consumers it cannot keep up with. Requests queue for weeks, pipelines break when source teams change something, and nobody feels responsible for data quality. Data mesh argues that the people who understand the data best should own it from source to published product.

An analogy is the difference between a single central kitchen cooking for a whole city and a food market. In the market, each stall cooks its own speciality and is responsible for its quality and labelling. The market provides shared services, such as power, water, payments and hygiene inspections, and sets rules every stall must follow. Customers can find and combine dishes from many stalls because the rules make them comparable.

What it is

Data mesh rests on four principles described by its originator.

Around these principles sit a few building blocks.

Where it came from

The term was introduced by Zhamak Dehghani, then a technology consultant at Thoughtworks, in a 2019 article that described how to move beyond a monolithic data lake to a distributed data mesh. A follow-up article in December 2020 set out the four principles and a logical architecture. In 2022 O'Reilly published her book "Data Mesh: Delivering Data-Driven Value at Scale".

Data mesh borrowed ideas from software engineering, especially domain-driven design, product thinking and the shift from monolithic applications to microservices owned by independent teams. Since then many organisations have adopted parts of it, often combining domain ownership with a central platform such as a lakehouse, and the related ideas of data products and data contracts have spread well beyond organisations that call their approach a mesh.

How it works

Identifying domains and products

1. Map the business domains, usually following existing team boundaries.
2. For each domain, identify the data others need. Source-aligned products expose a domain's facts, such as orders placed. Aggregate products combine several sources. Consumer-aligned products serve a specific use, such as a marketing dashboard.
3. Assign an owner and a team to each product.

Building a data product

A data product is more than a table. It typically includes the transformation code, the output data in one or more formats, a schema and data contract, documentation, quality checks with published results, access policies and service levels, such as "updated by 06:00 daily, 99 per cent of days". The product is registered in the catalogue so others can find it.

The platform

The platform team builds paved roads: templates to create a new product, storage and compute, pipeline orchestration, testing and data contract validation, lineage, monitoring, access management and catalogue registration. The goal is that a domain team with general engineering skills can publish a well-behaved product in days, not months.

Governance

A federated group agrees on global rules: identifiers for shared entities such as customers, naming conventions, classification of personal data, retention, interoperability formats and minimum quality standards. "Computational" means the rules are enforced by the platform where possible, through automated checks, policy-as-code and default settings, rather than by manual review.

Key concepts and terms

A worked example

This scenario is illustrative. A retailer with 25 engineering teams has a central data team of eight people who maintain about 400 pipelines. New data requests wait six to eight weeks, and a quarter of incidents start with an upstream schema change nobody announced.

1. The company chooses three pilot domains: orders, inventory and customer service. Each nominates a data product owner.
2. The platform team, formed from four members of the old central team, provides a template that creates a repository, a pipeline, a lakehouse table, quality checks and a catalogue entry.
3. The orders team publishes "orders placed", with a contract defining columns, a shared customer_id format, freshness by 05:00 and checks for duplicates. Breaking changes now require a new version and notice to consumers.
4. The governance group agrees on four global rules: shared customer and product identifiers, tagging of personal data, a minimum set of documentation fields and retention limits. The platform checks them automatically on every deployment.
5. Marketing builds a consumer-aligned "customer value" product from orders and customer service products, without asking the central team.
6. After nine months, the pilot domains own their pipelines, request waiting times for their data have fallen, and the company starts onboarding further domains in waves.

Strengths and limitations

Strengths:

• Removes the central team bottleneck by spreading ownership.
• Places accountability for data quality with the people who know the data.
• Treating data as a product improves documentation, reliability and discoverability.
• A shared platform avoids each team reinventing infrastructure.
• Scales with the organisation as new domains join.

Limitations:

• Requires significant organisational change, skills and sustained leadership support.
• Domain teams need data engineering capacity, which competes with feature work.
• Risk of duplication and inconsistent definitions if governance is weak.
• Building a good self-serve platform is a large, ongoing investment.
• Poorly suited to small organisations where a central team works fine.

How it compares with alternatives

Practical guidance and best practices

1. Start with a clear problem, such as slow delivery or poor quality, not with the label "data mesh".
2. Pilot with two or three motivated domains whose data is widely used.
3. Build the platform alongside the pilots, driven by their real needs.
4. Define what a minimum viable data product must include: owner, contract, documentation, quality checks and service levels.
5. Agree a small set of global standards first, especially shared identifiers, and automate them.
6. Use data contracts and versioning so producers can change without breaking consumers.
7. Make products discoverable through a catalogue with data lineage.
8. Measure outcomes, such as lead time for new data, incidents and product usage, and adjust.

Common mistakes

• Treating data mesh as a tool to buy rather than an organisational change.
• Decentralising ownership without giving domains people, skills or a usable platform.
• Skipping governance, so every domain defines "customer" differently.
• Publishing raw database tables and calling them products.
• Rolling out to every domain at once instead of learning from pilots.
• Disbanding central expertise entirely instead of turning it into the platform and enablement team.

Frequently asked questions

Is data mesh a technology?

No. It is an organisational and architectural approach. It uses technology, such as lakehouses, warehouses, catalogues and orchestration tools, but the core change is who owns data and how it is shared.

What is a data product?

A data set published by a domain team with an owner, documentation, a schema or contract, quality checks, access policies and service levels. It is designed for others to find and use without needing to ask the producing team for help.

Does data mesh mean every team builds its own infrastructure?

No. A central platform team provides shared, self-serve infrastructure. Domains build products on it, which keeps costs and standards under control.

When is data mesh a bad fit?

When the organisation is small, has few domains or already gets good results from a central team. The overhead of decentralised ownership and federated governance pays off mainly at scale.

How does data mesh relate to a lakehouse?

They address different questions. A lakehouse is a technical architecture for storing and processing data. A data mesh is about ownership and organisation. Many meshes are built on a shared lakehouse platform, with each domain owning its own tables.

Who created data mesh?

Zhamak Dehghani introduced the concept in 2019 while at Thoughtworks and later described it in detail in a 2022 book.

Glossary

• Bounded context: a domain-driven design term for an area with its own consistent model and language.
• Data catalogue: a searchable inventory of data sets with owners and metadata.
• Data contract: an agreement on schema, meaning and quality for shared data.
• Data fabric: an integration approach based on central metadata and automation.
• Discoverability: how easily consumers can find a data product.
• Paved road: a supported, standard way of doing something on a platform.
• Service level objective: a target for a product's freshness, availability or quality.
• Versioning: publishing changes as new versions so consumers can migrate safely.

Summary

Data mesh is an approach in which domain teams own their analytical data and publish it as products, supported by a self-serve platform and held together by federated, automated governance. It tackles the bottlenecks and quality problems of a single central data team, but it demands real organisational change, investment in a platform and discipline around shared standards. Most successful adoptions start small, use a shared technical foundation such as a lakehouse, and rely on data contracts, a data catalogue and data lineage to keep products trustworthy and connected.

Related

It belongs to Architecture in my stack, under Big data. Others in the same group: