Medallion architecture

Overview

Medallion architecture is a way to organize a lakehouse into three layers of increasing quality. The bronze layer keeps raw data as it arrived. The silver layer holds data that has been validated, deduplicated, typed and joined into consistent entities. The gold layer holds business-ready tables shaped for reports, dashboards, features and applications.

The idea is to improve data a little at a time, in clear stages, rather than in one giant job. Each stage has a simple promise. Bronze promises a faithful record. Silver promises trustworthy detail. Gold promises answers that are easy to use. This gives teams a shared language, makes problems easier to trace, and lets one clean dataset serve many uses, including machine learning on tables.

The pattern is often called the multi-hop architecture. It is common on lakehouse platforms built with table formats such as Delta Lake and Apache Iceberg, and with engines such as Apache Spark, Trino and Flink. It is a design pattern, not a product, so you can adapt it to your tools and needs.

What it is

The medallion pattern names three zones in a data platform and the rules for moving data between them. The names come from the medal ranking: bronze, silver, gold. Quality and business value rise at each step.

Analogy: making olive oil. Bronze is the harvest delivered in crates, with leaves and twigs included. Silver is the washed, sorted and pressed oil. Gold is the bottled, labeled product ready for the shelf. You keep the crate records so that if the bottle is wrong, you can look back and see why.

The three layers at a glance

What the pattern is not

• It is not a rule that you must have exactly three tables or three folders. Each layer holds many tables.
• It is not tied to one vendor or one format.
• It is not a quality guarantee on its own. You still need tests, contracts and monitoring.
• It does not forbid extra zones, such as a landing area before bronze or a sandbox beside gold.

Where each layer lives

Most teams give each layer its own namespace or database, such as bronze, silver and gold, or a prefix per domain like sales_bronze. Storage is often object storage such as Amazon S3, with tables managed by a format like Apache Iceberg or Delta Lake.

Where it came from

Early data warehouses used staging areas, an integrated core and data marts. Data lakes came next and often became messy, because anyone could drop files anywhere with no rules. People called these swamps.

The lakehouse idea combined lake storage with warehouse style tables and transactions. With reliable tables, it became practical to run many small refinement steps and to rely on the output of each one. The medallion naming was popularized by lakehouse platform vendors and community writing as a simple way to explain those steps. Other names exist for the same idea: raw, refined and curated; landing, staging and mart; or L1, L2 and L3.

The pattern fits well with the shift from ETL to ELT. Load raw data first, then transform it inside the platform with SQL and code. Tools such as dbt became popular for the later steps.

How it works

Step 1: Ingest into bronze

Data arrives from databases, event streams, files and APIs. Jobs copy it into bronze tables with little change. Common ingestion paths include batch loads, change data capture, and streams from Apache Kafka.

Bronze usually adds metadata columns:

• Source system name.
• Ingestion timestamp.
• Source file name or offset.
• Batch or run id.

Analogy: a mailroom. Every parcel is logged when it arrives, with a stamp and a time, and it is stored unopened.

Step 2: Keep bronze append-only and replayable

Bronze is normally append-only. You do not fix bad rows there. If a source sends a wrong value, the wrong value stays, and you correct it later in silver. This gives you a faithful history and lets you rebuild silver from scratch if rules change.

Step 3: Clean and conform into silver

Silver jobs read bronze and apply steps such as:

• Parse and cast types, such as strings to dates and decimals.
• Standardize formats, such as country codes and time zones.
• Remove duplicates, often using a business key and the latest timestamp.
• Handle nulls and invalid values by rules.
• Apply schema validation and quarantine bad rows.
• Join reference data and resolve keys across sources.
• Merge changes into current-state tables, sometimes with history.

The goal is one consistent version of each business entity, such as customer, order or device. See Silver layer.

Step 4: Use incremental processing

Rather than reread all of bronze each time, silver jobs process only new data. Table formats make this easier through snapshots, change feeds or streaming reads. Use a checkpoint or a high water mark to remember progress.

Step 5: Model and aggregate into gold

Gold jobs read silver and build tables for a purpose. Examples are daily sales by region, a star schema for reporting, or a feature table for a model. Gold tables are often aggregated, joined and named in business terms. See Gold layer.

Step 6: Serve

Consumers read gold through SQL engines, BI tools, APIs or machine learning pipelines. Because gold is stable and documented, users do not need to understand raw source quirks.

Step 7: Govern across layers

Permissions usually tighten toward bronze, since raw data may hold sensitive fields, and open toward gold. Lineage tools track which gold table came from which silver and bronze tables. See Data lineage and Data catalog.

Step 8: Reprocess when rules change

If a rule changes, rerun silver from bronze and then gold from silver. This is the main benefit of keeping raw data: you can fix history without asking the source system to resend.

A simple flow

sources -> [ bronze: raw + metadata ] -> [ silver: clean + conformed ] -> [ gold: modeled + aggregated ] -> consumers

Key concepts and terms

A worked example

This example uses PySpark and Iceberg style tables. Names and paths are placeholders.

Bronze: ingest raw orders

from pyspark.sql import functions as F

raw = (
    spark.read.json("s3://example-landing/orders/2026-01-05/")
)

bronze = (
    raw
    .withColumn("_ingested_at", F.current_timestamp())
    .withColumn("_source_file", F.input_file_name())
    .withColumn("_batch_id", F.lit("2026-01-05-run-01"))
)

bronze.writeTo("lake.bronze.orders_raw").append()

Nothing is cleaned. If the source sent "amount": "12,50" as text, bronze keeps it as text.

Silver: clean and conform

from pyspark.sql import functions as F, Window

b = spark.table("lake.bronze.orders_raw")

typed = (
    b
    .withColumn("order_id", F.col("order_id").cast("bigint"))
    .withColumn("amount", F.regexp_replace("amount", ",", ".").cast("decimal(12,2)"))
    .withColumn("order_ts", F.to_timestamp("order_ts"))
    .withColumn("country", F.upper(F.trim("country")))
)

valid = typed.filter("order_id IS NOT NULL AND amount IS NOT NULL AND amount >= 0")
bad   = typed.exceptAll(valid)

w = Window.partitionBy("order_id").orderBy(F.col("_ingested_at").desc())
latest = (
    valid
    .withColumn("rn", F.row_number().over(w))
    .filter("rn = 1")
    .drop("rn")
)

bad.writeTo("lake.silver.orders_quarantine").append()
latest.createOrReplaceTempView("orders_latest")

MERGE INTO lake.silver.orders AS t
USING orders_latest AS s
ON t.order_id = s.order_id
WHEN MATCHED AND s._ingested_at > t._ingested_at THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

The silver table now has typed columns, one row per order, and a quarantine table holds the bad rows for review.

Gold: build a business table

CREATE OR REPLACE TABLE lake.gold.daily_sales AS
SELECT
  DATE(order_ts) AS order_date,
  country,
  COUNT(*)       AS orders,
  SUM(amount)    AS revenue
FROM lake.silver.orders
GROUP BY DATE(order_ts), country;

Dashboards read daily_sales and never need to know about the raw JSON.

Numeric walk-through

Imagine 1,000 raw order rows arrive in bronze.

• 20 rows are exact duplicates sent twice by the source.
• 30 rows have a missing order id or a negative amount.
• 950 rows are valid, but 50 of those are updates to orders already in silver.

Results:

• Bronze holds 1,000 rows, unchanged.
• Quarantine holds 30 rows.
• After dedup, 950 valid rows remain, with 50 updating existing silver rows and 900 new.
• Gold is recomputed and shows totals from the clean data only.

Data quality checks as code

checks:
  silver.orders:
    - not_null: [order_id, order_ts, amount]
    - unique: [order_id]
    - range: {column: amount, min: 0, max: 1000000}
    - freshness: {column: _ingested_at, max_age_hours: 6}

Failed checks should stop the run, alert someone or divert rows, depending on severity.

Strengths and limitations

Strengths

• Clear separation of concerns, so each layer has a simple job.
• Replayability: raw data in bronze lets you rebuild after a bug or rule change.
• Better quality: validation happens at defined points.
• Reuse: one silver table can feed many gold tables.
• Easier debugging: you can compare a gold number to silver and bronze.
• Flexible governance: access rules differ by layer.
• Works with batch and streaming.
• Good fit for machine learning, since silver gives stable training data.

Limitations

• More storage, since data is stored several times.
• More jobs to build and maintain.
• Added latency, since data passes through several hops.
• Risk of ritual: teams create layers without clear rules, and the names become meaningless.
• Gold tables can multiply and drift apart, repeating the problem of many inconsistent reports.
• Not every dataset needs three layers. Small or simple data may need fewer.
• Late data, schema changes and deletions need careful handling across layers.
• Privacy deletions must reach every layer, including bronze.

How it compares with alternatives

Medallion pairs naturally with dbt style modeling, with Apache Spark for heavy transforms, and with table formats that give ACID guarantees.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns, mixed types, missing values and sometimes columns with weak or missing names. The medallion layers decide what kind of table a model sees and how safely it can use it.

Which layer feeds which stage

Mixed types and missing values

A foundation model is meant to cope with numbers, categories, dates and text in one table, and with holes. Silver is the right place to make types honest. A date stored as text becomes a date. A code that means "unknown" becomes a true null, or stays as a category by a documented rule. Done well, the model sees columns whose types match their meaning.

Done badly, cleaning removes the very signal a model could use. If every missing value is filled with a mean in silver, the model cannot learn that missingness itself is informative. A good practice is to keep nulls as nulls in silver and let model code decide how to impute. See Silver layer.

Raw structure and bronze

Pre-training on real tables benefits from variety, including messy tables. Bronze holds the raw shape, which can be valuable for research on robustness. Because bronze is replayable, you can also rebuild a training set under new rules without asking the source again.

Column names and semantics

Models that cope with missing or unhelpful column names learn from values and structure. The layers help here too. In silver, you can add a data dictionary that maps columns to business meaning. These mappings become labels for semantic and sector classification tasks. The model can predict the meaning, and the dictionary checks it.

Continual learning

Continual learning updates a model as data arrives. Bronze to silver incremental jobs produce a steady flow of new, clean rows. Snapshot ids from the table format record exactly which rows each update saw. This makes it possible to test whether the model forgets older patterns.

Imputation and anomaly detection

Imputation is easier to evaluate when you have a clean silver table: you hide known values, ask the model to fill them, and compare. Anomaly detection benefits from the quarantine pattern. Rows that fail rules are kept in a quarantine table, which is a ready-made set of known odd cases for testing detectors. See Anomaly detection.

Leakage and governance

Gold tables often contain aggregates computed over the whole period. Using them as features can leak future information into training. Build training features from silver with point-in-time logic, or from gold tables that are explicitly time-aware. Govern access by layer and track lineage so you know which source tables a model learned from. See Data lineage.

For how such models are built, tested and compared, see /model/research, /model/architecture and /model/benchmarks. Models like TabPFN show how a model can use a small clean table as context, which is exactly what a well kept silver slice offers.

Practical guidance and best practices

Define each layer in writing

Write one short contract for each layer: what it contains, who may write, who may read, and what checks must pass. Keep it where everyone can see it.

Keep bronze raw

Resist the urge to clean in bronze. Add only ingestion metadata. If you must drop sensitive fields at the door, record that choice clearly.

Make jobs idempotent

A rerun should not create duplicates. Use merge by key, or overwrite a partition, or write with a batch id and deduplicate.

Use table formats with transactions

ACID tables let you update silver safely and read consistent snapshots. See Delta Lake and Apache Iceberg.

Test at the boundaries

Apply schema checks when data enters silver, and business rule checks before data enters gold. Use data contract validation and schema drift detection to catch changes early.

Quarantine, do not discard

Send failed rows to a quarantine table with the reason. This supports fixes and gives you a record for audits.

Plan for schema evolution

Sources change. Decide how new columns flow through bronze and silver, and how breaking changes are handled. See Schema evolution.

Control gold sprawl

Give each gold table an owner and a purpose. Retire tables that nobody uses. Prefer shared dimensions over many near-duplicates.

Handle deletions and privacy

When a person asks to be erased, deletes must flow to silver, gold and bronze. Plan how with row level deletes, and consider tokenization for sensitive fields. See Data tokenization.

Maintain physical health

Streaming into bronze creates small files. Compact regularly and expire old snapshots. See Iceberg compaction.

Document lineage and ownership

Register tables in a catalog with descriptions and owners. See Data catalog.

Keep the layers meaningful

If a table in silver is still raw, or a table in gold is just a copy of silver, ask why. Names should match behavior.

Naming and layout conventions

Consistent names save hours of confusion. A simple convention might look like this.

What each layer should contain, in more detail

Bronze details

• One bronze table per source object, not per business entity.
• Original column names and values, even if ugly.
• Store nested data as it came, for example JSON strings or structs.
• Keep a record of the schema at ingestion time.
• Partition by ingestion date, since that is how you will read it.

Silver details

• One table per business entity, such as customer or order.
• Typed columns with documented meaning.
• A stable primary key for each entity.
• Technical columns such as _updated_at and _source_system.
• Either current-state tables or history tables with valid-from and valid-to dates.

Gold details

• Tables built for a known consumer or question.
• Often a star schema, wide feature tables, or pre-aggregated cubes.
• Business names for columns and metrics.
• Documentation of every metric definition.
• Tight tests, since people trust these numbers.

Handling late and out-of-order data

Data does not always arrive in order. A silver job should use the event time for business logic and the ingestion time for processing order. Merge by key and keep the record with the newest event time, not just the last one to arrive. For aggregates in gold, recompute the affected partitions when late data lands, or use watermarks in streaming jobs.

Handling schema changes

When a source adds a column, bronze should accept it, often with schema evolution turned on. Silver should decide whether to take it. When a source changes a type or removes a column, silver should fail loudly or quarantine, rather than guess. Drift checks and contracts turn these events into alerts instead of surprises.

A small team version

Not every team needs a large platform. A small team can start with three namespaces, a handful of tables and a few tests.

1. Land files into bronze with an ingestion timestamp.
2. Write one silver table per main entity with a primary key test.
3. Build two or three gold tables for the most used reports.
4. Add freshness and row count alerts.
5. Document each table in a catalog entry.

Then grow only when real needs appear.

Measuring whether the layers work

Common mistakes

• Cleaning data in bronze and losing the raw record.
• Treating silver as a dumping ground for every transform.
• Building gold tables directly from bronze, skipping silver and its checks.
• Overwriting silver in place without a way to see history or roll back.
• Skipping quarantine, so bad rows vanish without a trace.
• Creating dozens of near-identical gold tables with different numbers.
• Filling nulls in silver in ways that hide missingness from models.
• Forgetting to apply privacy deletions to bronze.
• Using gold aggregates as training features without checking for leakage.
• Making the layers too rigid, so that urgent small datasets wait weeks for pipelines.
• Not monitoring freshness, so stale gold tables look healthy.

Frequently asked questions

What is medallion architecture?

Medallion architecture is a data design pattern that organizes a lakehouse into bronze, silver and gold layers. Each layer improves the data: bronze stores it raw, silver cleans and conforms it, and gold shapes it for business use. The aim is better quality, traceability and reuse.

Do I need exactly three layers?

No. Three is a common starting point, but many teams add a landing zone or split silver into steps. The key is that each layer has a clear purpose and rules. Use fewer layers for simple data and more where the process demands it.

Is the medallion pattern only for Databricks or Delta Lake?

No. The pattern is vendor neutral. It works with Apache Iceberg, Delta Lake, Hudi and other table formats, and with engines such as Spark, Trino and Flink. The idea of staged refinement is what matters.

Why keep raw data in bronze if it is messy?

Raw data is your safety net. If a rule changes or a bug is found, you can rebuild silver and gold from bronze without asking the source to resend. It is also valuable for audits and for research on how real data behaves.

Where should data quality checks run?

Run schema and basic validity checks when moving from bronze to silver, and business rule and reconciliation checks before data reaches gold. Add freshness and volume checks on every layer. Failed rows should go to quarantine.

How is silver different from gold?

Silver holds detailed, cleaned entities that many teams can reuse, usually at the grain of the source events or records. Gold holds data shaped for a specific use, often aggregated and named in business terms. Silver is broad and trusted. Gold is narrow and convenient.

Is medallion architecture the same as ETL?

It is related but not identical. Medallion is a layout for storing data at different quality levels. ETL or ELT describes how data is moved and transformed. A medallion platform usually uses an ELT style, loading raw data first and transforming it inside the platform.

Where should machine learning use data from?

Training data usually comes from silver, since it keeps detail and has clean types, with point in time logic to avoid leakage. Serving features often come from gold tables built for that purpose. Raw bronze can be useful for research or for rebuilding datasets.

Does medallion architecture work with streaming?

Yes. Streams can write into bronze, and streaming jobs can refine bronze into silver and silver into gold. Latency grows with each hop, so choose trigger intervals with care. Compaction is important for streaming tables.

How do I keep gold tables from multiplying out of control?

Give every gold table an owner, a purpose and a review date. Build shared dimensions and metrics in one place, and check for duplicates before adding new tables. Retire tables that no one reads.

Glossary

• ACID: A set of guarantees for reliable transactions: atomicity, consistency, isolation and durability.
• Append-only: A write mode that only adds rows.
• Bronze layer: The raw ingestion layer.
• Conformed data: Data aligned to shared definitions across sources.
• Data contract: An agreement on the shape and quality of data between teams.
• Deduplication: Removing repeated records.
• ELT: Extract, load, then transform inside the platform.
• Gold layer: The business-ready layer.
• Idempotent: Giving the same result when repeated.
• Lakehouse: Lake storage with warehouse style table features.
• Lineage: The trail showing the origin and changes of data.
• Quarantine: A holding table for rows that failed checks.
• Silver layer: The cleaned and conformed layer.
• Snapshot: A fixed version of a table at a point in time.

Summary

Medallion architecture refines data in three understandable stages. Bronze keeps what arrived, silver makes it trustworthy, and gold makes it useful. The pattern brings replayability, clear ownership, better quality and reuse, at the cost of extra storage and more pipelines.

For teams working with tabular foundation models, the layers map cleanly to needs. Silver supplies stable, typed tables for pre-training, fine-tuning and evaluation. Bronze keeps raw variety and enables rebuilds. Gold serves features and reports. Keep nulls honest, track snapshots and lineage, guard against leakage, and keep each layer meaningful.

Related

It belongs to Table formats and lakehouse in my stack, under Data engineering and lakehouse. Others in the same group: