Bronze layer

Overview

The bronze layer is the first layer of a medallion architecture. It stores data exactly as it arrived from the source, with a few extra columns that describe the ingestion, such as when it was loaded and from which file. Nothing is cleaned, fixed or reshaped in a meaningful way. The layer is a faithful, replayable record of everything received.

This simple rule has big benefits. If a later rule changes or a bug is found, you can rebuild the cleaned layers from bronze without asking the source system to send data again. Bronze also supports audits, since it shows what the source really sent. Teams that train models on tables value it for the same reason: they can always return to the raw shape and rebuild a dataset under new rules.

Bronze tables are usually append-only, partitioned by ingestion date, and stored in an open table format such as Apache Iceberg or Delta Lake on object storage. They are written by ingestion jobs, streams and change data capture pipelines, and read mostly by the jobs that build the silver layer.

What it is

Bronze is the landing and history zone of the lakehouse. It answers one question: what exactly did we receive, and when?

Analogy: a receiving dock at a warehouse. Every delivery is logged with a time, a sender and a delivery slip, and the boxes are shelved unopened. Nobody repacks boxes at the dock. Repacking happens later in other rooms.

What bronze contains

• Source records in their original values and, as far as practical, original structure.
• Technical metadata: ingestion timestamp, source name, file name or stream offset, batch id.
• Sometimes a copy of the raw payload as a string, next to parsed columns.
• Records that later prove to be bad. They stay.

What bronze does not do

• It does not remove duplicates.
• It does not enforce business rules.
• It does not join sources.
• It does not guarantee clean types.
• It is not meant for direct use by analysts, although engineers and data scientists may peek.

Bronze versus a landing zone

Some platforms add a landing area of raw files before bronze. Files sit there in their original format, such as CSV or JSON. Bronze then holds them as tables with ingestion metadata. Other platforms treat the raw files themselves as bronze. Either choice works. The important point is that raw history is kept and is queryable.

Common bronze sources

Where it came from

The need for a raw zone is old. Data warehouse teams kept staging tables where source extracts were loaded before being transformed. When data lakes arrived, teams stored raw files in folders. The lakes were cheap and flexible, but many turned into swamps with no rules and no history.

The lakehouse made raw zones safer. With table formats that offer transactions and schema tracking, raw data could be stored as real tables and still be cheap. The medallion naming, with bronze for raw, became popular as a simple way to describe where raw data lives and what rules apply. Alternative names include raw, landing, staging and L0.

Ingestion tools also changed. Change data capture made it possible to capture every insert, update and delete from a database and store them as events. Streaming platforms made it natural to land events continuously. Both fit the bronze idea of keeping everything as it came.

How it works

Step 1: Pick the grain

Decide what one row in bronze means. For files, one row may be one record in a file. For change data capture, one row is one change event. For an API, one row may be one response item. The grain should match what the source delivers, not what you wish it delivered.

Step 2: Capture the data

The ingestion job reads from the source. It may be a batch job that picks up new files, a streaming job that reads from a log, or a connector. The job should remember how far it got, using a checkpoint, an offset or a list of processed files.

Analogy: a bookmark. If the job restarts, it opens the book at the bookmark and does not reread the whole thing or skip pages.

Step 3: Add ingestion metadata

The job adds columns that describe the load.

Step 4: Write append-only

Rows are appended to a bronze table. If a source sends the same record twice, both copies are stored. If a record is updated at the source, the new version arrives as a new row. History grows, and silver decides which version is current.

Step 5: Preserve the raw shape

Where schemas are unstable, a common trick is to store the raw payload as a string column next to a few parsed key columns. Later layers can parse it as rules mature. This protects against parse errors and schema changes.

Step 6: Handle schema changes gracefully

When a new column appears, bronze should accept it, often with automatic schema evolution on. When a type changes in a risky way, bronze may store the value in a wider type or in the raw string column and raise an alert. The aim is not to lose data. See Schema evolution and Schema drift detection.

Step 7: Partition and organize

Partition by ingestion date or hour, because reads from silver are mostly "everything new since last time." Avoid too many partitions, which create small files. See Iceberg compaction.

Step 8: Secure and retain

Raw data may contain sensitive values. Apply tight permissions, encryption and a retention policy. Decide how long raw history is kept and how deletions for privacy requests are handled.

Step 9: Hand off to silver

Silver jobs read new bronze data, using snapshots, change feeds or checkpoints, and produce clean tables. If silver logic changes, rerun it over bronze history.

Key concepts and terms

A worked example

This example ingests JSON files and a change stream into bronze tables. Hosts and paths are placeholders.

Batch files with Auto-style incremental pickup

from pyspark.sql import functions as F

# Read only files not processed yet using a streaming file source
stream = (
    spark.readStream
    .format("json")
    .schema(raw_schema)                  # may include a _rescued_data column
    .option("cleanSource", "off")
    .load("s3://example-landing/customers/")
)

bronze = (
    stream
    .withColumn("_ingested_at", F.current_timestamp())
    .withColumn("_source_file", F.input_file_name())
    .withColumn("_source_system", F.lit("crm"))
)

(
    bronze.writeStream
    .format("iceberg")
    .option("checkpointLocation", "s3://example-checkpoints/bronze/customers/")
    .trigger(availableNow=True)
    .toTable("lake.bronze.customers_raw")
)

The checkpoint remembers which files were handled, so a rerun does not reload them.

Keeping the raw payload

raw_lines = spark.read.text("s3://example-landing/events/2026-01-05/")

bronze_events = (
    raw_lines
    .withColumnRenamed("value", "_raw_payload")
    .withColumn("_ingested_at", F.current_timestamp())
    .withColumn("_source_file", F.input_file_name())
    .withColumn("event_id", F.get_json_object("_raw_payload", "$.event_id"))
    .withColumn("event_type", F.get_json_object("_raw_payload", "$.type"))
)

bronze_events.writeTo("lake.bronze.events_raw").append()

The full text is kept. Two key fields are parsed for convenience. If the format changes next month, nothing is lost.

Change data capture events

CREATE TABLE IF NOT EXISTS lake.bronze.orders_cdc (
  order_id     BIGINT,
  customer_id  BIGINT,
  amount       STRING,
  status       STRING,
  _op          STRING,      -- 'I', 'U' or 'D'
  _source_ts   TIMESTAMP,
  _offset      BIGINT,
  _ingested_at TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(_ingested_at));

Note that amount is stored as a string. If the source sends "12,50" or "N/A", bronze does not fail. Silver will parse it.

Numeric walk-through

A source sends 10,000 customer rows on Monday and 10,200 on Tuesday. Of the Tuesday rows, 9,900 are repeats of Monday rows, 200 are new, and 100 of the repeats have a changed address.

• Bronze after Monday: 10,000 rows.
• Bronze after Tuesday: 20,200 rows. All repeats stay.
• Silver, after deduplication by customer id: 10,200 distinct customers, with the 100 changed addresses updated.

If silver had a bug on Monday that dropped phone numbers, you fix the code, rerun silver over both days of bronze, and recover them. That is the payoff.

A basic quality view over bronze

SELECT
  _source_system,
  DATE(_ingested_at) AS day,
  COUNT(*)           AS rows_loaded,
  COUNT(DISTINCT _source_file) AS files,
  MAX(_ingested_at)  AS last_load
FROM lake.bronze.customers_raw
GROUP BY _source_system, DATE(_ingested_at)
ORDER BY day DESC;

Teams use this kind of view for freshness and volume checks, so that a missing file or a sudden drop in rows is spotted early.

Strengths and limitations

Strengths

• Full history, so you can audit and replay.
• Loose coupling to source changes, since bronze accepts what arrives.
• Fast to build and to extend with new sources.
• Protects against bugs in cleaning code, since raw data is untouched.
• Supports both batch and streaming ingestion.
• Provides evidence for debates about what the source sent.
• Useful for research on real data behavior.

Limitations

• Storage grows quickly, since duplicates and old versions stay.
• Data is messy, so direct use is error-prone.
• Sensitive raw data raises security and privacy duties.
• Privacy deletions are hard when raw data is kept.
• Small file problems are common with streaming ingestion.
• Without discipline, bronze can become a dumping ground with no owners.
• Parsing late means errors show up later in silver.

How it compares with alternatives

Bronze often sits alongside raw files. The files are the archive, and the tables are the working copy.

See Medallion architecture for the full pattern.

Role in tabular foundation models

Tabular foundation models learn from raw tables, so what bronze stores is directly relevant. Bronze is the closest layer to the "raw table" those models are meant to handle.

Rows, columns and mixed types

Raw sources contain integers stored as text, dates in several formats, free text in numeric columns, and mixed units. A model that is built to cope with mixed types can learn from this variety. Bronze keeps that variety. Researchers can compare model behavior on raw bronze tables and on cleaned silver tables to see how much cleaning helps.

Missing values

In raw data, missing values appear in many forms: empty strings, the word "null", zero, minus one, or a space. Bronze preserves them as they came. This matters for studying imputation, since the way a source encodes "missing" can itself carry information. A model trained only on cleaned data never sees these patterns. See Silver layer for how cleaning handles them.

No column names

Some sources send files with no header or with cryptic names such as c1, c2, f_07. Bronze can store those as they are, together with a record of the source. Models that work from values and structure can try to infer meaning, which supports semantic and sector classification experiments. A later catalog entry then holds the human label for comparison.

Pre-training on real tables

A collection of bronze tables from many sources is a natural pool for pre-training on real tables, subject to licensing, privacy and permission. It can be mixed with synthetic tables, which are generated rather than collected. Keep the two kinds in separate namespaces and tag them by origin, so evaluation stays honest. See /model/research and /model/architecture for how such models are trained, and /model/benchmarks for how they are compared.

Continual learning

Continual learning needs a steady, ordered feed of new data. Append-only bronze tables with ingestion timestamps give exactly that. A training job can take "everything since the last snapshot," update the model, and record the new snapshot. Later you can test for forgetting by evaluating on older slices.

Anomaly detection

Because bronze keeps repeats, odd values and malformed rows, it is a rich source of real anomalies. Quarantine tables in silver often start as bronze rows that failed checks. They make useful test sets for anomaly detectors. See Anomaly detection.

In-context learning

Models such as TabPFN use a small labeled table as context. Bronze is usually too messy for that use. The typical path is to build a clean context table in silver and pass that to the model, while bronze remains the audit trail for where the context came from.

Governance

Because bronze holds raw, possibly sensitive values, it needs the strictest controls. Before data is used for model training, apply tokenization or masking where required. See Data tokenization and Data lineage.

Practical guidance and best practices

Keep it append-only

Do not update or delete rows in bronze except for legal or privacy reasons. Treat it as a ledger.

Add the same metadata everywhere

Use the same technical column names across all bronze tables. This makes generic monitoring, replay and lineage tools simple.

Capture the source timestamp

Where the source gives an event or commit time, store it. Silver needs it for ordering and for resolving late data.

Make ingestion idempotent

If a job reruns after a failure, it must not load the same file twice. Use checkpoints, file manifests or unique batch keys.

Prefer tolerant types

For unstable sources, store fields as strings, or keep a rescued data column for values that did not fit the schema. Strict typing belongs in silver.

Watch for small files

Streaming ingestion creates many small files. Schedule compaction and snapshot expiry. See Iceberg compaction.

Monitor freshness and volume

Alert when a source stops sending, sends far fewer rows than usual, or sends a changed schema. Pair this with data contract validation.

Secure it properly

Restrict who can read bronze. Encrypt data at rest. Where sensitive fields are not needed downstream, tokenize or drop them at ingestion and record that decision.

Plan retention

Decide how long to keep raw history. Some teams keep it forever in cheap storage, others keep a window such as a year. Tie the choice to audit and replay needs.

Make the catalog entry complete

Register each bronze table with its source, owner, refresh pattern and sensitivity label. See Data catalog.

Use a landing area for files you cannot lose

If files are sent to you and not retained at the source, archive the original files too, since tables are derived from them.

Document the grain

Write down what one row means. Many silver bugs begin with confusion about whether a bronze row is a record, an event or a snapshot.

Designing bronze for different source types

Relational database sources

Capture changes rather than full copies where you can. A change event table keeps the operation, the key, the new values and the commit time. If you must take full snapshots, add a snapshot date and keep each one, or at least keep a rolling set. See Change data capture.

Event stream sources

Keep the topic, partition and offset with every row. These three fields uniquely identify an event and make duplicate detection simple. Keep the event key and headers, since they often carry routing or tenant information.

File sources

Keep the file name, size and modification time. Files can be resent with the same name and different content, so a hash of the content is a useful extra column. Keep the original files in an archive path, and never reuse a file name.

API sources

Store the request parameters, the response time and the raw response. APIs often page their results or change their shape, so the raw response helps you debug later.

A bronze table checklist

Reading bronze incrementally

Silver jobs should read only what is new. There are three common ways.

1. Filter by _ingested_at greater than the last processed value, stored in a small control table.
2. Use table snapshots: read the changes between the last processed snapshot and the current one.
3. Use a streaming read with a checkpoint.

-- Snapshot based incremental read in Spark with Iceberg
SELECT *
FROM lake.bronze.orders_cdc
WHERE _ingested_at > TIMESTAMP '2026-01-05 00:00:00';

# Reading only changes between two snapshots
changes = (
    spark.read.format("iceberg")
    .option("start-snapshot-id", last_snapshot)
    .option("end-snapshot-id", current_snapshot)
    .load("lake.bronze.orders_cdc")
)

Store the last processed position after the silver write succeeds, not before. If the job fails in between, it will repeat work, which is safe if silver is idempotent.

Dealing with sensitive data in bronze

Raw data can include names, contact details and identifiers. Some practical options follow.

• Tokenize or hash sensitive fields at ingestion, so bronze never holds the raw value.
• Keep a separate, tightly controlled vault table that maps tokens back to values.
• Store the sensitive fields in a restricted column group or table.
• Apply short retention to the most sensitive sources.
• Log every access.

Each option trades convenience against risk. The right choice depends on the law, the contract and the use case. See Data tokenization.

When a source is wrong

Sometimes the source sends wrong data and later sends a correction. Bronze keeps both. Silver applies the correction by taking the latest version. If the correction never comes, silver rules or a manual override table can fix the value without touching bronze. The raw wrong value remains visible, which is useful for audits.

Common mistakes

• Cleaning or deduplicating in bronze, which removes the raw record.
• Rejecting a whole load because one row fails parsing.
• Using strict types that make ingestion fail every time the source changes.
• Forgetting ingestion metadata, then being unable to trace a bad row.
• Letting analysts build reports directly on bronze.
• Not planning for privacy deletions in raw history.
• Ingesting the same file twice because there is no checkpoint.
• Over-partitioning by a high cardinality column.
• Keeping bronze forever without any storage plan.
• Overwriting bronze tables on each load, which destroys history.
• Skipping documentation, so nobody knows what a column means.

Frequently asked questions

What is the bronze layer?

The bronze layer is the first layer of a medallion architecture. It holds data as it arrived from the source, along with metadata about the ingestion. It is the replayable record that the cleaner layers are built from.

Should bronze data be cleaned?

No. The point of bronze is to keep what was received. Cleaning, typing and deduplication belong in the silver layer. If you clean in bronze, you lose the ability to see and replay the original.

Why store bad or duplicate rows?

They are evidence. They show what the source really sent and let you fix cleaning rules later. If you drop them, you cannot rebuild silver correctly, and you cannot explain past numbers.

What metadata should every bronze table have?

At minimum, an ingestion timestamp, the source system and the source file or offset. A batch id and, for change data, the operation type and source timestamp are also very useful. These columns support tracing, replay and monitoring.

Can analysts query bronze directly?

They can, but it is usually discouraged. Bronze has duplicates, odd types and unfinished records, so results are easy to get wrong. Most users should read silver or gold. Engineers and data scientists may use bronze for debugging or research.

How do I handle sources whose schema keeps changing?

Accept the change in bronze using schema evolution or a raw payload column, and raise an alert. Then decide in silver how to map the new shape. Contracts and drift checks make these changes visible rather than surprising.

How long should I keep bronze data?

It depends on audit needs, replay needs and cost. Many teams keep it long term because storage is cheap, while others keep a fixed window. Make sure your privacy and legal rules are met either way.

How does bronze handle privacy deletion requests?

Plan this in advance. Options include row level deletes in the table format, tokenizing sensitive fields at ingestion so deletes act on a token map, and short retention for the most sensitive sources. Deletes must also reach downstream layers.

Is bronze the same as a data lake?

Not quite. A data lake is the broader storage environment. Bronze is a specific zone within a lakehouse that holds raw data in table form with metadata and rules. A lake may hold many other things too.

Does bronze work for streaming data?

Yes. A streaming job can append events to a bronze table continuously, with checkpoints for recovery. Because streaming creates many small files, plan regular compaction.

Glossary

• Append-only: A write pattern that only adds new rows.
• Batch id: An identifier for a single ingestion run.
• Bronze layer: The raw data layer of a medallion architecture.
• Change data capture: A method that records every change in a database as an event.
• Checkpoint: Saved progress that lets a job resume.
• Ingestion metadata: Columns that describe how and when a row was loaded.
• Landing zone: A staging area for incoming raw files.
• Offset: The position of a record in a log.
• Raw payload: The original record kept as text.
• Replay: Rebuilding downstream data from stored raw data.
• Rescued data: Values that did not fit the expected schema, stored separately.
• Retention: The period for which data is kept.
• Schema evolution: Letting a table's schema change without rebuilding it.

Summary

The bronze layer is the honest record of what the platform received. It is append-only, carries ingestion metadata, tolerates messy and changing inputs, and gives every later layer something solid to rebuild from. Its rules are simple: keep it raw, tag it well, secure it, and do not clean it.

For work with tabular foundation models, bronze preserves the real texture of tables: mixed types, odd missing-value codes, cryptic column names and natural drift over time. That makes it valuable for research, continual learning and anomaly testing, as long as privacy and governance are handled with care. Clean, typed tables for production use come from the next layer.

Related

It belongs to Table formats and lakehouse in my stack, under Data engineering and lakehouse. Others in the same group: