Data contract validation

Overview

Data contract validation is the automatic checking of data against an agreement between the team that produces it and the teams that use it. The contract states what the data should look like: its columns and types, which values are allowed, how fresh it must be, how complete, and who owns it. Validation runs those rules on real data and accepts, flags or rejects what does not comply.

The goal is to stop bad data at the boundary between teams, before it spreads into reports, features and models. Instead of every consumer discovering problems on its own, the producer is accountable for meeting a written promise, and consumers can rely on it. For systems that learn from raw tables, such as tabular foundation models, contracts define exactly what a table must look like before the model is allowed to read it.

What it is

A data contract is a formal, machine readable description of a dataset and the promises made about it. Validation is the enforcement step: code that reads the contract and tests data against it.

What a contract usually contains

Contract versus schema

A schema describes shape. A contract adds quality, timing, ownership and change rules. You can think of a schema as the floor plan and the contract as the lease: it also says who maintains the building, when repairs are done and what happens if the roof leaks.

Producers and consumers

The producer is whoever creates and publishes the data: an application team, a vendor feed, an upstream pipeline. The consumers are the analysts, pipelines and models that depend on it. A contract makes the dependency explicit. Without one, the producer may change a column for good reasons and never know who it breaks.

An everyday analogy

Consider a bakery that supplies bread to a cafe. They agree that loaves are delivered by seven in the morning, are sliced, weigh about eight hundred grams and contain no nuts. If a delivery is late, unsliced or has nuts, the cafe rejects it and tells the bakery. Data contract validation is the cafe's receiving check, done the same way every day.

Where it came from

The idea has roots in software engineering. API contracts and consumer driven contract testing let services depend on each other safely. Interface definition languages and typed schemas for messages did similar work for streams.

In data engineering, the need grew as organisations moved to many teams, many sources and shared platforms. Typical pain points were pipelines breaking after upstream changes, dashboards showing wrong numbers, and endless debate over who was responsible. Around the 2020s, practitioners began to write about data contracts as a way to shift responsibility "left" toward the producer, in the same spirit as shifting testing left in software.

Related practice grew in parallel: data mesh ideas about data as a product with an owner, data quality frameworks that run expectations on tables, schema registries for event streams, and data catalogs that store metadata. See Schema registry, Data catalog, Schema evolution and Data lineage. There is no single universal standard for the contract format, but common ones are YAML or JSON documents with fields for schema, quality and service levels.

How it works

Step 1: Write the contract

Producer and consumers agree on the content and record it in a file kept in version control, next to the code that produces the data. It is reviewed like any code change.

Step 2: Choose where to validate

Validation can run at several points.

The earlier the check, the cheaper the fix. Many teams check at both the producer and the ingestion point.

Step 3: Read the data and the contract

The validator loads the contract and reads the data, or a sample and its metadata. For structural rules it needs only metadata. For value rules it scans columns or computes aggregates.

Step 4: Evaluate the rules

Each rule returns pass or fail with details. Examples are "column order_id exists and is a string", "order_id is unique", "amount is between zero and one million", "no more than one percent of email is null", "latest created_at is less than two hours old".

Step 5: Decide the outcome

Rules carry a severity. A critical failure blocks publication or loading. A warning allows data through but notifies. A policy defines what happens, for example quarantine the batch or fail the pipeline.

Step 6: Report and alert

Results are stored with the contract version and batch identifier. Alerts go to the producer's owner, not to the consumer. A dashboard shows pass rates over time, which turns quality into a measurable service level.

Step 7: Change the contract safely

When the producer wants to change the data, it proposes a new contract version. Consumers review it. Compatible changes (adding an optional column) can go ahead. Breaking changes (removing a field) need a notice period and a migration plan. See Schema evolution.

A sample contract

contract:
  name: customer_orders
  version: 2.1.0
  owner: orders-team@example.internal
  description: One row per order line, published hourly.

schema:
  - name: order_id
    type: string
    required: true
    unique: true
  - name: customer_id
    type: string
    required: true
  - name: amount
    type: decimal(12,2)
    required: true
    min: 0
    max: 1000000
  - name: currency
    type: string
    required: true
    allowed: [USD, EUR, GBP, TRY]
  - name: created_at
    type: timestamp
    required: true
  - name: coupon_code
    type: string
    required: false

quality:
  max_null_fraction:
    coupon_code: 0.95
  row_count:
    min: 1000
    max: 5000000

freshness:
  column: created_at
  max_age_minutes: 120

change_policy:
  breaking_change_notice_days: 30

Key concepts and terms

A worked example

We will validate a small orders table against the contract above using Python and pandas. In real systems you might use a data quality framework or warehouse tests, but the logic is the same.

Load the contract and data

import pandas as pd
import yaml
from datetime import datetime, timezone, timedelta

contract = yaml.safe_load(open("contracts/customer_orders.yaml"))
df = pd.read_parquet("incoming/orders.parquet")
failures = []

Schema rules

TYPE_MAP = {
    "string": "object",
    "timestamp": "datetime64[ns]",
}

for col in contract["schema"]:
    name = col["name"]
    if name not in df.columns:
        failures.append(("critical", name, "column missing"))
        continue
    expected = TYPE_MAP.get(col["type"])
    if expected and str(df[name].dtype) != expected:
        failures.append(("critical", name,
                         f"type {df[name].dtype}, expected {expected}"))
    if col.get("required") and df[name].isna().any():
        failures.append(("critical", name, "nulls in required column"))
    if col.get("unique") and df[name].duplicated().any():
        failures.append(("critical", name, "duplicate values"))

Value rules

for col in contract["schema"]:
    name = col["name"]
    if name not in df.columns:
        continue
    if "min" in col and (df[name] < col["min"]).any():
        failures.append(("high", name, f"values below {col['min']}"))
    if "max" in col and (df[name] > col["max"]).any():
        failures.append(("high", name, f"values above {col['max']}"))
    if "allowed" in col:
        bad = ~df[name].isin(col["allowed"])
        if bad.any():
            failures.append(("high", name,
                             f"{int(bad.sum())} values outside allowed list"))

Quality and freshness rules

rows = contract["quality"]["row_count"]
if not (rows["min"] <= len(df) <= rows["max"]):
    failures.append(("high", "table", f"row count {len(df)} out of range"))

for name, limit in contract["quality"]["max_null_fraction"].items():
    frac = df[name].isna().mean()
    if frac > limit:
        failures.append(("medium", name, f"null fraction {frac:.2f} > {limit}"))

latest = df["created_at"].max()
age = datetime.now(timezone.utc) - latest.tz_localize("UTC")
if age > timedelta(minutes=contract["freshness"]["max_age_minutes"]):
    failures.append(("high", "created_at", f"stale by {age}"))

Decide

critical = [f for f in failures if f[0] == "critical"]
if critical:
    print("REJECT batch")
    for f in failures:
        print(f)
    raise SystemExit(1)
print("ACCEPT", len(failures), "warnings")

What a run might show

Suppose currency contains the value usd in lowercase for 40 rows, and amount has one negative value. The validator reports two high severity failures, with counts. Because none are critical, the batch is accepted with warnings, and the producer is notified. If order_id had duplicates, the batch would be rejected.

Validating in SQL

Some checks are best done in the warehouse, close to the data.

-- uniqueness of order_id
SELECT order_id, COUNT(*) AS n
FROM raw.customer_orders
GROUP BY order_id
HAVING COUNT(*) > 1;

-- values outside the allowed list
SELECT currency, COUNT(*) AS n
FROM raw.customer_orders
WHERE currency NOT IN ('USD', 'EUR', 'GBP', 'TRY')
GROUP BY currency;

-- freshness
SELECT MAX(created_at) AS latest
FROM raw.customer_orders;

Tools such as dbt express similar checks as tests attached to models, and platforms like Snowflake run them at scale. Pipeline tools such as Apache Airflow can run the checks as tasks and stop downstream tasks on failure.

Running it in CI

Contract files are code. Pull requests that change a contract can run a compatibility check against the previous version, and pull requests that change a producer can run validation on sample output. See GitLab CI/CD and GitHub Actions.

Strengths and limitations

Strengths

• Makes expectations explicit and shared.
• Moves responsibility for quality to the producer.
• Catches problems at the boundary where they are cheap to fix.
• Gives consumers something they can build on and cite.
• Provides a basis for change management with notice and versions.
• Produces measurable quality and freshness history.
• Reduces blame and guesswork when things break.

Limitations

• Needs agreement and effort from producers; it is as much a people process as a technical one.
• Contracts can go stale if nobody maintains them.
• Too many strict rules produce noise and slow delivery.
• Too few rules give false comfort.
• Cannot define every semantic rule; meaning still needs documentation and review.
• Validation of large tables can be expensive if it scans every row.
• Formats are not standardised, so tooling differs between organisations.
• Does not replace monitoring for gradual statistical drift of values.

How it compares with alternatives

A practical stack uses a contract as the source of truth, a registry for streams, quality tests as the engine, a catalog for discovery and observability for the unknown unknowns.

Role in tabular foundation models

A tabular foundation model learns from raw tables. It reads rows and columns, deals with numeric, categorical and text values, tolerates missing entries, and may need to work when column names carry little meaning. It is pre-trained on many tables, often including synthetic ones with known structure, and at use time it applies in-context learning: you supply example rows and it predicts for new rows without retraining. A contract defines the input that such a model can safely accept.

A contract as the model's input specification

The contract lists each column's type, allowed values and missing value policy. That lets the serving layer encode every column correctly: numeric as numbers, categories as categories, free text as text. If the contract says age is an integer between 0 and 120 and a batch arrives with age in text, validation stops it before the model sees the wrong encoding.

Handling missing values on purpose

Models for tabular data often accept missing values natively, and imputation models fill them. A contract can state which columns may be missing and how often. This turns a vague worry into a testable rule: a spike in missingness is a contract violation and a reason to investigate before imputing. See Imputation networks.

Column names and semantics

When names are absent or unhelpful, a contract's descriptions and units are the only reliable meaning. They can be passed to systems that classify a table by domain or sector, and they help a person verify that a classification is sensible. When a model does use headers as hints, the contract protects those headers from silent renames. See Schema-drift detection.

Context tables for in-context learning

The example rows given to the model must match the structure of the rows to be predicted. A contract applied to both ensures the same columns, types and ranges. It can also require that the context sample is large enough and covers the target classes, which are quality rules a model benefits from.

Anomaly detection with a clean baseline

Anomaly detection assumes you know what normal looks like. Contract validation separates two problems: rows that break explicit rules (invalid) and rows that are valid but unusual (anomalous). Handling the first with rules and the second with a model keeps alerts meaningful. See Anomaly detection.

Pre-training data and evaluation sets

When collecting tables for pre-training or evaluation, a contract-like check removes malformed files, mixed types, duplicate keys and leaked target columns. Synthetic tables come with an exact known schema, which makes them useful to test validators themselves. Results of model evaluation are summarised at /model/benchmarks.

Continual learning and versioning

As a model is updated, the data it trains on changes. Recording the contract version for each training run supports reproducibility and helps explain behaviour changes. Track these links in a tool such as MLflow.

Feeding models from streams and lakes

Events from streaming systems and files in open table formats often feed model inputs. Contracts line up with registries, Kafka Connect pipelines and Delta Lake tables, so that a model reads only validated data. See also Iceberg cataloging.

For the model side see /model/architecture and /model/research.

Honest boundaries

A contract protects inputs. It does not prove a model is accurate, fair or well calibrated, and it cannot replace evaluation on held out tables.

Practical guidance and best practices

Start small and valuable

• Pick the three to five tables that matter most.
• Write contracts for them first.
• Start with schema and freshness, then add value rules.

Co-design with consumers

• Ask consumers which columns and rules they depend on.
• Do not invent rules nobody needs.
• Review the contract together and record the decision.

Keep contracts in version control

• Store them next to producer code.
• Review changes through pull requests.
• Tag versions and keep a change log.

Validate early and in several places

• Validate at the producer before publishing.
• Validate again at ingestion.
• Validate before models or reports read the data.

Use severity levels

• Block on rules that protect correctness.
• Warn on rules that signal risk.
• Review warnings regularly and promote or remove them.

Make failures actionable

• State the table, rule, count of bad rows and an example.
• Route to the producer's owner.
• Keep rejected data in quarantine for replay.

Manage change

• Define compatible and breaking changes in the contract.
• Require a notice period for breaking changes.
• Run both versions in parallel during migration when possible.

Keep validation affordable

• Prefer metadata checks and aggregates over full scans.
• Sample for expensive rules and scan fully for critical ones.
• Run heavy checks on a schedule rather than every batch.

Measure and share

• Publish pass rates and freshness per contract.
• Review them with producers.
• Use trends to decide where to invest.

Protect privacy

• Mark sensitive columns in the contract.
• Do not copy sensitive values into validation reports.
• Align retention with the contract terms.

Rollout plan for a first contract

Warn-only mode is the key step. It lets you learn how the data really behaves before you start rejecting batches.

Contract tests for the producer

Producers can test their own output in their build. This catches a breaking change before it is released.

def test_output_matches_contract():
    sample = build_sample_output()
    contract = load_contract("contracts/customer_orders.yaml")
    errors = validate(sample, contract)
    assert errors == [], errors

If a developer renames a field, this test fails in their pull request, long before any consumer is affected.

Common mistakes

• Writing contracts nobody enforces. A document without validation is a wish.
• Letting consumers write rules alone. Producers must own and accept them.
• Too strict on day one. Teams switch the checks off.
• Validating only at the end. Problems are found late and far from the cause.
• Ignoring freshness. Correct but stale data still breaks decisions.
• No versioning. Nobody knows which rules applied to which batch.
• Silent auto fixes. Cleaning data without a record hides the producer's problem.
• Alerting the wrong team. The consumer cannot fix the source.
• Forgetting nested and optional fields. Gaps appear in semi structured data.
• Treating the contract as static. Business changes need an update path.
• Putting secrets or real personal values in examples. Use placeholders.
• Skipping tests of the validator. A broken validator passes everything.

Frequently asked questions

What is a data contract?

A data contract is a written, machine readable agreement between the producer of a dataset and its consumers. It describes the schema, meaning of fields, quality rules, freshness expectations, ownership and how changes are handled, so that everyone shares the same expectations.

What is data contract validation?

It is the automated process of testing real data against a data contract. The validator checks columns, types, allowed values, uniqueness, null limits, volume and freshness, then accepts, flags or rejects the data and reports which rules failed and why.

How is a data contract different from a schema?

A schema describes the shape of the data. A contract includes the schema and also quality rules, timing promises, ownership and change policy. In other words, a schema says what the table looks like, and a contract says what the producer promises about it.

Who owns a data contract?

The producer owns it, because the producer controls the data and must keep its promises. Consumers take part in defining it and review changes. A named owner and contact in the contract make accountability clear when something fails.

Where should validation run?

Ideally as close to the producer as possible, before data is published, and again at ingestion into shared storage and before important consumers read it. Early checks are cheaper to fix, and later checks protect against damage in transformations.

What happens when validation fails?

The outcome depends on severity. Critical failures usually block the batch or stop the pipeline and send the data to quarantine. Lower severity failures may let the data through with a warning. In every case the producer's owner is told, with enough detail to fix the cause.

How do data contracts relate to schema registries?

A schema registry stores and versions schemas, often for event streams, and checks compatibility when producers publish. A data contract is broader, covering quality, freshness and ownership, and it can use a registry as the place where the schema part is enforced.

Do data contracts help machine learning models?

Yes. They define and enforce the structure and quality of the inputs a model receives. This prevents silent breakage from changed types, missing columns or sudden spikes in missing values, and they provide versioned records that make training and evaluation reproducible.

How strict should the rules be?

Match strictness to the cost of an error. Rules that protect correctness, such as key uniqueness or required fields, should block. Rules that signal risk, such as shifts in null rates, can warn. Start with a few important rules and add more when real incidents show the need.

Can contract validation be automated in CI?

Yes. Contract files can be checked for syntax and compatibility on every pull request, and producer code can be tested against the contract using sample data. Scheduled jobs can also validate live data and raise alerts when a rule fails.

Glossary

• Breaking change: A change that can make consumers fail.
• Compatible change: A change that existing consumers tolerate.
• Consumer: A user or system that reads data.
• Contract: An agreed description of data and promises about it.
• Data product: A dataset managed with ownership and service levels.
• Expectation: A testable statement about data.
• Freshness: How recent the newest data is.
• Producer: The team or system that publishes data.
• Quarantine: A holding area for rejected data.
• Referential integrity: Matching keys between tables.
• Severity: The seriousness of a rule failure.
• SLA: A promise about service level, such as freshness.
• SLO: An internal target that supports an SLA.
• Validation: Checking data against rules.
• Versioning: Numbering changes to a contract.

Summary

Data contract validation checks data against an agreed description of schema, quality, freshness and ownership, and it does so automatically at the boundary between teams. It shifts responsibility to producers, makes expectations explicit, and turns silent breakage into early, owned alerts. Done well, it uses version controlled contracts, several validation points, sensible severity levels, quarantine and a managed process for change. For tabular foundation models, a contract is the input specification: it protects column types, missing value behaviour, context tables for in-context learning and the clean baselines that anomaly detection and imputation need. It guards inputs but does not replace model evaluation.

Related

It belongs to DataOps in my stack, under Data engineering and lakehouse. Others in the same group: