Data lineage

Overview

Data lineage is the record of where data came from, where it went and what happened to it along the way. A lineage view might show that a number in a dashboard comes from a column in a reporting table, which was calculated from two source tables, which were loaded from an operational system. Each hop and each transformation is recorded. The result is a map that people can follow in either direction.

Lineage answers three practical questions. Where did this value come from? What will break if I change this? Can I prove how this number was produced? Teams use it for debugging, impact analysis, audits and trust. For machine learning on tabular data, lineage shows which source tables, joins and filters produced a training set, which helps prevent leakage, explain results and respond to changes. This article covers the idea, how lineage is captured, its limits and its role around tabular foundation models. See also Informatica EDC, Informatica Axon and data catalog.

What it is

Lineage is a graph. Nodes are data assets: tables, columns, files, reports, models. Edges are flows or transformations: a job that reads one table and writes another, a SQL view, a mapping, a model training run.

Think of a family tree for data. A person has parents, grandparents and children. A table has sources, upstream sources of those sources, and downstream tables and reports. You can walk up the tree to find ancestors or down to find descendants.

Upstream and downstream

• Upstream means toward the origin. If you ask "where does this come from?", you look upstream.
• Downstream means toward consumers. If you ask "who uses this?", you look downstream.

Levels of detail

Column-level lineage is the most useful for many teams, but it is harder to capture correctly than table-level lineage.

Related terms

• Provenance usually means the origin and history of a specific piece of data, often in research and scientific settings. Lineage is the engineering term for the same family of ideas across a pipeline.
• Data flow diagram is a hand-drawn picture of flows. Lineage is meant to be generated from real systems and kept current.
• Data mapping is a design document that says how fields should map. Lineage should reflect what actually happens.

Business and technical lineage

Technical lineage shows tables, columns and jobs. Business lineage shows the same flow in business terms, such as "customer billing process feeds monthly revenue report". Business lineage is often built by linking technical assets to glossary terms, see Informatica Axon.

Where it came from

Lineage has long roots. In science, provenance mattered because a result is only trustworthy if you can show what data and methods produced it. In databases, researchers studied how to explain why a query result row exists, tracing results back to input rows. These ideas are known in database research as why-provenance and where-provenance, among others.

In business, lineage became important with data warehouses. Extract, transform and load (ETL) tools moved data through many steps, and teams needed to know which source fed which report. Regulation in finance, insurance and health added pressure. Rules about risk reporting and privacy require firms to show where numbers come from and where personal data flows.

With cloud lakes, streaming, SQL transformation tools and notebooks, data paths became more varied. Manual diagrams could not keep up. Tools began to harvest lineage automatically from SQL, job definitions, orchestrators and query logs. Open specifications also appeared so that tools can exchange lineage events in a common format. One widely known example is OpenLineage, which defines a standard way for jobs to report runs, inputs and outputs.

More recently, machine learning has extended lineage to features, training sets and models. Questions such as "which data trained this model version?" have become routine for responsible AI practice.

How it works

Lineage capture falls into a few broad methods. Most real systems combine them.

Method 1: Parse code

A tool reads SQL, view definitions, mapping files or scripts and works out which inputs feed which outputs. For example, from this statement:

CREATE TABLE mart.customer_spend AS
SELECT c.customer_id,
       c.region,
       SUM(o.amount) AS total_spend
FROM staging.customers c
JOIN staging.orders o ON o.customer_id = c.customer_id
WHERE o.status = 'paid'
GROUP BY c.customer_id, c.region;

A parser can conclude:

• mart.customer_spend.customer_id comes from staging.customers.customer_id,
• mart.customer_spend.region comes from staging.customers.region,
• mart.customer_spend.total_spend comes from staging.orders.amount through a sum,
• rows are filtered by staging.orders.status and joined by customer_id.

Parsing is accurate for static code. It struggles with dynamic SQL built at runtime, because the final statement does not exist until execution.

Method 2: Observe runtime events

Jobs report what they read and wrote while they run. An orchestrator or engine emits an event such as "job X read tables A and B and wrote table C at this time". This captures the real behavior, including dynamic cases, but only if every job is instrumented. See Apache Airflow for an example of an orchestrator that can emit such information, and Apache Spark for an engine whose plans reveal inputs and outputs.

Method 3: Read query logs

Warehouses record executed queries. A tool reads the logs and derives lineage from the statements that actually ran. This catches ad hoc work, but logs can be large, may be retained only briefly and may not include everything.

Method 4: Infer from similarity

When no code or logs are available, a tool can guess lineage by matching names and values. A column in table B that holds the same values as a column in table A may be a copy. This is useful for old or opaque systems but gives probable, not certain, lineage. Treat inferred links as hints and label them as such.

Method 5: Manual entry

For manual spreadsheets, file drops and undocumented steps, people record the flow by hand. This is slow and can go out of date, but it fills gaps that tools cannot reach.

Putting it together: a graph store

Captured edges are stored in a graph or relational structure. A simple model has these parts:

Asset(id, type, name, system)
Edge(source_asset, target_asset, job, transformation, captured_at, method)
Run(job, started_at, finished_at, status)

The method field is valuable. It records whether an edge came from parsing, runtime events, logs, inference or manual entry, so users can judge how much to trust it.

Walking the graph

To find all upstream assets of a report field, a tool performs a traversal: start at the node, follow incoming edges, collect nodes, and repeat. To find downstream impact, follow outgoing edges. In SQL terms this is a recursive query:

WITH RECURSIVE upstream AS (
  SELECT source_asset, target_asset, 1 AS depth
  FROM lineage_edge
  WHERE target_asset = 'report.revenue_by_region.revenue'
  UNION ALL
  SELECT e.source_asset, e.target_asset, u.depth + 1
  FROM lineage_edge e
  JOIN upstream u ON e.target_asset = u.source_asset
)
SELECT DISTINCT source_asset, depth FROM upstream ORDER BY depth;

Large lineage graphs are usually cached or indexed for speed, and they are trimmed with depth limits and filters so that people do not see thousands of nodes at once.

Analogy: a package tracking page

When you order a parcel, the tracking page shows each step: packed, shipped, in transit, delivered. If the parcel is damaged, you check where it was last handled well. Lineage does the same for data. If a number looks wrong, you walk backward step by step to find where it went off.

Time and versions

Data pipelines change. A lineage record is only useful if it says when. Good systems keep history: which version of a job produced which version of a table on which date. This lets you answer questions about last quarter's report, not just today's pipeline. Combining lineage with table versioning, as in lake table formats, helps with reproducibility. See Apache Iceberg and Delta Lake.

Lineage and the layers of a lake

A layered design such as medallion architecture naturally gives clear lineage steps: raw landing data in bronze, cleaned data in silver, and business-ready data in gold. Lineage shows how each layer derives from the previous one and which jobs apply cleaning rules.

Key concepts and terms

A worked example

A finance team notices that the monthly revenue figure in a report dropped by 6 percent overnight, while sales activity looks normal. They use lineage to investigate.

Step 1: Start at the report field

The lineage view for report.revenue_by_region.revenue shows this chain:

crm.orders (source system)
   |  nightly extract job
   v
bronze.orders
   |  cleaning job: remove test orders, convert currency
   v
silver.orders
   |  aggregation job: sum paid orders by region and month
   v
gold.revenue_by_region
   |  report refresh
   v
report.revenue_by_region.revenue

Step 2: Find what changed

The lineage graph keeps job run history. The team sees that the cleaning job changed its version two days ago. They compare outputs.

The numbers are illustrative. The big drop happens at silver, so the cause lies in the cleaning job.

Step 3: Drill into column-level lineage

Column-level lineage for silver.orders.amount_usd shows it derives from bronze.orders.amount and bronze.orders.currency, joined with a rates table. The team finds that the rates table changed its currency code format from EUR to eur, so the join missed lowercase codes and dropped those rows.

Step 4: Fix and prevent

They fix the join to normalize case. Then they check downstream impact: three reports and one forecasting feature table are affected. All are rebuilt. They add a data contract check on the currency code format. See data contract validation.

Step 5: Document

The incident note includes a screenshot of the lineage path, the changed job version and the list of affected downstream assets. Next time, the same lineage helps respond faster.

A small lineage graph in Python

The following toy code shows the traversal idea without any special tool.

edges = [
    ("crm.orders", "bronze.orders"),
    ("bronze.orders", "silver.orders"),
    ("rates.daily", "silver.orders"),
    ("silver.orders", "gold.revenue_by_region"),
    ("gold.revenue_by_region", "report.revenue"),
    ("silver.orders", "features.late_payment"),
]

def upstream(node):
    found, stack = set(), [node]
    while stack:
        cur = stack.pop()
        for src, dst in edges:
            if dst == cur and src not in found:
                found.add(src)
                stack.append(src)
    return found

def downstream(node):
    found, stack = set(), [node]
    while stack:
        cur = stack.pop()
        for src, dst in edges:
            if src == cur and dst not in found:
                found.add(dst)
                stack.append(dst)
    return found

print(sorted(upstream("report.revenue")))
print(sorted(downstream("silver.orders")))

Output:

['bronze.orders', 'crm.orders', 'gold.revenue_by_region', 'rates.daily', 'silver.orders']
['features.late_payment', 'gold.revenue_by_region', 'report.revenue']

The second line is the impact list for a change to silver.orders. Real systems add columns, versions and run history, but the traversal is the same.

Strengths and limitations

Strengths

• Faster debugging. Root causes are found by walking the path instead of reading all code.
• Safer change. Impact analysis shows who is affected before a change ships.
• Audit and compliance. Evidence of origin and handling supports regulatory questions.
• Privacy tracing. You can find every place a sensitive field flows.
• Trust. Consumers see where a number comes from and can judge it.
• Cost savings. Unused tables and duplicate pipelines can be found and retired.
• Better onboarding. New staff understand the landscape faster.
• Support for ML. Training sets and features can be traced to their sources.

Limitations

• Incomplete coverage. Spreadsheets, scripts, manual exports and unmanaged tools may be invisible.
• Dynamic code. Runtime-generated SQL and complex scripts are hard to parse.
• Column-level difficulty. Expressions, user-defined functions and wildcards make exact mapping hard.
• Inferred links can be wrong. Always show the capture method.
• Staleness. If capture is not automatic and continuous, lineage drifts from reality.
• Scale. Large graphs overwhelm users without filters and summaries.
• Meaning is not lineage. A path shows movement, not whether the transformation is correct.
• Cost. Instrumenting jobs and storing history takes effort.
• Row-level lineage is rare. It is expensive and usually limited to special cases.
• Tool boundaries. Lineage across different vendors needs common formats or custom links.

How it compares with alternatives

These are complements. A mature setup combines code-derived lineage, runtime events and a catalog for search, plus manual notes for the few steps that tools cannot see.

Role in tabular foundation models

Lineage is not part of a model. It is the record around the data that a model uses. Tabular foundation models learn from raw tables: rows and columns, mixed types, missing values and often no reliable column names. They may be pre-trained on synthetic or real tables, and applied through in-context learning, where labeled example rows and new rows are given together. Lineage supports this work in several concrete ways.

Knowing where training tables come from

For pre-training and fine-tuning on real tables, teams must know the source, owner, permissions and time of each table. Lineage connects a training table to its origins. If a source later becomes restricted or is found faulty, the affected training sets can be located quickly. The research context is covered on /model/research.

Preventing leakage

Leakage occurs when a feature contains information that would not be known at prediction time, such as a value computed after the outcome. Column-level lineage shows how each feature was derived and from which time period. If a feature depends on a source updated after the label event, it is a leakage candidate. In-context learning is especially sensitive to this, because the example rows are taken directly as evidence.

Explaining mixed types and missing values

A column may be a number because it was cast from text, or null because a join found no match. Lineage reveals those causes. A null that results from a failed join is an error. A null that means "not applicable" is a signal. Knowing which one it is changes how the model should treat the gap. See imputation networks for model-based filling of missing values, and note that lineage tells you whether filling is appropriate.

Working without column names

When a table has no trusted names, lineage can still supply context outside the table: the source column from which a derived column came, or the job that created it. Even when the model itself uses only values, people reviewing results can use lineage to interpret columns. This pairs with value-based inference such as semantic sector heads, where the model infers meaning from data.

Anomaly detection with a cause

A model can flag an unusual row or a sudden change in a column. Lineage helps explain why. If the anomaly starts after a job version change or a new source was added, the cause is probably in the pipeline, not the world. Compare anomaly detection for the modeling side.

Continual learning and drift

When a model is updated or monitored over time, upstream changes must be detected. Lineage gives the list of upstream jobs and sources for each model input. When one changes, owners are alerted. This works with schema drift detection and data contract validation.

Model documentation and accountability

Responsible practice asks for a record of what data a model used. A lineage graph that includes the training table, the model version, the evaluation tables and the serving inputs provides that trace. It supports model cards, reviews and audits. The structure of the model itself is described on /model/architecture.

Fair evaluation

Benchmarks require that test tables are not derived from training tables. Lineage can verify that a test set does not share upstream sources with the training set in a way that would leak information. See /model/benchmarks for how evaluation is framed.

Privacy and tokenization

If a sensitive field is removed or tokenized before training, lineage can prove where and when that happened, and show that no path bypasses the step. See data tokenization.

Limits of the role

Lineage shows the path, not the quality. A model trained on a faithfully traced but biased table is still biased. Use lineage with quality checks, profiling and review.

Practical guidance and best practices

Decide the questions first

List what lineage must answer: debugging, impact analysis, audit, privacy, ML traceability. The questions decide the level of detail and the systems to cover first.

Start with critical paths

Pick the reports, models and regulated outputs that matter most. Capture lineage for those end to end before expanding.

Prefer automatic capture

Parse code, collect runtime events and read logs wherever possible. Manual lineage is a last resort and needs an owner and a review date.

Record the method and time

Store how each edge was captured and when. Show users which links are exact and which are inferred.

Standardize event formats

Where possible, use a common event format across orchestrators and engines so that lineage can be gathered into one place. This reduces custom glue.

Keep history

Store lineage by version and date so you can reconstruct last quarter's flow. Link to table snapshots for reproducibility.

Make column-level lineage a priority for sensitive and critical fields

Column-level detail is costly. Use it where it matters most: personal data, financial figures and model features.

Connect to a catalog and glossary

Combine lineage with business terms and owners. See Informatica EDC and Informatica Axon. A graph with names but no owners is less useful in incidents.

Show manageable views

Provide filters by depth, system and asset type. Offer summary views that collapse internal steps. Large graphs need guided reading.

Use lineage in change management

Require an impact check in the review of any pipeline or schema change. Notify downstream owners automatically.

Add lineage to ML metadata

Record dataset versions, feature definitions, training code version and model version together. Make it possible to go from a prediction back to the training set.

Test lineage itself

Pick a few known flows and check that the graph shows them correctly. Re-test after major changes to tools or pipelines.

Protect lineage data

Lineage can reveal sensitive structure, such as where personal data lives. Restrict access appropriately.

Retire what is unused

Use downstream analysis to find tables with no consumers and remove them after review. This reduces cost and noise.

A lineage checklist for a new model

Storing this as structured metadata next to the model makes audits and incident reviews much faster.

A minimal event record

Runtime lineage tools usually exchange small structured events. The example below is a simplified sketch with placeholder names, not the exact schema of any standard.

event: job_completed
job: build_training_table
run_id: run-0001
finished_at: 2026-01-15T03:10:00Z
inputs:
  - silver.orders
  - silver.customers
outputs:
  - features.late_payment_training
transformations:
  - join on customer_id
  - filter status in (paid, overdue)
code_version: example-commit-abc123

Even this small record answers the main questions: what ran, what it read, what it wrote and which code did it.

Choosing the depth of detail

Match the effort to the risk. Over-detailed lineage on low-value assets costs time and adds noise.

Common mistakes

• Drawing it by hand once. The diagram is wrong within weeks.
• Assuming the graph is complete. Gaps in coverage are normal. Know where they are.
• Trusting inferred links as facts. Label and review them.
• Capturing only table-level lineage for sensitive fields. Privacy questions need column-level detail.
• No history. You cannot explain last month's number from today's graph.
• Ignoring non-SQL steps. Scripts, notebooks and file drops often hide key transformations.
• Showing everything. An unfiltered graph overwhelms users.
• Treating lineage as quality assurance. It shows paths, not correctness.
• Forgetting models and features. ML assets also need to be in the graph.
• No ownership. Without owners, impact analysis finds problems but nobody acts.
• Leaving out external data. Third-party feeds are part of the origin story.
• Not updating after migration. Moving systems breaks old lineage unless it is refreshed.
• Overinvesting in perfection. Focus on use cases, not on mapping every field in every system.

Frequently asked questions

What is data lineage in simple terms?

Data lineage is a record of where data came from, which steps changed it and where it ended up. It works like a tracking page for data, so you can follow a number in a report back to its sources or follow a source forward to everything that depends on it.

What is the difference between data lineage and data provenance?

The two ideas are closely related. Provenance often refers to the origin and history of a particular piece of data, especially in science and research, while lineage is the engineering term for tracing flows through pipelines and systems. In practice, people sometimes use the words interchangeably.

Why is column-level lineage important?

Table-level lineage says which tables feed which, but many questions are about individual fields, such as where an email address flows or how a revenue figure is calculated. Column-level lineage answers these precisely, which makes it valuable for privacy work, audits and tracing machine learning features.

How is lineage captured automatically?

Common methods include parsing SQL and job definitions, collecting events from running jobs and orchestrators, reading query logs, and inferring links from similarity. Most tools combine several methods, and the best ones record how each link was captured so users can judge its reliability.

Can lineage be wrong?

Yes. Parsed lineage can miss dynamic code, inferred lineage can guess incorrectly, and any lineage can become stale if capture stops. That is why good systems show the capture method and time, and why teams should spot-check critical paths.

How does lineage help with impact analysis?

It lets you start at an asset you plan to change and follow downstream links to every table, report and model that depends on it. You can then warn the owners and test those assets before the change goes live.

Does lineage help machine learning teams?

It does. It shows which source tables, joins and filters produced a training set, helps detect leakage, supports documentation of what data a model used, and warns when upstream changes might affect model inputs. It does not judge data quality by itself.

Is lineage the same as a data catalog?

No. A catalog is an inventory with search, descriptions, tags and ownership, while lineage is the flow graph between assets. Many catalog products include lineage as a feature, and the two are most useful together.

What is the difference between lineage and change data capture?

Change data capture streams row-level changes from a source system so that targets can stay in sync. Lineage records how assets relate and which jobs move or transform data between them. One moves changes, the other maps relationships.

How much lineage do we really need?

It depends on your risks and goals. Regulated outputs, sensitive data and production models justify detailed, column-level and historical lineage, while low-risk internal tables may need only table-level links. Start with critical paths and expand when there is a clear need.

Glossary

• Asset: a table, column, file, report or model in the graph.
• Business lineage: lineage described in business terms.
• Column-level lineage: lineage at the field level.
• Data contract: an agreement on the structure and rules of data between producer and consumer.
• Downstream: toward consumers.
• Edge: a link between two assets.
• Event: a record that a job read or wrote data.
• Impact analysis: finding what a change would affect.
• Inferred lineage: lineage guessed from names or values.
• Leakage: use of information that would not be available at prediction time.
• Lineage graph: the network of assets and flows.
• Orchestrator: a tool that schedules and runs pipeline jobs.
• Parsing: reading code to extract relationships.
• Provenance: origin and history of data.
• Reproducibility: ability to recreate a result from recorded inputs and code.
• Root cause analysis: tracing a problem back to its source.
• Technical lineage: lineage in terms of systems and objects.
• Transformation: an operation that changes data.
• Upstream: toward the sources.
• Versioned lineage: lineage kept with dates and job versions.

Summary

Data lineage records the path of data from source to use, down to individual columns when needed. It is captured by parsing code, collecting runtime events, reading query logs, inferring from similarity and, where unavoidable, manual entry. Its main uses are debugging, impact analysis, audit, privacy tracing and trust. Its main limits are incomplete coverage, dynamic code, staleness and the fact that a path shows movement rather than correctness.

For tabular foundation models, lineage gives the traceability that raw tables lack. It shows where training and in-context example tables come from, helps find leakage, explains nulls and mixed types, supports documentation and fair evaluation, and warns when upstream changes may affect a model. Combine it with a catalog, a glossary and quality checks for best results. Continue with Informatica EDC, Informatica Axon, data catalog, and the model pages /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Governance in my stack, under Data engineering and lakehouse. Others in the same group: