Data catalog

Overview

A data catalog is a searchable inventory of an organization's data assets, such as tables, files, dashboards, models and streams, together with the information needed to find, understand and trust them. For each asset it records what the data is, where it lives, who owns it, how it is described, how it relates to other assets and who is allowed to use it. It acts as a system of record for the question "what data do we have, and what does it mean?"

Catalogs matter because the volume and variety of data outgrow what any person can remember. Without a catalog, analysts ask colleagues, copy old queries and guess at column meanings. With one, they search, read a description, check quality signals and follow lineage to the source. Catalogs also support governance rules such as privacy and retention, and they supply the semantic context that machine learning needs, including models that learn from raw tables such as TabPFN and TabSTAR.

What it is

A data catalog is built from metadata, which is data about data. It does not usually store the business data itself. It stores descriptions and pointers.

Think of a library catalog. It does not contain the books. It lists each book with title, author, subject, shelf location and availability, and it lets you search and reserve. A data catalog does the same for datasets, with extra fields for ownership, quality, sensitivity and lineage.

Types of metadata

Core functions

• Discovery: search by name, description, tag, owner or column.
• Understanding: show schema, descriptions, sample values and usage.
• Trust: show quality checks, freshness and certification status.
• Governance: record owners, classifications, access policies and retention.
• Lineage: trace how data moves and changes across systems.
• Collaboration: let users comment, ask questions and request access.
• Automation: feed policies, alerts and other tools through an API.

Catalog, dictionary and glossary

These terms overlap, and it helps to separate them.

A catalog usually includes both a dictionary and a glossary, and links them: the term "customer" in the glossary points to the columns across systems that represent it.

Where it came from

The roots are old. Database systems have always kept a system catalog, a set of internal tables describing their own tables and columns. Data warehouses in the 1990s added metadata repositories that documented sources, transformations and reports, often kept in spreadsheets or specialized tools.

The data lake problem

When organizations began pouring raw files into object storage, the metadata problem got worse. A lake without a catalog turns into a swamp: nobody knows what a file contains, who wrote it or whether it can be deleted. Services such as the AWS Glue Data Catalog emerged to register tables over files and make them queryable. Access rules over those tables are handled by services such as AWS Lake Formation.

Enterprise catalogs

In parallel, enterprise vendors built catalogs aimed at governance and stewardship, with business glossaries, workflow for approvals and automated scanning of sources. Examples in this space include Informatica EDC, which scans and profiles metadata, and Informatica Axon, which focuses on governance and the business glossary. Open source and cloud-native catalogs followed, built around crawlers, search indexes and graph models of relationships.

Table format catalogs

Open table formats introduced a more specific meaning of "catalog": a service that tracks which tables exist and where their current metadata files are. The page on Iceberg cataloging covers this technical role, which is related to, but narrower than, the broader discovery catalog described here.

Drivers of growth

• Privacy law such as GDPR and KVKK requires organizations to know where personal data lives and why it is held.
• The data mesh idea gave domains ownership of data products, which need to be published and discoverable.
• Machine learning raised the need to know the origin, meaning and quality of training data.
• Self-service analytics put more people in front of more datasets.

How it works

Catalogs differ in design, but most follow the same flow.

Step 1: Connect to sources

The catalog connects to databases, warehouses, lakes, streaming systems, BI tools, pipelines and ML platforms through connectors. Each connector knows how to ask a system what it contains.

Step 2: Harvest technical metadata

Crawlers read schemas, table and column names, types, partitions, sizes and timestamps. They run on a schedule or react to events. When a schema changes, the new version is recorded, which makes the catalog a useful history for schema evolution.

Analogy: a census taker visits every house on a regular route, writing down who lives there and what has changed since the last visit.

Step 3: Profile the data

Optionally, the catalog or a linked tool computes statistics: row counts, null rates, distinct counts, min and max, value distributions, and top values. Profiles help users judge fitness and help automation classify columns.

Step 4: Classify and tag

Classification attaches labels such as "email address", "national id", "financial amount" or "sensitive". It can use column name patterns, regular expressions on values, dictionaries of known terms, or machine learning. Tools like Presidio detect personal data in text. Tags drive access and masking rules.

Step 5: Enrich with business context

People add what machines cannot guess: plain-language descriptions, the owning team, the steward, intended use, known caveats and links to glossary terms. Good catalogs make this easy and show which assets lack descriptions. Some use language models to suggest drafts that a human reviews.

Step 6: Capture lineage

Lineage is collected from query logs, pipeline definitions, transformation tools such as dbt, orchestrators and engines. The result is a graph: which sources feed which tables, and which tables feed which reports. See data lineage for details.

Step 7: Index for search

All metadata is indexed so users can search by words, filters and relationships. Many catalogs combine keyword ranking such as BM25 with semantic or hybrid search, so that a query like "monthly revenue by region" finds assets even if those exact words are not in the name. Search engines like Elasticsearch are common underneath.

Step 8: Apply governance

Policies connect to the catalog: who may see an asset, which columns must be masked, how long data may be kept. Access requests are routed to owners. Some catalogs enforce policies directly, others publish them to engines that enforce them.

Step 9: Keep it alive

A catalog is useful only if current. Mature setups use automated refresh, ownership reminders, usage-based ranking, deprecation flags and stale-asset reports. The aim is a catalog that people open daily because it saves them time.

Architecture at a glance

Key concepts and terms

A worked example

Registering and enriching an asset

A team publishes a table of customer orders. Below, a catalog entry is expressed as YAML, a form that many catalogs and metadata tools can ingest or resemble.

asset:
  name: sales.orders
  type: table
  platform: warehouse
  owner: team-commerce
  steward: data-governance
  description: >
    One row per customer order. Amounts are in minor currency units.
    Cancelled orders are included with status = 'cancelled'.
  domain: Sales
  tier: gold
  certified: true
  retention: 7 years
  tags: [finance, pii-indirect]
  columns:
    - name: order_id
      type: bigint
      description: Unique order identifier.
      glossary: Order
    - name: customer_id
      type: bigint
      description: Key to the customer table.
      glossary: Customer
      classification: pseudonymous-id
    - name: amount
      type: bigint
      description: Order value in minor units, for example cents.
      unit: minor currency unit
    - name: status
      type: string
      allowed_values: [open, shipped, delivered, cancelled]
  upstream: [raw.orders_events, raw.customers]
  downstream: [dashboards.revenue_daily, ml.churn_features]
  quality:
    freshness_sla: 2 hours
    tests: [not_null(order_id), unique(order_id)]

Querying the catalog

Many catalogs expose an API. A simplified Python sketch of a discovery call:

import requests

resp = requests.get(
    "https://catalog.example.internal/api/search",
    params={"q": "orders revenue", "type": "table", "certified": "true"},
    timeout=10,
)
for hit in resp.json()["results"][:5]:
    print(hit["name"], "|", hit["owner"], "|", hit["description"][:60])

Finding sensitive columns automatically

A rule-based classifier can run during harvesting:

import re

PATTERNS = {
    "email": re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$"),
    "phone": re.compile(r"^\+?[0-9 ()-]{7,15}$"),
}

def classify_column(values):
    sample = [v for v in values if v][:200]
    if not sample:
        return None
    for label, pat in PATTERNS.items():
        hit = sum(bool(pat.match(str(v))) for v in sample)
        if hit / len(sample) > 0.8:
            return label
    return None

Production classifiers combine several signals and send borderline cases to a person, since false positives and false negatives both have costs.

A lineage query

-- Which downstream assets break if sales.orders.amount changes meaning?
WITH RECURSIVE down AS (
  SELECT target FROM lineage_edges WHERE source = 'sales.orders.amount'
  UNION
  SELECT e.target FROM lineage_edges e JOIN down d ON e.source = d.target
)
SELECT target FROM down;

Column-level lineage turns a vague worry into a concrete list of affected reports and models.

Strengths and limitations

Strengths

• Faster discovery: users find data without asking around.
• Shared meaning: one definition per term reduces conflicting numbers.
• Trust signals: owners, quality checks and certification show which data is reliable.
• Governance support: classification, retention and access policies live next to the assets.
• Impact analysis: lineage shows what changes will affect.
• Less duplication: teams see that a dataset already exists.
• Onboarding: new staff learn the data landscape quickly.
• Audit readiness: records of ownership and data flows help with compliance.

Limitations

• Stale content: catalogs decay if not refreshed and owned.
• Incomplete coverage: some systems have no connector, and shadow data stays invisible.
• Cost of descriptions: automatic harvesting captures structure, not meaning, so human effort is needed.
• Lineage gaps: dynamic code, manual exports and spreadsheets break the chain.
• Low adoption: if the catalog is hard to use, people go back to asking colleagues.
• Over-tagging: too many tags and rules make the catalog noisy.
• A catalog describes data, it does not fix data. Quality still needs tests and process.
• Tool sprawl: a company can end up with several overlapping catalogs, one per platform.

How it compares with alternatives

Related systems often work together. A schema registry holds event schemas, the catalog links to them. A warehouse such as Snowflake or a platform such as Databricks ships its own catalog, which an enterprise catalog may federate. A model registry such as MLflow tracks models, and the catalog can link models to their training data.

Role in tabular foundation models

Tabular foundation models learn from raw tables, so the question is how a catalog helps a model that reads rows and columns directly.

Catalogs supply the meaning that raw tables lack

A raw table has numbers and strings. The catalog says what they mean: a column of integers is a count of units, a string column is an enumerated status, a float is a measurement in millimeters. This semantic layer is exactly what is missing when column names are absent or cryptic. Models such as TabPFN and TabICL, pre-trained on synthetic tables, work without any names and infer roles from values. When descriptions are available, text-aware models such as TabSTAR can turn them into extra input signal. A catalog is the source of that signal, and it keeps it consistent across tables.

Row and column structure, and mixed types

The catalog's technical metadata records the type, nullability and cardinality of each column. A model pipeline can use this to route each column to the right encoder: continuous values to a numeric path, categories to an embedding path, text to a text encoder, dates to a decomposition. This mirrors how tokenizer training fits cell encoders from data. Where the catalog's declared type disagrees with the data, as when a numeric column is stored as text, the discrepancy is a data quality finding to fix before modeling.

Missing values

Catalog profiles show null rates per column and over time. They distinguish a column that is rarely filled from one that became empty after an upstream change. Documenting why values are missing (not applicable, not collected, lost) is valuable because a model should treat these cases differently. This ties to self-supervised training with hidden cells, as in masked cell modeling, and to imputation networks. If values were imputed, the catalog should say so, and say how.

In-context learning needs the right context

With in-context learning, a model reads a labeled set of rows as context and predicts for new rows. The quality of the answer depends on choosing the right table and the right rows. The catalog helps select appropriate, certified and fresh datasets, and tells you which columns are identifiers or leakage risks that should be excluded. For example, a column that records the outcome after the fact is a leak. A good description flags it.

Sector and semantic classification

Assigning a table to an industry or semantic type is a classification problem that uses both values and text. Catalog metadata, such as domain labels, glossary links and column descriptions, provides training labels and features for it. Heads like semantic sector heads can then be checked against human-curated tags, and mismatches can feed back into the catalog as suggested classifications. This is the active metadata idea: models enrich the catalog, and the catalog grounds the models.

Anomaly detection

Catalog quality profiles give a baseline for what normal looks like for each column. Anomaly detection on those statistics, such as a jump in null rate or a shift in distribution, flags upstream problems. Together with schema drift detection, it lets a team learn about trouble before a model's inputs degrade.

Continual learning and governance of training data

As models are updated over time, teams must record which tables and versions each model saw. Lineage links models to their data, and certification marks which datasets are approved for training. Methods like experience replay and elastic weight consolidation address forgetting inside the model, but the catalog covers the other half: knowing exactly what was fed in, when, under which legal basis and with what sensitivity labels.

Privacy and compliance for model inputs

Classification and retention metadata keep sensitive columns out of training sets unless approved. Policies stored in the catalog can drive masking or tokenization of sensitive values before a table reaches a model. This is increasingly required under rules such as GDPR.

Summary of the mapping

See /model/architecture for how such a model consumes tables, /model/benchmarks for how it is measured, and /model/research for background.

Practical guidance and best practices

Start with a clear scope

• Pick the questions the catalog must answer first, such as "where is customer data" or "which tables feed the revenue report".
• Start with the most used and most critical assets, not everything.
• Define success in terms of use: searches, time to find data, share of important assets with owners.

Assign ownership

• Every asset gets an owner and a steward.
• Owners confirm descriptions and handle access requests.
• Show ownership prominently and remind owners when information goes stale.

Write good descriptions

• State what one row represents.
• State the unit, time zone and currency of key columns.
• Explain codes and statuses.
• Note known caveats, such as late-arriving data.
• Keep it short and factual. A paragraph beats a page.

Build the glossary carefully

• Define each term once, with a named owner.
• Link terms to the columns that implement them.
• Resolve conflicts early, for example two meanings of "active customer".
• Version definitions when they change.

Automate what can be automated

Integrate with the daily workflow

• Surface the catalog inside query tools and BI tools.
• Link from pipelines and dashboards back to catalog pages.
• Include catalog updates in the definition of done for new datasets.
• Make requests for access one click from the asset page.

Link to contracts and quality

• Attach data contract results to the asset page.
• Show test status and freshness next to the description.
• Mark deprecated assets and name their replacements.

Measure and maintain

• Track the share of assets with owner, description and classification.
• Track search success and zero-result queries to improve naming and tags.
• Retire unused assets on a schedule.
• Review sensitive classification regularly.

For machine learning teams

• Register training datasets and link them to model versions.
• Record the feature schema, encoders and any imputation rules.
• Record the legal basis and retention limit for training data.
• Keep a certified reference table for evaluation that does not change.

Search and naming tips

Search is the front door of a catalog, so small habits pay off.

• Use clear table and column names. A name such as customer_orders_daily is far easier to find than tbl_co_d.
• Put synonyms in descriptions. If users say "revenue" and the column says "net sales", mention both.
• Add a short summary line at the top of each asset page.
• Use a small controlled set of domains and tiers, and keep free tags for edge cases.
• Review searches that return nothing, then fix names, tags or glossary links.
• Rank certified and recently used assets higher than abandoned ones.

Adoption playbook

Roles and responsibilities

• Platform team: runs the tool, connectors, search quality and uptime.
• Domain owners: accountable for the data they publish.
• Stewards: maintain definitions, review classifications, resolve disputes.
• Consumers: report issues, add comments and use the request flow.
• Security and privacy staff: set classification rules and review sensitive access.

Common mistakes

• Treating the catalog as a one-time project instead of a living product.
• Loading thousands of assets with no owners or descriptions.
• Relying only on automated harvesting and never adding business meaning.
• Having no clear owner for the glossary.
• Allowing free-form tags to multiply into near duplicates.
• Ignoring column-level lineage and then being unable to assess impact.
• Not covering shadow systems such as spreadsheets and exports.
• Skipping access control on the catalog itself, which can leak sensitive metadata.
• Making the catalog separate from the tools people use daily.
• Confusing the catalog with data quality. Showing a flag is not fixing a problem.
• Duplicating catalogs per platform with no federation.
• Forgetting to record imputed values and synthetic data as such.
• Failing to link models and features to their source data.

Frequently asked questions

What is a data catalog?

A data catalog is a searchable inventory of data assets, such as tables, files, dashboards and models, with metadata about meaning, structure, ownership, quality, sensitivity and lineage. It does not typically hold the data itself. It helps people find data, understand it and decide whether to trust it.

What is the difference between a data catalog and a data dictionary?

A data dictionary defines the fields of a particular system or table. A data catalog covers all assets across many systems and adds search, ownership, lineage, quality signals and governance. A catalog usually contains dictionaries and a business glossary, and connects them to each other.

What kinds of metadata does a catalog store?

Typical categories are technical metadata such as schemas and types, business metadata such as descriptions and glossary terms, operational metadata such as refresh times and job status, governance metadata such as owners and classifications, quality metadata such as test results, and lineage metadata showing sources and consumers.

Why is data lineage part of a catalog?

Lineage shows where data came from and where it goes. It supports impact analysis before changes, root cause analysis when numbers look wrong, and compliance questions about personal data flows. Having lineage in the catalog lets users jump from an asset to its sources and consumers in one place.

How does a data catalog relate to a schema registry?

A schema registry stores versioned schemas for events and checks compatibility when producers publish. A data catalog is broader, covering many asset types with business context and search. The two work well together, with the catalog linking to registry entries so people can see the structure of a stream next to its description and owner.

Does a data catalog help with privacy compliance?

Yes. By classifying columns, recording legal basis and retention, and showing where personal data flows, a catalog helps organizations answer questions required under privacy regulations. It does not make a system compliant by itself, since policies must also be enforced in the systems that process the data.

How do tabular foundation models benefit from a data catalog?

Catalog metadata supplies the meaning that raw tables often lack, including descriptions, units, types and domain labels. It helps route columns to the right encoders, choose certified tables for context, exclude leaky or sensitive columns, and track which data a model saw. Models can also enrich the catalog by suggesting classifications.

How do you keep a catalog from going stale?

Automate harvesting and profiling, assign an owner to every asset, send reminders for unreviewed descriptions, rank search results by real usage, and flag deprecated or unused assets. Including catalog updates in the process for releasing a new dataset also keeps new content flowing in.

Who should own the data catalog?

The platform is usually run by a data platform or governance team, but the content belongs to the domain teams that own the data. Owners and stewards of each asset keep descriptions and classifications accurate, while the central team provides tooling, standards and support.

Can a language model write catalog descriptions?

It can draft them from schemas, sample values and query history, which saves time on large inventories. Drafts should be reviewed by someone who knows the data, because a fluent but wrong description is worse than none. Marking machine-suggested text clearly helps reviewers and readers.

Glossary

• Active metadata: metadata that triggers actions in other systems, such as alerts or policy updates.
• Asset: any item registered in the catalog.
• Business glossary: a controlled list of business terms and definitions.
• Certification: a mark showing an asset has been reviewed and approved.
• Classification: labeling data by type or sensitivity.
• Column-level lineage: lineage traced for individual columns.
• Crawler: a process that reads metadata from a source system.
• Data contract: an agreement on shape and quality between producer and consumer.
• Data dictionary: definitions of fields in a system.
• Data mesh: an approach where domain teams own data as products.
• Data owner: the accountable person or team for an asset.
• Data product: a dataset packaged with documentation, ownership and a contract.
• Data profile: summary statistics for a dataset.
• Data steward: a person who maintains definitions and quality in a domain.
• Lineage: the flow of data from source to consumer.
• Metadata: data that describes other data.
• Provenance: the origin and history of data.
• Tag: a label attached to an asset.

Summary

A data catalog is a searchable, governed inventory of data assets that combines technical, business, operational, quality and lineage metadata. It is built by connecting to sources, harvesting and profiling metadata, classifying and enriching it, capturing lineage, indexing for search and applying governance rules. It works only when owned and kept current, and it complements, not replaces, quality tests, schema registries and table format catalogs. For tabular foundation models, the catalog supplies the semantics that raw tables lack, guides encoder choice, documents missingness and imputation, supports sector classification and records which data each model saw. Related reading includes Data lineage, Schema evolution, AWS Glue Data Catalog and /model/architecture.

Related

It belongs to Governance in my stack, under Data engineering and lakehouse. Others in the same group: