Informatica EDC

Overview

Informatica Enterprise Data Catalog (EDC) is a catalog that scans an organization's data systems and builds a searchable inventory of what exists. It reads metadata from databases, files, integration tools and reporting tools. It profiles values, suggests classifications such as "email address" or "national identifier", and maps how data moves between systems. The result is a single place to find data and trace where it came from.

An enterprise has thousands of tables across many platforms, and no person knows them all. A catalog turns that sprawl into something people can search and trust. For tabular machine learning, this is practical help: before a model can learn from a table, someone must find the right table, understand its columns, know whether it is sensitive and know where it came from. This article explains how a catalog like EDC works, how it relates to Informatica Axon and data lineage, and how it supports tabular foundation models.

What it is

EDC is a metadata platform. Metadata is data about data: names, types, sizes, relationships, sample values and more. EDC does not store your business data. It stores descriptions of it and links between those descriptions.

Think of a very large warehouse with no signs. A catalog is the worker who walks every aisle with a scanner, writes down what is on each shelf, notes which boxes were moved from where, and puts the list in a search box.

What EDC catalogs

The exact list of supported sources depends on the product version and the connectors installed. Check current documentation for the full list.

Core functions

• Discovery: find assets across systems.
• Profiling: compute statistics such as counts, distinct values, nulls and value patterns.
• Classification: tag columns with types of content, such as personal data or financial codes.
• Relationship detection: suggest joins, similar columns and duplicate data.
• Lineage: show how data flows between assets.
• Search and browse: let users find assets by name, description, tag or content.
• Enrichment: attach business terms, owners and notes, often by linking to a governance tool.
• Impact analysis: show what would be affected by a change.

Who uses it

Data engineers use it to understand sources and plan changes. Analysts use it to find trustworthy tables. Stewards use it to classify and describe data. Privacy and security teams use it to locate sensitive fields. Machine learning teams use it to find and assess training data.

Where it came from

Enterprise catalogs came from a recurring problem. Data warehouses and integration tools had always kept some metadata, but each tool kept its own. Teams could not answer simple questions such as "where does this report field come from?" without reading code.

Metadata management grew as a discipline in the 2000s. Early repositories required manual entry, which became outdated quickly. As data lakes and cloud platforms multiplied the number of data stores, manual entry became impossible. Vendors began building scanners that harvest metadata automatically and use profiling and pattern rules to suggest classifications.

Informatica is a long-established vendor of data integration, data quality and master data products. Because its integration tools already described data flows, it was natural to combine that knowledge with a scanning catalog. EDC is its enterprise catalog product, designed to work with its governance and quality tools. Product names and cloud packaging have changed over time, so check current vendor material for the current form.

Across the industry, catalogs have moved from static inventories toward active systems. They now use machine assistance to propose tags, detect similar columns and rank search results. They also expose interfaces so that other tools can read catalog data, for example to enforce access rules or to feed model documentation.

How it works

Step 1: Connect to a source

An administrator defines a connection to a source, such as a database, file store or integration tool. In EDC terms this is often called a resource. The connection uses a read-only account with the least privilege that is still enough to read metadata. Credentials are stored securely and not in scripts. See HashiCorp Vault for the general idea of secret management.

Step 2: Scan metadata

A scanner reads the structure of the source: schemas, tables, columns, types and relationships. For integration tools, it reads job definitions. For reporting tools, it reads report definitions and fields. The scanner turns all of this into catalog objects.

Analogy: a librarian recording title, author and shelf for each book without reading it.

Step 3: Profile the data

Profiling goes a step deeper. The scanner samples or reads values and calculates statistics:

• row count,
• null count and percentage,
• distinct count,
• minimum, maximum and typical values,
• most frequent values,
• common patterns, such as AAA-9999 or name@domain.

Profiling can be expensive on large tables, so teams usually sample. Sampling is a tradeoff: it is cheaper but can miss rare values.

Step 4: Classify columns

Classification rules decide what kind of data a column holds. Rules can look at:

• the column name (for example, contains "email"),
• the data pattern (matches a phone number format),
• the values compared with a reference list (country codes),
• statistical similarity to already classified columns.

The rule engine assigns a tag plus a confidence level. A steward can accept or reject the suggestion. Accepted tags improve future suggestions and searches.

This step is especially valuable for finding sensitive data in tables with unhelpful names.

Step 5: Detect relationships and similarity

The catalog looks for columns that appear related, such as a likely foreign key where values in one column are a subset of values in another. It also finds similar columns across systems, helpful for spotting copies. These hints guide data modelers and speed up integration work.

Step 6: Build lineage

Lineage is a graph of how data moves. EDC derives it from integration job definitions, SQL, view definitions and report definitions. Nodes are assets, and edges are flows or transformations. A user can start at a report field and move upstream to sources, or start at a source column and move downstream to everything that depends on it.

Some lineage is exact because it is parsed from code. Some is inferred by matching names and values, and should be reviewed. See data lineage for the general concept and its limits.

Step 7: Enrich and connect

The catalog can link assets to business terms, owners and policies. Often these come from a governance tool such as Informatica Axon. The link brings business meaning to technical assets.

Step 8: Search and consume

Users search by keyword, filter by system, tag or owner, and open an asset page. The page shows description, columns, profile statistics, classifications, related assets and lineage. Search ranking can use signals such as usage and popularity.

Step 9: Keep it fresh

Scans are scheduled. After each scan, the catalog updates what changed: new tables, dropped columns, altered types. A catalog that is not refreshed becomes untrustworthy. Incremental scans reduce cost.

Step 10: Share with other tools

Catalog data can feed other systems through interfaces. Examples include access control tools that use tags to mask columns, quality tools that pick columns to test, and documentation generators. The details depend on version and setup.

How the pieces fit together

Sources            Scanners          Catalog                Consumers
--------           --------          -------                ---------
Warehouse   --->   metadata  --->    Assets + profile --->  Search, impact analysis
Lake files  --->   profiling --->    Classifications  --->  Masking and access rules
ETL jobs    --->   lineage   --->    Lineage graph    --->  Audit and model documentation
BI reports  --->   parsing   --->    Links to terms   --->  Governance workflows

Key concepts and terms

A worked example

A team wants to train a model to predict late payments. They need to find suitable tables, check that sensitive fields are handled, and understand where the data comes from.

Step A: Search

An analyst searches the catalog for "invoice payment". Three tables appear.

The analyst picks the certified table. The temporary and old copies are ignored. Without a catalog, the team might have used the wrong copy.

Step B: Inspect the profile

The asset page for invoice_payments shows column profiles:

The numbers are illustrative. The profile reveals two things: paid_date has many nulls, and two columns are sensitive.

Step C: Interpret the nulls

A steward's note says that paid_date is null while an invoice is unpaid. So these nulls are not errors. They carry meaning. If the model treats null as "missing at random", it will misread them. The team decides to encode "unpaid" explicitly and to build the label from paid_date versus due_date.

Step D: Handle sensitive fields

Classification shows customer_email and tax_ref as sensitive. The team removes them from the training table. They keep a hashed customer key instead, created with a one-way hash plus a secret value from a secret manager.

Step E: Trace lineage

The lineage view shows amount is computed by a nightly job from two source tables and then adjusted by a currency conversion step. The team learns that amounts are in a single reporting currency. That affects feature scaling and comparison across regions.

Step F: Plan for change

Impact analysis shows that if the finance team changes the conversion job, the model's feature will shift. The team asks to be notified of changes to this job.

A sketch of a classification rule in pseudo-configuration

classification_rule:
  name: Email address
  match:
    column_name_contains: [email, e_mail, mail_addr]
    value_pattern: "^[^@\\s]+@[^@\\s]+\\.[a-z]{2,}$"
    min_match_ratio: 0.8
  result:
    tag: Email address
    sensitivity: personal
    confidence: high

The structure is a generic sketch to show the idea of name hints plus pattern checks plus a match threshold. It is not the exact syntax of any product.

A small script that mimics the pattern check

import re

email = re.compile(r"^[^@\s]+@[^@\s]+\.[a-z]{2,}$", re.I)

def match_ratio(values):
    non_null = [v for v in values if v]
    if not non_null:
        return 0.0
    hits = sum(1 for v in non_null if email.match(v))
    return hits / len(non_null)

sample = ["a@example.internal", "b@example.internal", "n/a", "c@example.internal", None]
print(round(match_ratio(sample), 2))   # 0.75

With a threshold of 0.8, this column would not be tagged automatically. It would go to a steward as a low-confidence suggestion. That is how a catalog balances automation and review.

Strengths and limitations

Strengths

• Discovery at scale. Automated scanning covers far more than manual lists.
• One search box. People find data without asking around.
• Sensitive data location. Pattern and value checks find hidden personal fields.
• Lineage and impact analysis. Changes can be assessed before they cause outages.
• Profiling insight. Basic statistics are visible without writing queries.
• Integration with governance. Technical assets can be tied to business meaning and ownership.
• Reduced duplication. Similarity detection helps find copies and prevent new ones.
• Audit support. The inventory and classification history help with compliance questions.

Limitations

• Metadata only. It describes data but does not clean or move it.
• Connector coverage. Not every source or tool is supported equally. Gaps appear in custom code and unusual systems.
• Lineage gaps. Lineage parsed from code is strong, but dynamic SQL, scripts and manual steps may be missed. Inferred lineage can be wrong.
• Profiling cost. Full profiling of huge tables can be slow and expensive. Sampling misses rare cases.
• Classification errors. Pattern rules cause false positives and false negatives. Human review is needed for important decisions.
• Staleness. Without scheduled scans, the catalog drifts from reality.
• Setup and operations. Large deployments need infrastructure, administration and tuning. See infrastructure as code for repeatable deployment ideas.
• Adoption risk. If users do not trust or enrich the catalog, it becomes a dead inventory.
• Version differences. Features differ between releases and editions. Confirm in current documentation.

How it compares with alternatives

A general overview of the idea is in data catalog. Many organizations run more than one: for example a cloud catalog for operational table metadata and an enterprise catalog for cross-platform search and lineage.

Role in tabular foundation models

A catalog does not train or run a model. Its role is upstream: it helps people find, understand and responsibly use the tables that models learn from. Tabular foundation models read rows and columns of mixed types, with missing values and often no trustworthy column names. A catalog supplies the context that such tables lack.

Finding the right tables

Pre-training and fine-tuning use many tables, whether synthetic or real. For real tables inside an organization, the catalog is the inventory. Search, certification and ownership signals help pick tables that are complete, current and approved. Duplicate detection reduces the risk of training on the same data many times. For research background see /model/research.

Column meaning when names are missing

Foundation models for tables are often designed to work without column names, because real names can be cryptic or absent. The catalog can still provide meaning outside the model: profiled value patterns, classifications and linked business terms. These can be used by humans, or as optional side information, to decide which columns are identifiers, dates, categories or sensitive fields. Pattern-based tags are a useful sanity check on any automatic type inference done by a model. This relates to semantic sector heads, which infer category from values.

Mixed types and profiling

A model must handle numbers, categories, dates and text in one table. Profiling reveals type distributions, cardinality and value ranges before training. A high-cardinality string column behaves very differently from a five-value category. Distinct counts and top values help set sensible handling rules. The architecture implications of such mixed inputs are described on /model/architecture.

Missing values and imputation

The null percentage in a profile is a first signal for imputation planning. A stewardship note that explains why a field is empty (not applicable, not yet known, or lost) tells you whether to impute or to keep the gap as information. See imputation networks. Catalog profiling can also be rerun after imputation to check that distributions did not shift unexpectedly.

Anomaly detection

Profiles over time show drift in counts, ranges and patterns. A sudden jump in nulls or a new category is often the first sign of a broken upstream job. This pairs with model-based anomaly work described in anomaly detection. The catalog supplies the baseline expectations, and the model supplies row-level judgment.

Sensitive data discipline

Before tables are used for pre-training, fine-tuning or in-context prompts, personal and confidential columns need removal, masking or tokenization. Classification helps find them. For techniques see data tokenization. A catalog does not replace legal review, but it makes the technical search for risky fields far more complete.

In-context learning inputs

In-context learning gives a model labeled example rows plus new rows to predict. The quality of those examples comes from the source table. Lineage helps verify that labels were produced by the right process and that features were available at prediction time. This helps avoid leakage, one of the biggest causes of unrealistic results.

Continual learning and drift

When a model is updated with new data, upstream changes matter. Lineage and scheduled scans reveal changed columns, new types or modified transformation jobs. This gives an early warning that the model's inputs have shifted. See schema drift detection.

Documentation and benchmarks

Model documentation needs a trace from training table to source system. A catalog with lineage provides much of that trace. For fair benchmarking, the catalog also records versions and owners of evaluation tables, helping prevent contamination between training and test sets. Compare with /model/benchmarks.

Limits of the connection

The catalog describes tables. It does not guarantee they are correct, representative or unbiased. A model trained on a poor table still learns poor patterns. Treat catalog information as strong guidance, not proof.

Practical guidance and best practices

Define your goals first

Decide which questions the catalog must answer: find sensitive data, support audits, speed up analysts, or support ML data selection. Goals determine which sources to scan first.

Start with high-value sources

Scan the warehouse, key lake zones and main integration tool first. Add others over time. Aim for useful coverage fast instead of perfect coverage slowly.

Use least-privilege accounts

Create read-only accounts that can see metadata and, where needed, sample data. Limit profiling of sensitive tables to approved environments. Store credentials in a secret manager.

Tune profiling

• Sample large tables rather than scanning everything.
• Increase sample size for columns that drive decisions.
• Exclude columns known to be free text with personal comments from content profiling if policy requires it.
• Schedule heavy scans in quiet hours.

Review classification suggestions

Treat suggestions as proposals. Set a confidence threshold for automatic acceptance, and send the rest to stewards. Log accepted and rejected suggestions to improve rules.

Create custom rules for your own data

Built-in rules cover common types. Add rules for internal codes, product identifiers and local formats. Test them against sample columns before enabling.

Link to the glossary

Tie assets to business terms and owners. See Informatica Axon. A catalog without business context is a technical index only.

Certify trusted assets

Create a simple certification process. Mark tables as certified, in review or deprecated. Make search ranking show certified assets first.

Verify lineage on critical paths

Spot-check lineage for the most important reports and ML data sets. Where parsing misses a step, add manual links and document why.

Schedule scans and monitor them

Run scans on a schedule with alerts for failures. A failed scan that nobody notices leads to a stale catalog. Use incremental scans where available.

Measure usage

Track searches, asset views and enrichment actions. Remove or fix assets nobody uses. Ask frequent users what is missing.

Plan deployment as code

Large catalog deployments need servers, storage and networking. Define them in code for repeatability, see infrastructure as code.

Prepare data for ML explicitly

Add ML-friendly tags: target column, time column, entity key, leakage risk, sensitive and allowed-use. Make those tags searchable so model builders can filter quickly.

Document ownership

Every asset should have a named owner or steward. Orphaned assets cause delays during incidents and audits.

A readiness checklist for ML data selection

Before a table is used to train, fine-tune or prompt a model, a short checklist drawn from catalog information can save days of rework.

Using the catalog with a pipeline

A typical pattern is to query catalog information through an interface and make a decision in code. The sketch below shows the idea with placeholder names and a fake host.

import json
from urllib.request import Request, urlopen

BASE = "https://catalog.example.internal/api"

def get_asset(asset_id, token):
    req = Request(f"{BASE}/assets/{asset_id}",
                  headers={"Authorization": f"Bearer {token}"})
    with urlopen(req) as resp:
        return json.load(resp)

def blocked_columns(asset):
    return [c["name"] for c in asset["columns"]
            if c.get("sensitivity") in ("personal", "confidential")]

# asset = get_asset("invoice_payments", token)
# drop = blocked_columns(asset)

The endpoint and field names above are invented for illustration. Real products define their own interfaces, so use the vendor documentation. The pattern is what matters: read tags from the catalog and drop or mask sensitive columns before building a training table.

Scan scheduling by source type

Handling free text

Free-text columns such as notes and comments can hold personal details in no fixed pattern. Profile them with care. Consider treating them as sensitive by default, and exclude them from model input unless a clear review has approved their use.

Common mistakes

• Scanning everything at once. Slow, noisy and hard to validate.
• Trusting automatic tags blindly. False positives waste time, and false negatives create risk.
• Ignoring refresh. A catalog that is months old misleads users.
• No owners. Findings cannot be acted on if nobody is responsible.
• Profiling production tables without care. Heavy scans can slow live systems.
• Over-privileged scan accounts. A catalog account does not need write access.
• Assuming lineage is complete. Check dynamic code, manual files and external tools.
• Treating the catalog as the source of truth for data content. It describes data, not guarantees it.
• Leaving out business context. Technical names alone are hard to use.
• Not training users. People will keep asking colleagues instead of searching.
• Failing to retire stale assets. Old copies confuse search results.
• Forgetting derived assets. Feature tables, extracts and model outputs need entries too.
• Skipping change communication. Users need to know when definitions or lineage change.
• Copying sample values into notes. Sample data may be sensitive. Keep notes free of real personal values.

Frequently asked questions

What does Informatica EDC do?

It scans data systems, collects metadata, profiles values, suggests classifications and builds lineage so that people can search for and understand data across an organization. It is an inventory and map of data assets rather than a store of the data itself.

Does the catalog store my actual data?

Its main job is to store metadata and statistics, not the business data itself. Profiling may read sample values and may keep some summaries or sample content depending on configuration, so teams should review the settings and restrict sample storage for sensitive tables.

How is EDC different from a governance tool?

A catalog discovers and describes technical assets automatically, while a governance tool curates business meaning, ownership, policy and approval workflows. They work best together, with the catalog supplying scanned assets and the governance tool supplying definitions and accountability.

How accurate is automatic classification?

It is useful but not perfect. Name and pattern rules can mislabel columns, and unusual formats can be missed. Good practice is to use confidence scores, send uncertain results to a steward and add custom rules for local formats.

Can it show where a report number comes from?

Yes, when the report and the upstream jobs are scanned and parsed, lineage can trace a field back through transformations to source columns. Coverage depends on the connectors and on how much logic is expressed in code the scanner can read, so critical paths should be verified.

Is it useful for machine learning teams?

Yes. It helps find certified tables, identify sensitive columns, check null rates and value distributions, and trace labels and features back to their sources. These steps reduce wasted effort and reduce the risk of leakage and privacy problems.

How often should scans run?

It depends on how fast your systems change. Many teams scan critical sources daily or weekly and others monthly, using incremental scans where possible. The goal is that the catalog is fresh enough to be trusted for the decisions it supports.

Can a catalog find personal data in tables with cryptic column names?

Often yes, because value patterns and similarity can reveal content even when names are unhelpful. Results should still be reviewed, since free-text fields and unusual formats can hide personal data from automated rules.

What is the difference between profiling and lineage?

Profiling describes what is inside a column: counts, nulls, patterns and common values. Lineage describes how data moves between assets and what transforms it. Profiling answers "what does this look like", and lineage answers "where did this come from and what uses it".

Do I need a catalog if I only use a cloud data lake?

You need some form of catalog, because without one the tables cannot be queried or found reliably. A cloud-native catalog may be enough for a single platform, while an enterprise catalog adds value when you have many systems, need cross-platform lineage or require richer classification and governance links.

Glossary

• Asset: any object tracked by the catalog.
• Business term: a glossary entry linked to technical assets.
• Catalog: an inventory of data assets with search and context.
• Certification: a label marking an asset as reviewed and trusted.
• Classification: a tag describing content type or sensitivity.
• Confidence score: a measure of how sure the rule is about a tag.
• Connector: a component that reads from a specific type of source.
• Data domain: a content type that the catalog can recognize in columns.
• Impact analysis: finding what depends on an asset.
• Incremental scan: a scan limited to changes since the last run.
• Lineage: the traced path of data between assets.
• Metadata: information describing data.
• Profiling: calculating statistics about values.
• Resource: a configured source connection.
• Sampling: reading a subset to estimate statistics.
• Scanner: the component that harvests metadata.
• Sensitive data: information that needs protection, such as personal identifiers.
• Steward: the person who reviews and maintains asset descriptions.
• Similarity detection: finding columns or tables that look alike.

Summary

Informatica EDC is an enterprise data catalog. It connects to many systems, scans metadata, profiles values, classifies columns and builds lineage, giving people a searchable map of their data. Its strengths are discovery at scale, sensitive data location and impact analysis. Its limits are connector coverage, partial lineage, classification errors and the need for regular refresh and human review.

For tabular foundation models, a catalog is the supporting layer that helps select good tables, understand columns without relying on names, plan for missing values, spot drift and protect sensitive fields. It does not replace modeling or legal review, but it reduces guesswork before data reaches a model. Learn more in data lineage, data catalog, Informatica Axon, and the model pages /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Governance in my stack, under Data engineering and lakehouse. Others in the same group: