Synthetic data generation

Overview

Synthetic data generation is the practice of creating artificial datasets with a program instead of collecting them from the real world. The program may be a simple random generator, a physical simulator, a statistical model fitted to real data, or a large neural network. The output looks like real data in the ways that matter for the task, but each row is produced by a process you control.

The method is useful for three reasons. It can supply data when real data is scarce, private or expensive. It can cover rare cases that real logs almost never contain. And it can produce truly vast numbers of varied examples, which is what pre-training a foundation model requires. For tables, this last reason is central: models such as TabPFN learn from millions of generated datasets, each with its own structure, so that they can later predict on real tables they have never seen. This page explains how generation works, where it helps, and where it can mislead.

What it is

Synthetic data is data that was not recorded from real events but produced by a model of how data could be produced. Generation means sampling from that model.

Three broad families

• Rule-based or simulation-driven. You write the rules. A traffic simulator generates vehicle counts. A random program generates tables with chosen relations between columns.

• Statistical or learned from real data. You fit a model to a real dataset, then sample new rows from it. Examples include copulas, Bayesian networks, GANs, VAEs and diffusion models.

• Prior-based generation. You define a broad distribution over whole datasets, then sample many datasets. This does not copy any one real table. It aims to cover the space of plausible tables.

An everyday analogy

Think of a flight simulator. Pilots practise on a program that mimics weather, failures and traffic. They can rehearse engine fires a thousand times without a real fire. The simulator is not the real sky, but good training on it transfers. Synthetic data plays the same role for learning models: a safe, cheap, controllable practice ground.

What it is not

• It is not a guaranteed substitute for real data. A generator only knows what it was built or trained to know.

• It is not automatically private. A model fitted to real data can leak real records if it memorises.

• It is not the same as data augmentation, which transforms existing rows slightly. Generation can create wholly new rows or datasets.

Where it came from

Synthetic data is older than machine learning as a field.

• Monte Carlo simulation in the 1940s used random sampling to study physical systems. This is the root of sampling-based generation.

• Statistical disclosure control in the 1990s proposed releasing artificial microdata, generated by multiple imputation, to protect confidentiality of survey respondents.

• Simulators for robotics and vision in the 2010s produced labelled images at scale, with domain randomisation to help transfer to the real world.

• Generative adversarial networks (2014) and variational autoencoders (2013) gave a learned route to realistic samples. Tabular versions followed in the late 2010s, such as CTGAN (2019), which handled mixed numeric and categorical columns.

• Diffusion models (2020 onward) added another strong family of generators, now adapted to tables.

• Prior-data fitted networks (2022) showed that a transformer could be trained entirely on datasets sampled from a prior, then used to predict on real small tables. This turned synthetic generation from a side tool into the core of pre-training.

• Large language models are now used to write synthetic text, code and instructions for training other models.

For how this evolution feeds tabular foundation models, see /model/research.

How it works

The mechanics depend on the family. The steps below describe the most important cases.

Case A: Fit a model to real data, then sample

1. Clean the real table and decide which columns to keep.

2. Choose a generator suited to mixed types, such as a copula, a Bayesian network, a conditional GAN or a diffusion model.

3. Train the generator on the real rows.

4. Sample new rows.

5. Check fidelity, utility and privacy before using the output.

The analogy is a forger studying paintings. The better the study, the closer the copy. But the forger might also memorise one painting stroke for stroke, which is the privacy risk.

Case B: Simulate from known rules

1. Write equations or programs that describe the process.

2. Choose parameter ranges and noise levels.

3. Run the simulation many times with different random seeds and parameters.

4. Record the inputs and outputs as rows.

The analogy is a weather model. You know the physics, you set starting conditions, and you read off the results. Labels come for free, since you know the ground truth.

Case C: Sample whole datasets from a prior

This is the approach behind pre-training tabular foundation models. Rather than one dataset, the goal is a huge, diverse collection of datasets.

1. Sample a structure. Draw a random graph that says which variables influence which. Often this is a structural causal model, where each node is computed from its parents plus noise.

2. Sample functions. For each edge or node, pick a random function: a small neural network, a decision tree, a linear map, or a mix.

3. Sample column types. Turn some nodes into numeric columns, some into categories by binning, some into ordered values.

4. Sample noise and distortions. Add skew, outliers, heavy tails and rounding.

5. Sample missingness. Mask some cells with random or structured patterns.

6. Pick a target. Choose one node as the label, for classification or regression.

7. Draw rows. Generate as many rows as needed, with varied sizes.

8. Shuffle and drop names. Permute column order and remove or randomise names so that the model cannot rely on them.

The analogy is a language teacher who invents thousands of different puzzle languages, each with its own grammar. A student who practises on all of them learns how to infer grammar from a few sentences, which is what a new real language demands.

Why a causal graph helps

Real tables come from a world where some things cause others. A graph prior ensures that generated columns have realistic dependence: some are redundant, some are noisy copies, some are pure noise, and some interact in non-linear ways. A model trained on such variety learns to ask which columns matter, which is exactly the question a real table raises.

Controlling diversity

A good prior is wide. Common knobs include:

• Number of rows and columns.

• Number of classes, and class balance.

• Depth and width of the functions.

• Strength and type of noise.

• Fraction of informative versus useless columns.

• Mix of numeric and categorical types.

• Missing rate and missing mechanism.

If the prior is too narrow, the model fails on real data that falls outside it. If it is too wide and unstructured, learning becomes hard. Tuning this balance is a core part of architecture design and training design.

Checking quality

Three questions guide checking.

• Fidelity. Do the marginal and joint distributions match the target?

• Utility. Does a model trained on synthetic data perform well on real data? This is often called train on synthetic, test on real.

• Privacy. Can anyone recover or link real records from the synthetic set?

Key concepts and terms

A worked example

This example builds a tiny prior-based generator in Python. It samples a random causal graph, applies random functions, and returns a table with a binary target.

import numpy as np
rng = np.random.default_rng(0)

def random_function(n_in):
    """Pick a random non-linear function of n_in parents."""
    w = rng.normal(size=n_in)
    kind = rng.choice(["linear", "tanh", "square", "step"])
    def f(x):
        z = x @ w
        if kind == "linear": return z
        if kind == "tanh":   return np.tanh(z)
        if kind == "square": return z ** 2
        return (z > 0).astype(float)
    return f

def sample_dataset(n_rows=200, n_nodes=6):
    nodes = []
    for j in range(n_nodes):
        n_parents = rng.integers(0, j + 1) if j > 0 else 0
        parents = rng.choice(j, size=n_parents, replace=False) if n_parents else []
        nodes.append((parents, random_function(max(1, len(parents)))))
    X = np.zeros((n_rows, n_nodes))
    for j, (parents, f) in enumerate(nodes):
        noise = rng.normal(scale=0.3, size=n_rows)
        if len(parents):
            X[:, j] = f(X[:, parents]) + noise
        else:
            X[:, j] = rng.normal(size=n_rows)
    target = (X[:, -1] > np.median(X[:, -1])).astype(int)
    features = X[:, :-1]
    # make one column categorical by binning
    features[:, 0] = np.digitize(features[:, 0], np.quantile(features[:, 0], [.33, .66]))
    # random missing values
    mask = rng.random(features.shape) < 0.05
    features[mask] = np.nan
    # shuffle columns, no names kept
    perm = rng.permutation(features.shape[1])
    return features[:, perm], target

X, y = sample_dataset()
print(X.shape, y.mean())

Walking through it

• Each node can depend on earlier nodes, which keeps the graph acyclic.

• Each dependency uses a random function, so some relations are linear and some are not.

• The last node is turned into a binary label by splitting at its median, which gives a balanced target.

• One column is binned into three levels to act as a category.

• About five percent of cells are masked, to teach tolerance of gaps.

• Columns are shuffled and have no names, so a model cannot cheat with order or labels.

Using it for pre-training

Call sample_dataset millions of times with different settings. For each dataset, split the rows into a context part and a query part, show the context labels to the model, and train it to predict the query labels. This is the episodic training used for in-context learning. See few-shot learning and meta-learning.

A fitted-generator example

from sdv.single_table import GaussianCopulaSynthesizer
from sdv.metadata import SingleTableMetadata

meta = SingleTableMetadata()
meta.detect_from_dataframe(real_df)

synth = GaussianCopulaSynthesizer(meta)
synth.fit(real_df)
fake_df = synth.sample(num_rows=5000)

Then check utility by training a model on fake_df and testing on held-out real rows.

A numeric check on utility

Suppose a classifier trained on real data scores 0.90 on a held-out real test set. A classifier trained on synthetic data scores 0.86 on the same test set. The utility gap is 0.04. If the gap is small and privacy tests pass, the synthetic set may be good enough for sharing or augmentation. If the gap is 0.20, the generator misses important structure.

Strengths and limitations

Strengths

• Unlimited scale. You can generate as many rows or datasets as compute allows.

• Free labels. In simulation and prior-based methods, ground truth is known.

• Rare case coverage. You can oversample faults, fraud or edge conditions.

• Controlled experiments. You can vary one factor at a time and measure its effect.

• Privacy potential. Well-designed synthetic data can reduce exposure of real records.

• Fast iteration. Teams can build and test pipelines before real data arrives.

• Broad pre-training. A wide prior gives models a general-purpose inductive bias.

Limitations

• Reality gap. Anything the generator omits will be missing from the model.

• Garbage in, garbage out. A generator trained on biased data reproduces the bias.

• Privacy is not automatic. Overfit generators can memorise and leak rows.

• Mode collapse and smoothing. Learned generators often miss rare modes and tails.

• Hard-to-check validity. Constraints such as "age must be above zero" or "end date after start date" need to be enforced explicitly.

• Model collapse in loops. Training repeatedly on data generated by earlier models can degrade quality over generations.

• False confidence. Good scores on synthetic benchmarks may not carry to real data.

• Mixed types are harder. Tables with many categories, skewed numbers and missing values challenge many generators.

How it compares with alternatives

Choosing between them

• If you need to share data but cannot expose real records, consider fitted generators with privacy checks.

• If you need a broad prior for a foundation model, use prior-based generation at large scale.

• If you need rare cases, simulate them or oversample with conditional generators.

• If real labelled data is abundant and unrestricted, use it first, and use synthetic data only to fill gaps.

Role in tabular foundation models

Synthetic data generation is arguably the most important idea behind modern tabular foundation models. It answers a hard question: where do you find millions of labelled tables to pre-train on?

The data problem for tables

Images and text exist in huge public collections. Tables do not. Real tables are private, small, inconsistent, and rarely labelled in a uniform way. A model that must learn general rules for tabular prediction needs far more variety than any available corpus offers. Generation closes that gap.

Pre-training on synthetic tables

Models such as TabPFN are trained on datasets sampled from a prior built on random causal structures and random functions. Each training step uses a freshly generated dataset. The model learns a general procedure for inferring patterns from rows, which it later applies to real tables through in-context learning. TabICL follows a similar path with a design aimed at larger tables. See /model/architecture for how these networks read rows and columns.

Mixed data types

The prior must produce numeric, categorical, ordinal and sometimes text-like columns, so the model learns to treat each properly. This includes skewed amounts, integer counts, small categories and high-cardinality identifiers. A model that has only seen clean Gaussians would stumble on real data. See TabSTAR for text-aware approaches.

Missing values

Because the generator can mask cells under chosen mechanisms, the model learns missing data patterns directly. It sees cells missing at random, missing because of another variable, and missing because of their own value. This is the synthetic counterpart to masked cell modeling, and it supports imputation networks.

No column names

Generators can drop names and shuffle column order by design. The model must then find the role of each column from the data alone. This builds robustness to real exports with meaningless headers, and it avoids shortcuts that depend on name text.

In-context learning

Each synthetic dataset is one episode. The model sees labelled context rows and must predict query rows. Over millions of episodes, it learns to do inference inside its forward pass. This is the link to few-shot learning and zero-shot learning.

Anomaly detection

Generators can inject anomalies of known type, such as point outliers, shifted clusters or broken relationships, and label them. This yields supervised training signal for anomaly detection that real data seldom offers. It also gives test cases where the truth is known, which helps compare detectors such as Isolation Forest.

Continual learning

New synthetic families can be added over time to extend the model, for example a new type of seasonality or a new missingness pattern. To avoid forgetting older skills, training mixes new generated data with replayed old data. See experience replay and EWC.

Sector and semantic classification

Synthetic text and tables can be generated for specific sectors and column types, such as invoice lines, patient visits or sensor logs. These help teach a model to tag tables and columns by sector and meaning. See semantic sector heads. Care is needed so that generated examples reflect real vocabulary and not just the generator's own habits.

Mixing synthetic and real data

Many strong systems use both. Synthetic data teaches breadth and the inference procedure. Real data teaches the quirks that generators miss. A common design is to pre-train on synthetic data, then continue on a mixture that includes real tables, then test on held-out real benchmarks. For results, see /model/benchmarks.

Practical guidance and best practices

Start from the question

• Decide what the synthetic data is for: pre-training, testing, sharing, or augmentation.

• Each goal has different quality needs. Pre-training needs breadth, sharing needs privacy, testing needs known truth.

Design the prior or generator on purpose

• List the properties of real data that matter, such as skew, correlation, missingness and cardinality.

• Build those into the generator and test each one.

• Add constraints for valid ranges, formats and dependencies.

Keep diversity high

• Vary sizes, types, noise and structure widely.

• Include easy and hard problems.

• Include useless columns and duplicated columns, since real tables have both.

Validate against real data

• Compare marginals and correlations.

• Train on synthetic data and test on real data.

• Use a held-out real set that the generator never saw.

• Inspect samples by eye. Odd values often show up fast.

Test privacy

• Check nearest-neighbour distance from each synthetic row to real rows.

• Look for exact copies.

• Run membership-inference style tests.

• Consider differential privacy when the stakes are high.

Guard against feedback loops

• Do not train a generator on data produced by an earlier version of itself without fresh real data.

• Track provenance: label rows as synthetic and keep the generator version.

• Hold out real data for evaluation only.

Tune the mix

• Try different ratios of synthetic to real data.

• Weight real data more in the final stage.

• Use curriculum scheduling to move from simple to complex synthetic tasks.

Remove duplicates and near duplicates

• Generators can repeat themselves, especially learned ones. Apply deduplication before training.

Document everything

• Record the generator code, seeds, parameters and checks.

• Store summary statistics of each batch.

• Share a short data sheet describing known gaps.

Common mistakes

• Assuming realism without testing. A table can look right in a summary and still fail the train on synthetic, test on real check.

• Narrow priors. Training on one style of generator gives a model that fails on anything else.

• Leaky targets. A generator that builds the label from one clean column makes the task too easy and teaches the wrong lessons.

• Ignoring constraints. Negative ages, future birth dates and impossible combinations erode trust.

• Treating privacy as solved. Memorised rows can appear in the output.

• Evaluating on synthetic data only. Scores on the generator's own distribution say little about the real world.

• Forgetting missing values and noise. Real data is messy. Clean synthetic data produces fragile models.

• Mode collapse unnoticed. Check rare classes and tails, not only averages.

• Recursive training. Repeatedly training on generated data can drift and degrade.

• No provenance. If synthetic and real rows are mixed without labels, later audits become hard.

Frequently asked questions

What is synthetic data generation?

It is the process of creating artificial data with a program or a model instead of recording it from real events. The program can be a simulator, a statistical model fitted to real data, or a random generator that samples whole datasets. The goal is to get data that is useful for a purpose such as training, testing or sharing.

Why is synthetic data used to pre-train tabular models?

Real labelled tables are scarce, private and inconsistent, so there is not enough variety to teach a model general rules. Generators can produce millions of different tables with known structure at low cost. A model trained on this variety can then predict on real tables through in-context learning.

Is synthetic data private?

Not automatically. A generator fitted to real data can memorise and reproduce real rows, especially if it is overfit. Privacy needs to be tested with checks such as nearest-neighbour distances and membership inference, and strong guarantees can be added with differential privacy methods.

How do I know if synthetic data is good?

Check three things. Fidelity asks whether distributions and relations match the target. Utility asks whether a model trained on synthetic data works on real data. Privacy asks whether real records can be recovered. A good set passes all three for its intended use.

Can synthetic data replace real data completely?

For most real tasks it cannot, because a generator only knows what it was built or trained to know. It works best as a complement: it adds breadth, rare cases and safe sharing, while real data supplies the quirks of the actual domain. For pre-training, it can supply the bulk, but real benchmarks are still needed for evaluation.

What is a structural causal model in this context?

It is a set of variables where each one is computed from its parent variables plus noise. Using such models as the generator ensures that columns depend on each other in realistic ways, with some being redundant, some noisy and some irrelevant. Sampling many random versions gives a wide variety of tables.

What is model collapse?

Model collapse is the loss of quality and diversity that can occur when models are trained repeatedly on data produced by earlier models. Rare cases fade and errors compound. Keeping fresh real data in the loop and tracking provenance helps prevent it.

How does synthetic data help with missing values and anomalies?

A generator can mask cells in controlled ways and inject known anomalies, so the model sees labelled examples of both. This gives training signal that real data rarely provides, and it gives test cases where the truth is known. The patterns should still be checked against real data.

Should column names be included in synthetic tables?

It depends on the goal. If you want a model that works on raw exports with meaningless headers, drop or randomise names and shuffle column order. If you want a model that uses names when they exist, include realistic names for some fraction of the tables and leave them out for the rest.

How much synthetic data is enough?

There is no fixed amount. For fitted generators, enough rows to represent the real distribution are needed. For pre-training, the number of distinct datasets matters more than rows per dataset. The practical answer is to increase diversity and volume until a held-out real benchmark stops improving.

Glossary

• Augmentation: small changes to existing data to create variants.

• Causal graph: a diagram showing which variables influence which.

• Conditional generation: producing data that satisfies a given condition.

• Copula: a model of dependence between variables.

• Differential privacy: a formal privacy guarantee for outputs.

• Domain randomisation: wide variation of simulator settings.

• Fidelity: closeness of synthetic data to real data.

• Generator: a program or model that creates data.

• Mode collapse: a generator covering only part of the data.

• Model collapse: degradation from training on generated data repeatedly.

• Prior: a distribution used to sample datasets or parameters.

• Provenance: a record of where data came from.

• Reality gap: the difference between synthetic and real data.

• Simulation: running a model of a process to produce data.

• Synthetic data: data created by a generator rather than observed.

• Utility: usefulness of data for a task.

Summary

Synthetic data generation creates artificial datasets from simulators, statistical models or broad priors. It offers scale, free labels, coverage of rare cases and potential privacy benefits, but it brings a reality gap, the risk of leaking memorised rows, and the danger of narrow or recursive generation. Good practice sets a clear goal, builds a diverse generator, validates against held-out real data, and tests privacy.

For tabular foundation models it is the main source of pre-training data. Random causal structures, random functions, mixed column types, injected missing values and dropped column names teach a model a general procedure for learning from raw tables, which it applies through in-context learning. The same idea supports imputation, anomaly detection, sector and semantic tagging, and continual extension of the model. Related reading includes TabPFN, few-shot learning, architecture design, /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Foundation models from scratch in my stack, under Foundation models and applied ML. Others in the same group: