Curriculum learning

Overview

Curriculum learning is a training strategy in which a model sees examples in a meaningful order, usually from easy to hard, instead of in random order. The idea comes from how people learn: children start with letters before essays, and students start with addition before calculus. In machine learning, the same idea can make training more stable, speed up early progress and sometimes lead to a better final model, especially when the task is hard, the data is noisy or the model is large.

A curriculum has two parts. A difficulty measure says which examples are easy and which are hard. A schedule says when harder examples are introduced. Both can be hand designed or learned from the model itself. For models that learn from raw tables, such as those described on /model/architecture, the curriculum often acts on the shape and complexity of the tables and on the structure of synthetic data generators: small tables with simple relations come first, and large tables with tangled relations, many columns, noise and missing values come later.

What it is

Curriculum learning is a method of organizing the training data over time. Rather than drawing every batch uniformly from the full dataset, you control the mix of examples that appear at each stage of training.

The core idea

A model that starts training knows nothing. Its first updates are noisy and large. If those early updates come from very hard or very noisy examples, the model may chase noise and settle in a poor region of the parameter space. If the early updates come from clean, simple examples, the model builds basic features first. Later, harder examples refine those features.

An everyday analogy

Consider learning to play the piano. A teacher does not start with a complex concerto. You begin with single notes, then simple scales, then short pieces with one hand, then both hands, then faster tempos and finally full works. Each stage relies on skills from earlier stages. A curriculum for a model follows the same logic: it provides a path of steadily increasing challenge.

Closely related ideas

• Anti-curriculum: harder examples first. Occasionally useful but usually worse.
• Self-paced learning: the model itself decides which examples are currently easy, based on its loss.
• Hard example mining: focus on examples the model gets wrong. This is the opposite emphasis from an easy-first curriculum, though it is often used later in training.
• Data mixing and scheduling: change the proportions of different data sources over time.
• Progressive resizing or growing: start with small inputs or small models and grow them.
• Transfer learning: learn one task first and reuse it for another. See transfer learning.

What counts as "difficulty"

Difficulty is not a single concept. Common choices include:

Where it came from

Ideas about learning in stages appear in psychology and education long before machine learning. In the animal training literature, a technique called shaping rewards successive approximations to a target behavior. In neural network research, a 1993 paper on learning grammar, titled "Learning and development in neural networks: the importance of starting small", showed that a recurrent network could learn complex sentence structure when training began with simple sentences or when its memory was limited at first, but failed when trained on complex sentences from the start.

The modern formulation came from a 2009 paper titled "Curriculum learning". It proposed that training on examples organized by increasing difficulty can speed convergence and improve generalization, and tested the idea on simple shape recognition and language modeling tasks. It framed a curriculum as a continuation method: a way to solve a hard non-convex optimization problem by starting with a smoothed or easier version and gradually moving to the real problem.

A related 2010 paper on self-paced learning proposed that the model should pick its own pace by selecting examples with low loss first and adding harder ones as it improves.

Later work extended the concept to reinforcement learning, where an agent practices on easy tasks before hard ones, to generative models, where image resolution grows during training, and to language models, where sequence length and data quality are scheduled. Survey papers over the following decade organized the field into questions of how to measure difficulty and how to schedule examples.

Results have been mixed, which is part of the honest story. Curricula sometimes give clear gains, sometimes give small speed ups and sometimes give nothing, particularly when the data is already clean and abundant and the model is large enough to learn everything regardless of order.

How it works

A curriculum system answers three questions: how hard is each example, in what order or mix should they appear, and how fast should the difficulty rise.

Step 1: measure difficulty

Pick a difficulty score for each example or for each data source. Approaches:

Hand designed scores. Use domain knowledge. For text, sentence length. For tables, number of columns. For a game, the number of steps to the goal.

Model based scores. Train a small model first and use its loss on each example as difficulty. Examples with high loss are hard. This is cheap and flexible, but it measures difficulty for that small model, not necessarily for the final one.

Teacher based scores. A strong pre-trained model rates examples. Disagreement between several models can also be a signal.

Dynamic scores. Recompute difficulty during training from the current model. This is what self-paced learning does.

Step 2: choose a pacing function

The pacing function says what share of the data, or which difficulty level, is available at each point in training.

Step 3: build each batch

At each step, sample a batch from the currently allowed pool. Two common ways:

• Cap: only examples with difficulty below a cap are eligible, and the cap rises over time.
• Mixture: examples from all levels are eligible, but the probabilities shift from easy toward hard.

The mixture approach is more forgiving. It keeps some hard examples early and some easy examples late, which avoids forgetting simple cases.

Step 4: keep some randomness

Pure sorting by difficulty can bias gradient updates, since consecutive batches look alike. Mix shuffling within each difficulty band, and keep a small share of random samples from the full set at all times.

Step 5: end on the real distribution

By the end of training, the model should see the true data distribution, including hard cases, in realistic proportions. Otherwise the model becomes calibrated to the easy end.

Why it can help: three views

Optimization view. The loss surface of a hard task has many poor local regions. Easy examples give a smoother surface in the early phase. The model finds a good basin, then continues to refine.

Signal to noise view. Easy examples tend to have clear patterns. Early gradients then point in consistent directions, and the model learns faster per step.

Representation view. Simple examples teach basic building blocks, such as edges in images or basic arithmetic relations in tables. Later examples compose them.

Why it sometimes does not help

If the dataset is large, clean and varied, and the model has ample capacity, order matters little. With modern large-scale training, the benefit is often small compared with data quality and quantity. Where curricula do shine is when data is scarce, noisy, long-range or synthetic with controllable difficulty.

An analogy for pacing

Think of weight training. If you begin with the heaviest weight, you risk injury and learn bad form. If you stay on light weights forever, you stop improving. A good plan raises the load slowly as strength grows, and it tracks how the body responds. An adaptive pacing function does the same for a model: it raises difficulty when the training loss shows the current level has been mastered.

Key concepts and terms

A worked example

A numeric walk-through

Suppose a classifier is trained for 10,000 steps on a pool where each example has a difficulty score between 0 and 1. A linear pacing function sets the difficulty cap as follows:

• cap(t) = 0.2 + 0.8 × (t / 6,000) for t up to 6,000,
• cap(t) = 1.0 afterward.

The last 4,000 steps use the full distribution, so the final model is tuned to the real data.

A simple curriculum sampler in Python

import numpy as np

class CurriculumSampler:
    def __init__(self, difficulty, total_steps, start=0.2, ramp_frac=0.6, seed=0):
        self.d = np.asarray(difficulty, dtype=float)
        self.total = total_steps
        self.start = start
        self.ramp = int(total_steps * ramp_frac)
        self.rng = np.random.default_rng(seed)
        self.order = np.argsort(self.d)  # easiest first

    def cap_fraction(self, step):
        if step >= self.ramp:
            return 1.0
        return self.start + (1.0 - self.start) * step / self.ramp

    def sample(self, step, batch_size, random_share=0.1):
        k = max(batch_size, int(len(self.order) * self.cap_fraction(step)))
        pool = self.order[:k]
        n_rand = int(batch_size * random_share)
        n_cur = batch_size - n_rand
        cur = self.rng.choice(pool, size=n_cur, replace=True)
        rnd = self.rng.choice(len(self.d), size=n_rand, replace=True)
        return np.concatenate([cur, rnd])

# difficulty could be the loss of a small proxy model, row count, noise level, ...
difficulty = np.random.rand(50_000)
sampler = CurriculumSampler(difficulty, total_steps=10_000)
for step in range(10_000):
    idx = sampler.sample(step, batch_size=256)
    # batch = dataset[idx]; loss = model(batch); loss.backward(); opt.step()

The sampler keeps ten percent of every batch random. That small share protects against drift and forgetting.

Self-paced weighting

def self_paced_weights(losses, lam):
    # include examples whose current loss is below a threshold lam
    return (losses < lam).float()

# during training
losses = per_example_loss(model, batch)
w = self_paced_weights(losses.detach(), lam=current_lambda(step))
loss = (w * losses).sum() / w.sum().clamp(min=1.0)

Here lam rises over time, so the model gradually accepts harder examples. A variant uses soft weights that decay smoothly with loss.

A synthetic data curriculum for tables

A practical example is a generator of synthetic tables. Each table is produced by sampling a structure and a data process. You control difficulty through generator settings.

curriculum:
  stages:
    - name: warmup
      steps: 0-20000
      rows: [20, 200]
      columns: [3, 10]
      noise: low
      missing_rate: 0.0
      interactions: pairwise
      class_count: [2, 3]
    - name: grow
      steps: 20000-80000
      rows: [50, 1000]
      columns: [5, 40]
      noise: medium
      missing_rate: [0.0, 0.1]
      interactions: nonlinear
      class_count: [2, 6]
    - name: full
      steps: 80000-200000
      rows: [50, 5000]
      columns: [3, 100]
      noise: [low, high]
      missing_rate: [0.0, 0.4]
      interactions: deep
      class_count: [2, 10]
      mixed_types: true

In the first stage, a model learns that a column can predict the label through a simple relation. In later stages, it learns to deal with noise, irrelevant columns, missing cells, mixed categorical and numeric columns and interactions between several columns.

Strengths and limitations

Strengths

• Faster early progress: clean, simple examples give consistent gradients.
• More stable training: avoids early divergence when examples are noisy or extreme.
• Better use of scarce data: helpful when data is limited.
• Natural fit for synthetic data: difficulty can be controlled directly by the generator.
• Compute savings: small inputs are cheap, so early stages cost less. Short sequences and small tables use less memory too, which pairs well with gradient checkpointing later when inputs grow.
• Interpretable: stages can be inspected and ablated one by one.

Limitations

• Difficulty is hard to define: a score that is easy for people may not be easy for a model.
• Extra design work: you need a measure, a schedule and tuning.
• Risk of bias: showing only easy cases early may teach shortcuts that later must be unlearned.
• Small or no gains at scale: with large clean data and big models, random order is often competitive.
• Forgetting: if easy cases vanish late in training, performance on them can drop.
• Evaluation difficulty: gains can be small relative to seed noise, so you need multiple runs to be sure.
• Interaction with other schedules: curricula combine with learning rate schedules, data mixtures and batch size schedules, and the effects can be entangled.

When it tends to help

How it compares with alternatives

Curriculum versus data cleaning

Cleaning removes bad data. A curriculum keeps all data but arranges it. If some examples are simply wrong, remove them. If they are right but difficult, order them.

Curriculum versus hard example mining

An easy-first curriculum and hard example mining can both be useful at different times. A common pattern is to start with a curriculum for stable early learning and then switch to emphasis on hard examples to polish the model.

Curriculum versus model growth

Some methods grow the model, for example by adding layers or widening, rather than growing the data. Data curricula are simpler because they require no architecture change.

Role in tabular foundation models

Tabular foundation models are pre-trained on many different tables so that they can predict on a new table through in-context learning, with no per-table training. Their raw inputs have row and column structure, mixed numeric and categorical types, missing values, and column names that are often absent or meaningless. Curriculum learning is a natural fit for building them. See /model/architecture and /model/research for the family of models.

Why tables suit curricula

The pre-training data for these models is often synthetic: tables sampled from random programs or random causal structures (see synthetic data generation). A generator exposes direct knobs for difficulty:

• number of rows and columns,
• number of informative versus irrelevant columns,
• depth and nonlinearity of the relations between columns and the target,
• amount of label noise and feature noise,
• fraction of missing cells and type of missingness,
• number of classes and class imbalance,
• share of categorical versus numeric columns, and the cardinality of the categories.

Because the generator is under your control, a curriculum is easy to define and easy to measure. You do not need to guess the difficulty of an example. You set it.

Stage by stage

1. Simple relations on small tables. The model learns the basic skill of in-context learning: look at labeled rows in the context and use them to predict a query row. Few columns, clean values, short contexts.
2. More columns and distractors. The model learns to ignore irrelevant columns and to weigh informative ones, without relying on column names.
3. Mixed types. Categorical columns, binary flags and numeric columns appear together. The model learns type-aware cell handling.
4. Noise and missing values. The model learns robustness, and it learns what a missing cell implies.
5. Interactions and nonlinear structure. Relations involving several columns appear.
6. Large contexts. Many rows and many columns. Memory and compute grow, so this stage is last. This is where memory-saving tools like gradient checkpointing and fused attention are needed.
7. Real tables. A mix of real data joins the synthetic mixture, so the model sees realistic distributions.

Row and column structure

Models that attend across cells, rows and columns scale in cost with table size, so a size-based curriculum has a direct compute benefit: early steps with small tables are cheap and let the model learn the basics while using a small fraction of the final compute per step. It also keeps early gradients less noisy, since small contexts produce more consistent in-context tasks.

Context length curricula

In-context learning depends on the number of labeled rows in the context. A model first trained on short contexts and later on long ones can learn to use more examples gradually. This mirrors the practice in language modeling of starting with short sequences and growing them. It also encourages the model to extrapolate to context sizes larger than the ones it saw most often, which is useful when a user supplies a table bigger than anything in pre-training.

Missing values and imputation

A curriculum on missingness starts with no missing cells, then adds a small rate, then higher rates and different mechanisms: cells missing at random, missing depending on other columns, and missing depending on their own value. Masked-cell objectives, in the spirit of masked cell modeling, can use a mask rate that rises over training. Low mask rates are easy to fill. High mask rates demand that the model use global structure. The same staging helps imputation networks.

Continual learning

When a model keeps learning from new families of tables, the order in which families arrive matters. A curriculum can plan the order, and replay of earlier stages protects against forgetting. See elastic weight consolidation for a related approach to reducing forgetting.

Anomaly detection

A model that learns what normal rows look like can be trained with a curriculum on anomaly strength: first obvious outliers, then subtle ones. This teaches the difference between a clear break from the pattern and a faint one. See anomaly detection for the task itself.

Sector and semantic classification

When a head classifies tables by domain or sector, a curriculum can begin with clearly separated categories and add similar, easily confused ones later. This is analogous to starting with cats versus cars and then moving to fine-grained breeds.

Dedup and diversity

A curriculum needs diverse material at each stage. If a stage repeats near identical tables, the model gains nothing from the later part of the stage. Removing duplicates, as covered in deduplication, keeps each stage informative.

Related models

Models such as TabPFN and TabICL use synthetic pre-training with in-context learning, and training schedules over table size and complexity are an important part of how such systems are built. Research summaries are on /model/research, and evaluation practice is on /model/benchmarks.

Practical guidance and best practices

Start with a baseline

Always train a baseline with random order first. Compare curriculum results against it using the same compute budget and several seeds. A curriculum that only matches the baseline is not worth the complexity.

Define difficulty you can control

Prefer difficulty knobs that you directly set, such as table size or noise level, over learned difficulty scores. They are easier to debug and explain.

Use gentle pacing

Increase difficulty gradually. Large jumps shock the model and can cause loss spikes. Smooth ramps are safer than steps.

Keep a floor of easy and random data

Even late in training, keep a small share of easy examples and a small share of uniform samples. This protects against forgetting and against drift from the target distribution.

End on the real distribution

The last phase should match the data the model will meet in use, in realistic proportions.

Watch per-stage metrics

Track validation loss on a fixed set for each difficulty level, not just overall. A curriculum can improve hard cases while quietly hurting easy ones.

levels = {"easy": val_easy, "medium": val_med, "hard": val_hard}
for name, ds in levels.items():
    print(step, name, evaluate(model, ds))

Make the schedule data driven when possible

Adaptive pacing uses a signal, such as the training loss on the current stage dropping below a target, to decide when to advance. This avoids wasting steps on mastered content and avoids advancing too early.

def advance(stage, recent_loss, target, patience_met):
    if recent_loss < target and patience_met:
        return min(stage + 1, MAX_STAGE)
    return stage

Combine with other schedules thoughtfully

• Learning rate warm-up and decay run alongside. Do not change both heavily at once when debugging.
• Batch size can grow with context size to keep memory under control.
• Mixed precision lowers the memory of larger stages. See mixed precision.

Document the schedule

Write the stages, the boundaries, the generator settings and the pacing rule in a config file. Curricula are easy to forget and hard to reproduce without a record.

Run ablations

Remove one stage or change one knob at a time. If removing a stage does not hurt, drop it.

Practical checklist

Common mistakes

• Skipping the baseline. Without a random-order baseline, you cannot know the curriculum helped.
• Trusting human intuition about difficulty. What looks easy to a person may not be easy for a model.
• Sorting strictly and never shuffling. Pure sorted order creates correlated batches and biased gradients.
• Never returning to the full distribution. The final model then underperforms on hard real cases.
• Advancing too fast. Big jumps in difficulty cause loss spikes.
• Advancing too slowly. Spending most of training on easy data wastes compute.
• Removing easy data later. The model forgets simple cases.
• Confusing curriculum with learning rate warm-up. They are different tools. One orders data, the other adjusts step size.
• Changing many knobs at once. If size, noise and missingness all change in one jump, you cannot tell what helped.
• Using a learned difficulty score from a weak proxy. The proxy's idea of hard may not match the final model's.
• Ignoring seed variance. Small gains need several seeds to confirm.
• Letting the curriculum leak. In table models, make sure synthetic stage settings do not overlap in a way that gives away evaluation tables. Check overlap with held-out benchmarks.

Frequently asked questions

What is curriculum learning in simple terms?

It is a way to train a model by showing it easy examples first and harder ones later, much like a school syllabus. The goal is steadier training and often faster progress. The schedule decides when harder material is added.

Does curriculum learning always improve results?

No. Gains are common when data is noisy or scarce, tasks are hard or data is synthetic with controllable difficulty. With large, clean datasets and large models, random order often performs about as well. Always compare against a random-order baseline with several seeds.

How do you decide which examples are easy?

You can use a hand designed rule, such as length or size, a score from a small proxy model, or the current loss of the model being trained. For synthetic data, the best option is to set difficulty directly through generator settings, such as the number of columns or the noise level.

What is self-paced learning?

Self-paced learning is a form of curriculum where the model chooses its own examples. It includes examples whose loss is below a threshold and raises the threshold over time, so harder examples are added as the model improves. It needs no external difficulty score.

Is curriculum learning the same as transfer learning?

No. Transfer learning starts from weights learned on another task and adapts them. Curriculum learning orders the data within a training run. They can be combined, for example by pre-training on simple synthetic tables and then fine tuning on a real one.

How is a curriculum used for tabular foundation models?

Pre-training data for these models is often synthetic, so difficulty can be controlled directly. Early stages use small, clean tables with simple relations. Later stages add more rows and columns, noise, missing values, mixed types and complex interactions, followed by real tables. This keeps early training stable and cheap and builds skills step by step.

Can a curriculum cause forgetting?

Yes, if easy examples disappear from later stages. Keep a floor of easy and random samples in every phase, and track validation metrics per difficulty level so you notice any drop early.

What is the difference between a curriculum and a learning rate schedule?

A curriculum changes which data the model sees over time, while a learning rate schedule changes the size of the update steps. Both vary over training and are often used together, but they are separate controls and should be tuned and ablated separately.

What is anti-curriculum?

Anti-curriculum means presenting hard examples before easy ones. It is rarely better than an easy-first order, but studies include it as a control to show that the order really matters and that gains are not just from seeing the same data.

How long should each stage last?

There is no universal answer. A common approach is to ramp difficulty over the first half to two thirds of training and spend the rest on the full distribution. Adaptive pacing, which advances when training loss reaches a target, avoids guessing, and ablations show whether a stage is needed at all.

Glossary

• Adaptive pacing: raising difficulty when a performance signal says the current level is mastered.
• Anti-curriculum: training with hard examples first.
• Continuation method: solving a hard optimization problem by moving from an easier version toward the real one.
• Curriculum: a schedule that orders training data by difficulty.
• Difficulty measure: a score of how hard an example or source is.
• Forgetting: loss of earlier skills during later training.
• Hard example mining: emphasizing high-loss examples.
• In-context learning: making predictions from labeled examples given in the input, with no weight updates.
• Pacing function: the rule that sets how fast difficulty rises.
• Proxy model: a small model used to estimate difficulty.
• Self-paced learning: letting the model's loss decide which examples to include.
• Shaping: rewarding successive approximations to a goal behavior, an idea from animal training.
• Synthetic data: data produced by a generator rather than collected from the world.
• Warm-up: a period of small learning rates at the start of training.

Summary

Curriculum learning orders or schedules training data from simple to complex, using a difficulty measure and a pacing function. It can make training more stable, speed early progress and help most when data is scarce, noisy or synthetic with controllable complexity. Its limits are real: difficulty is hard to define, gains shrink with large clean data and big models, and poor schedules can cause bias or forgetting. Always compare with a random-order baseline, keep some easy and random samples at every stage, and finish on the real distribution.

For tabular foundation models, a curriculum fits naturally. Synthetic table generators expose clear knobs for rows, columns, noise, missing values, mixed types and interactions, so training can move from small, clean tables to large, messy ones, and then to real tables. This keeps early steps cheap, builds skills in order and supports related tasks such as imputation, anomaly detection and sector classification. See also deduplication, gradient checkpointing, synthetic data generation, /model/architecture, /model/benchmarks and /model/research.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: