Curriculum scheduling

Overview

Curriculum scheduling is the practice of planning when each kind of data enters training. Instead of shuffling everything into one pool and sampling at random, you decide the order or the mixture over time. The usual pattern moves from simple examples to complex ones. Early steps use clean, short or easy data so the model learns basic structure. Later steps add harder, longer or noisier data so the model learns the details and the edge cases.

The goal is practical. A good schedule can make pre-training more stable, reduce wasted compute, and improve the final quality. A poor schedule can slow learning or hide a weakness until late. This article explains the idea, the main methods, a worked example, and how scheduling applies to models that learn from raw tables. For the broader idea see Curriculum learning.

What it is

Curriculum scheduling is a training-time plan. It answers three questions:

• Which data should the model see first?

• How fast should harder data be introduced?

• When should the mixture of sources change?

The school analogy

A child does not start with calculus. She learns counting, then addition, then fractions, then algebra. Each stage builds on the last. Teachers also revisit older material so skills do not fade. Curriculum scheduling applies the same logic to machine learning. The model is the student, the data is the textbook, and the schedule is the lesson plan.

Curriculum versus random sampling

Standard training samples each batch uniformly from the dataset. This is simple and works well in many cases. It treats every example as equally useful at every moment. Curriculum scheduling relaxes that assumption. It says that the usefulness of an example depends on what the model already knows. An example that is too hard early on may only add noise. An example that is too easy late in training adds little.

Three kinds of schedule

Real systems often combine all three.

Static and dynamic schedules

A static schedule is fixed before training. You decide the stages and their lengths in advance. A dynamic schedule reacts to the model. It measures loss or accuracy and then chooses what comes next. Dynamic methods include self-paced learning, where the model picks examples it finds easy, and teacher-guided methods, where a second model chooses examples.

What a schedule is not

It is not the same as the learning rate schedule, although the two are related and are often tuned together. It is also not data cleaning, though cleaning is usually done first. A schedule does not remove data. It only changes the timing and the weighting.

Where it came from

The idea has roots in psychology and education, where the benefits of ordered teaching were studied long before neural networks. In machine learning, a well-known paper on curriculum learning in the late 2000s argued that training on easier examples first can speed learning and lead to better solutions for hard non-convex problems. A related idea, self-paced learning, proposed letting the learner choose its own easy examples and gradually widen the set.

Early use

Early work focused on small models and toy tasks, such as shape recognition and simple language tasks. The common finding was that ordering helped most when the problem was hard to optimize or when the data was noisy. In reinforcement learning, curricula of increasingly hard environments became a standard tool.

The large model era

As language models grew, the question changed from "does order matter" to "how should we mix many sources over a very long run." Practitioners found that the mixture of web text, books, code and other sources, and the order in which they are introduced, can affect the result. Many training reports describe staged training, with a main phase on broad data and a late phase on cleaner or more specialized data, often together with a lower learning rate. Details differ between projects and are not always published, so it is best to treat specific recipes as case-by-case.

Related lines of work

Several nearby ideas share the same spirit:

• Active learning picks which examples to label next. See Active learning.

• Continued pre-training adds a new stage after the main run. See Continued pre-training.

• Data mixing and domain reweighting tune source proportions.

• Sequence length warm-up starts with short sequences and grows them.

• Synthetic data generators can produce tasks with a controllable level of difficulty. See Synthetic data generation.

How it works

A curriculum has four parts: a way to measure difficulty, a way to split or score data, a pacing rule, and a mixing rule. This section goes through each in order.

Step 1: Define what "easy" means

Difficulty is a modeling choice. Common definitions are:

• Length: short inputs are easier than long ones.

• Complexity: fewer features, fewer classes or simpler relationships are easier.

• Noise: clean labels are easier than noisy ones.

• Loss under a reference model: examples with low loss under a small model are easy.

• Domain familiarity: data close to what the model has already seen is easier.

• Human labels: an expert marks tasks as basic or advanced.

Analogy: sorting hiking trails. You could rank by distance, by climb, by terrain or by a ranger's rating. Different rankings give different training plans, and none is universally right.

Step 2: Score and bucket the data

Once you pick a definition, compute a score for every example or every data source. Then group the data into buckets, such as five levels. For generated data, you may control difficulty at the source, for instance by setting the number of columns, the noise level or the depth of the function that creates the labels.

Step 3: Choose a pacing function

A pacing function says how much of the difficulty range is available at each step. Common shapes are:

Analogy: a gym trainer increasing weights. She can add a fixed amount each week, raise it smoothly, or have deload weeks.

Step 4: Decide how to mix sources

Often the choice is not "show easy first" but "which share of the batch comes from each source." A mixture schedule might start with 80 percent synthetic and 20 percent real, then slide to 20 and 80. Another might keep a small share of every level throughout, so that nothing is forgotten.

A common safety rule is to never fully remove earlier data. Keep a small replay share. This guards against forgetting what the model learned in earlier stages.

Step 5: Run and monitor

During training, track loss and evaluation scores per bucket and per source. If a newly introduced bucket causes a sharp loss spike, the step was too big. If progress on a bucket stalls, the schedule may be moving on too soon.

Step 6: Anneal at the end

Many recipes end with a short phase of high-quality data and a decaying learning rate. The idea is that the final weights should be shaped by the cleanest, most representative examples. This phase is sometimes called annealing or cooldown.

Dynamic and teacher-driven methods

In self-paced learning, the loss of each example decides whether it is included. Examples with loss below a threshold are used, and the threshold rises over time. In teacher-student curricula, a second policy learns to pick the next task so that the student improves fastest. These methods adapt to the model, but they add complexity and need careful testing to avoid feedback loops.

Why it can help

There are a few plausible reasons, and the weight of each depends on the problem:

• Easier gradients early reduce instability and wasted steps.

• Simple patterns learned first become building blocks for complex ones.

• Cleaner early data keeps noisy labels from setting bad habits.

• Short sequences are cheaper, which saves compute in the early phase.

• A late phase on target-like data aligns the model with the final use.

Why it may not help

Randomly shuffled data is a strong baseline, and in many large-scale settings a curriculum shows small or mixed gains. Reported benefits are sensitive to the difficulty measure and to the pacing. It is wise to test a curriculum against a well-tuned random baseline and to treat the result as an experiment, not a rule.

Key concepts and terms

A worked example

This example builds a small schedule for a model trained on generated tables. The numbers are made up for teaching and are not results.

The setup

A generator produces tables with a controllable number of columns, noise level and function complexity. We define a difficulty level from 1 to 5.

The plan

The run has 100,000 steps. We use a step schedule with a replay share.

Each row sums to 100 percent. Level 1 never disappears, which is the replay share.

Code for a mixture sampler

import random

STAGES = [
    (0,      20_000, [0.70, 0.30, 0.00, 0.00, 0.00]),
    (20_000, 50_000, [0.20, 0.40, 0.40, 0.00, 0.00]),
    (50_000, 80_000, [0.10, 0.20, 0.30, 0.40, 0.00]),
    (80_000, 100_000,[0.10, 0.10, 0.20, 0.30, 0.30]),
]

def weights_at(step):
    for start, end, w in STAGES:
        if start <= step < end:
            return w
    return STAGES[-1][2]

def sample_level(step, rng=random):
    w = weights_at(step)
    return rng.choices(range(1, 6), weights=w, k=1)[0]

def make_batch(step, batch_size, generator):
    return [generator(level=sample_level(step)) for _ in range(batch_size)]

A smooth version

Instead of fixed stages, you can blend weights. Here the share of harder levels grows linearly:

def smooth_weights(step, total=100_000):
    t = min(step / total, 1.0)
    raw = [max(0.05, 1.0 - 1.5 * abs(t * 4 - (lvl - 1)) / 2) for lvl in range(1, 6)]
    s = sum(raw)
    return [r / s for r in raw]

The floor of 0.05 keeps every level alive. The function peaks on level 1 at the start and on level 5 near the end.

Reading the result

You would compare three runs: random mixture, the stage plan, and the smooth plan. For each run, track validation loss on tables from every level, not only the hardest. A useful curriculum should improve the hard levels without hurting the easy ones. If the plan helps only the final level but hurts level 1, the replay share is too small.

A numeric check of the mixture

At step 60,000, the weights are 0.10, 0.20, 0.30, 0.40, 0.00. With a batch of 64 tables, the expected counts are 6.4, 12.8, 19.2, 25.6 and 0. Round these to 6, 13, 19, 26 and 0, which sum to 64. When rounding, always check that the counts sum to the batch size.

Using difficulty from a reference model

For real data, you may not have a generator that sets difficulty. A simple recipe is:

1. Train a small model for a short time on the whole dataset.

2. Compute the loss of each example under that model.

3. Rank examples by loss and split into quantile buckets.

4. Use the buckets as levels in the schedule.

This approach is cheap, but it ties difficulty to one small model. Examples that are hard for it may not be hard for a larger one.

Strengths and limitations

Strengths

• Can stabilize early training, especially for large models and long runs.

• Can cut compute by using short or simple inputs in the early phase.

• Gives a clear way to inject domain knowledge about what is easy and what is hard.

• Works well with generators that have a difficulty knob.

• Helps with source mixing, which is hard to tune by hand.

• Late-stage quality data can sharpen the final model.

• Easy to log and audit, since the plan is explicit.

Limitations

• Gains are not guaranteed and can be small compared with good tuning of other settings.

• Difficulty is hard to define, and a poor definition can hurt.

• Extra hyperparameters: number of stages, pacing, weights.

• Risk of forgetting early data if the replay share is too low.

• Static plans do not adapt to a model that learns faster or slower than expected.

• Dynamic plans can create feedback loops, where the model only sees what it already finds easy.

• Hard to compare fairly. Each project needs its own baseline.

• The schedule interacts with the learning rate, batch size and regularization, so changing one may require retuning the others.

How it compares with alternatives

Curriculum scheduling is often used together with the others. For example, a run might use a mixture schedule, then hard example mining in a late stage, and finish with continued pre-training for a specific domain.

Role in tabular foundation models

Tabular foundation models are trained to handle many different tables with one set of weights. They must cope with changing numbers of rows and columns, mixed types, missing values and, sometimes, no column names. Scheduling matters here because the training distribution is very wide, and the order in which tasks appear can shape what the model learns first.

Pre-training on synthetic tables

Many tabular foundation models, such as TabPFN, are pre-trained mostly on tables drawn from generators. The generators sample the structure of the table: how many columns, which are categorical, how the target depends on features, how much noise there is. Because the generator has explicit knobs, difficulty can be set directly. This makes tabular pre-training a natural place for a curriculum:

• Start with few columns, few rows and simple functions.

• Increase width, depth of dependencies and noise.

• Add missing values and mixed types after the model has learned the basic mapping.

• Keep a share of easy tables throughout, so simple cases stay accurate.

See Synthetic data generation for how such tables are produced.

Row and column size schedules

The cost of attention grows with the number of rows and columns. A sequence length schedule starts with small tables, which are cheap, and later adds large ones. This saves compute in the early phase and trains the model to use the full range at the end. It mirrors the idea of growing the context length in language models. For architecture background see /model/architecture.

Mixed types and missing values

Mixed types and missing values add difficulty. A curriculum can introduce numeric features first, then categorical features with growing cardinality, then dates and text-derived fields. Missingness can start at a low rate and rise. If the model must do imputation, the schedule can move from random missing cells to harder, structured patterns of missingness. See Imputation networks.

No column names

If the model is meant to be schema-agnostic, it must learn from the structure of values alone. A curriculum can begin with tables whose columns have clear statistical signatures, then move to tables with weaker signals and more look-alike columns. Over time the model learns not to rely on names.

Real versus synthetic data

A mixture schedule can start with synthetic data, which is abundant and clean, and add real tables later, which are scarce and messy. The late real-data phase helps the model adapt to the quirks of actual business and science data. The risk is that a tiny real set is overfit, so it should be mixed with a replay share of synthetic tables. For published comparisons see /model/benchmarks and /model/research.

In-context learning

In-context learning needs the model to read labeled rows and predict for a query row, without weight updates. A curriculum can grow the number of in-context examples over time, from a handful to many hundreds. Early on, the model learns to copy simple patterns from a few rows. Later it learns to average over many rows and weigh them by similarity.

Continual learning and forgetting

When new tasks or domains are added after the main run, the model can forget older skills. Scheduling helps by blending old and new data in the same batches and by lowering the learning rate for the new stage. See Continued pre-training. Replay is the most common safeguard.

Anomaly detection and sector classification

For anomaly detection, a curriculum can begin with clear, large anomalies and move to subtle ones. For semantic or sector classification, it can begin with coarse labels and later split them into fine labels, which is a hierarchical curriculum. In both cases the principle is the same: build easy skills first, then refine. See Anomaly detection.

Practical guidance and best practices

Start with a baseline

Always run a well-tuned random shuffle baseline first. If a curriculum does not beat it by a margin larger than run-to-run noise, do not adopt it. Repeat each configuration with several seeds.

Keep the design simple

• Use two to five stages, not twenty.

• Use a step schedule first. Move to smooth pacing only if it helps.

• Keep a replay share of at least 5 to 10 percent for earlier levels.

• Change one thing at a time: difficulty definition, pacing or mixture.

Choose the difficulty measure with care

• Prefer a measure you can compute cheaply and explain.

• Check that the measure correlates with actual model loss.

• Avoid measures that leak the label.

• Revisit the measure when the model grows or the data changes.

Monitor by slice

Report validation metrics per level, per source and per table size. A single average hides regressions. Plot the curves against steps and mark the stage changes.

Coordinate with other schedules

Learning rate, batch size and curriculum affect each other. A common pattern is warm-up, a long stable phase, and a decay at the end. Place the high-quality data phase in the decay window. If you change the learning rate schedule, test the curriculum again.

Document the plan

Write the schedule in a config file and store it with the model. Include stage boundaries, weights, the difficulty definition and the random seeds. This makes the run reproducible and the results interpretable.

curriculum:
  name: staged_tables_v1
  stages:
    - steps: [0, 20000]
      mix: {level1: 0.7, level2: 0.3}
    - steps: [20000, 50000]
      mix: {level1: 0.2, level2: 0.4, level3: 0.4}
    - steps: [50000, 100000]
      mix: {level1: 0.1, level2: 0.2, level3: 0.3, level4: 0.4}
  replay_floor: 0.05
  seeds: [1, 2, 3]

Be ready to roll back

If a stage causes a loss spike or divergence, return to a checkpoint from before the stage, use a smaller step in mixture change, and try again. Keep checkpoints at each stage boundary.

Common mistakes

1. Assuming that easy first always wins, without testing against a random baseline.

2. Using a difficulty measure that does not match real difficulty.

3. Dropping early data completely and then seeing forgetting.

4. Making stages too short, so the model never settles.

5. Making stages too long, so late data has little effect.

6. Changing the learning rate schedule and the curriculum in the same experiment.

7. Evaluating only on the hardest slice and missing regressions on easy ones.

8. Letting a dynamic curriculum chase easy examples forever.

9. Forgetting to shuffle within a stage, which creates ordering artifacts.

10. Introducing noisy or mislabeled data early and letting it set bad habits.

11. Not logging the schedule, which makes results impossible to reproduce.

12. Overfitting a small high-quality set in the final stage because it is repeated too many times.

Frequently asked questions

What is curriculum scheduling in machine learning?

Curriculum scheduling is the planning of when each kind of data enters training. Instead of sampling all examples at random from the start, you order them or change their mixture over time, usually moving from simple to complex. The aim is to make training more stable and efficient and to improve the final model.

Is curriculum scheduling the same as curriculum learning?

They are closely related. Curriculum learning is the broader idea of teaching a model in a meaningful order. Curriculum scheduling stresses the practical plan: stage boundaries, mixture weights and pacing rules that say exactly what the model sees at each step. In many papers and projects the two terms are used almost interchangeably.

Does a curriculum always improve results?

No. Random shuffling is a strong baseline, and many studies find only small or mixed gains. A curriculum tends to help when the problem is hard to optimize, when data quality varies a lot, or when sources differ widely in size and difficulty. The only reliable way to know is to test it against a tuned baseline with several seeds.

How do I measure difficulty?

You can use simple proxies such as length, number of features or noise level, or you can use the loss of a small reference model. For generated data, difficulty can be set by the generator's parameters. Whatever you choose, check that it relates to real model loss and that it does not leak the label.

What is a pacing function?

A pacing function controls how quickly harder data becomes available. It can be a step function that jumps at fixed points, a smooth ramp, or a cycle that returns to easy data. The right shape depends on the task, and step schedules are usually the easiest to start with and to debug.

How does scheduling prevent forgetting?

By keeping a replay share. Even in a late stage, a small portion of each batch comes from earlier levels or sources. This keeps old skills active while the model learns new ones. Lowering the learning rate in later stages also reduces the risk of overwriting what was learned before.

How is curriculum scheduling used for tabular foundation models?

Tabular foundation models pre-trained on generated tables can use generator knobs, such as the number of columns, noise and function complexity, as difficulty levels. A schedule might begin with small, clean tables and grow toward wide, noisy tables with missing values and mixed types. It can also blend synthetic and real tables over time.

What is the difference between a data mixture and a curriculum?

A data mixture is a fixed set of proportions across sources. A curriculum changes those proportions or the available difficulty over time. A constant mixture is a special case of a curriculum with a single stage. Many modern training recipes use staged mixtures, which sit between the two.

Should the last stage use the best data?

Often yes. A common practice is to finish with a short phase of clean, representative data while the learning rate decays. The idea is that the final weights should reflect the data most similar to real use. Be careful not to repeat a small set too many times, which can lead to overfitting.

Can a curriculum be learned automatically?

Yes, to a degree. Self-paced learning lets the model choose examples based on its loss. Teacher-student methods train a second policy to pick tasks. These approaches adapt to the model but add complexity and can create feedback loops, so they need careful monitoring.

Glossary

• Anneal: to decay the learning rate, often while using high-quality data in the final stage.

• Anti-curriculum: ordering from hard to easy.

• Bucket: a group of examples with similar difficulty.

• Catastrophic forgetting: loss of earlier skills during later training.

• Curriculum: a plan for the order or mixture of training data.

• Difficulty score: a value that says how hard an example is.

• Data mixture: the proportions of each data source.

• Pacing function: the rule that sets how fast difficulty increases.

• Replay: reusing earlier data in later stages.

• Self-paced learning: letting the learner choose which examples to use.

• Stage: a block of steps with a fixed data plan.

• Warm-up: a gradual start to a quantity such as the learning rate.

Summary

Curriculum scheduling plans when each kind of data enters training. It uses difficulty scores, pacing rules and mixture weights to move a model from simple examples to complex ones, usually with a replay share and a clean final phase. It can improve stability, save compute and sharpen the final model, but it must be tested against a strong random baseline, because gains are not guaranteed.

For tabular foundation models, scheduling fits naturally with synthetic pre-training, where difficulty can be set by the generator: table size, noise, function complexity, mixed types and missing values. It also helps blend synthetic and real tables, grow the number of in-context examples, and add new domains without forgetting. Keep the plan simple, log it, evaluate by slice, and treat each curriculum as an experiment. For related reading see Curriculum learning, Continued pre-training, Active learning, TabPFN, /model/research and /model/benchmarks.

Related

It belongs to Foundation models from scratch in my stack, under Foundation models and applied ML. Others in the same group: