Neural ODE

Overview

A neural ODE is a neural network whose hidden state changes according to an ordinary differential equation (ODE). Instead of stacking a fixed number of layers, you define a small network that gives the rate of change of the hidden state. A numerical solver then integrates that rate from a start time to an end time. The state at the end time is the output.

This idea gives you continuous depth and continuous time. Depth becomes a quantity the solver chooses, not a number you fix up front. Time becomes a real axis, so you can query the model at any moment, including moments between observations. That makes neural ODEs a natural fit for irregularly sampled time series, physical systems and density models. Their cost is speed: each forward pass runs a solver, and that is slower than a plain stack of layers. See also Autoencoders, Normalizing flows and PINN for related ideas.

What it is

The core definition

A standard residual block updates a hidden vector like this: new state equals old state plus a learned change. Written out, h(t+1) = h(t) + f(h(t), t). This is exactly one step of the Euler method for solving a differential equation.

A neural ODE takes the limit of many tiny steps. It writes the update as dh/dt = f(h(t), t, theta). Here f is a neural network with parameters theta. Given the starting state h(0), the output is h(T), found by integrating the equation.

An everyday picture

Think of a leaf floating on a river. The river has a current that depends on where the leaf is. The neural network is the map of the current. You do not store a list of positions. You store the map, and you can drop a leaf anywhere and follow it for as long as you like.

An ordinary deep network is like moving the leaf in ten fixed jumps. A neural ODE lets the leaf drift smoothly, and the solver decides how many small steps it needs along the way.

The three parts

• A vector field network f that says how the state should change.
• An ODE solver that integrates the field over time.
• A way to compute gradients through the solver, often the adjoint method.

What it is not

• It is not a new loss function or a training trick. It is a model family.
• It is not limited to time series. It can also act as a layer in image or tabular models.
• It is not always more accurate than a regular network. The gains are in structure, memory use and handling of irregular time.

Where it came from

Roots in residual networks

Residual networks, introduced in 2015, showed that adding a learned update to the input makes very deep models easy to train. Researchers soon noticed that this form looks like a discretised ODE. Several papers around 2017 explored this link and treated deep networks as numerical schemes. See ResNet for the background.

The 2018 paper

The key paper is "Neural Ordinary Differential Equations", published in 2018 at a major machine learning conference, where it received a best paper award. It showed how to use black-box ODE solvers inside a network and how to train with the adjoint method. It also introduced continuous normalizing flows and latent-variable models for time series.

Later developments

• Continuous normalizing flows were refined with efficient trace estimators in a 2018 paper known as FFJORD.
• Augmented neural ODEs, from 2019, added extra dimensions so the flow could represent functions that plain flows cannot.
• Latent ODE models and ODE-RNNs, also from 2019, handled irregularly sampled data.
• Neural controlled differential equations, from 2020, let the hidden state respond to an input path, which is a better fit for streaming data.
• Work on stiff systems, regularisation of solver cost and faster solvers continued in the years after.

How it works

Step 1: define the vector field

You pick a small network f. It takes the current state and time and returns a vector the same size as the state. Because the output has the same size as the input, the dynamics are well defined.

Common choices are two or three fully connected layers with smooth activations such as tanh or softplus. Smooth activations help the solver, because the field has no sharp corners.

Step 2: choose a solver

An ODE solver steps from time 0 to time T. Simple solvers such as Euler use fixed steps. Higher-order solvers such as Runge-Kutta methods estimate the error of each step and adapt the step size.

• Euler: cheap, first-order, can drift.
• Midpoint and RK4: fixed step, higher accuracy per step.
• Dormand-Prince (often called dopri5): adaptive, the usual default.
• Implicit solvers: better for stiff problems, but each step is more costly.

Think of walking down a hill in fog. On a gentle slope you take long strides. On a steep, twisty path you take short careful steps. An adaptive solver does the same.

Step 3: integrate

The solver calls f many times. Each call is called a function evaluation. The count of such calls, known as NFE, is the main cost measure. A trained model may need more evaluations as its dynamics become more complex.

Step 4: read the output

The final state h(T) feeds a head, such as a linear layer for classification or regression. For time series you can read the state at many requested times, since the solver can stop at each one.

Step 5: compute gradients

Backpropagating through every solver step stores all intermediate states. That uses memory in proportion to the number of steps. The adjoint method avoids this.

The adjoint method solves a second ODE backwards in time. This second ODE carries the gradient of the loss with respect to the state. Memory use stays roughly constant in depth. The trade is extra compute and some numerical error, since the backward pass re-derives states instead of reading stored ones.

An analogy: to find out how an early decision changed the final result, you can keep a full diary of the journey, or you can walk the path backwards and reconstruct each step. The diary uses paper. The walk uses time.

Step 6: update parameters

With gradients in hand, a normal optimiser such as Adam updates theta. Training then loops like any other deep model.

A discrete view for comparison

Key concepts and terms

Why the flow map is invertible

Under mild smoothness conditions, ODE trajectories cannot cross. This means the map from h(0) to h(T) is one to one. That is a useful fact for density modelling, but it also limits what the model can represent in the original space. For example, a plain flow in two dimensions cannot swap two points that lie on opposite sides of a circle without crossing paths. Adding extra dimensions, as in the augmented model, gives the paths room to go around.

A worked example

A tiny numeric walk-through

Take a one-dimensional field f(h) = -h. The exact solution of dh/dt = -h is h(t) = h(0) * exp(-t). Start at h(0) = 1 and integrate to T = 1.

With Euler and a step of 0.5:

• Step 1: h = 1 + 0.5 * (-1) = 0.5
• Step 2: h = 0.5 + 0.5 * (-0.5) = 0.25

The exact value is exp(-1), about 0.368. Euler gives 0.25, which is off. With a step of 0.25 you get 0.75^4, about 0.316. With a step of 0.1 you get 0.9^10, about 0.349. The error shrinks as steps get smaller. A higher-order solver reaches a good answer in far fewer steps.

This is the whole story in miniature. The network learns f. The solver turns f into a trajectory. The accuracy and cost depend on the solver.

Training a neural ODE in Python

The following sketch uses PyTorch and the torchdiffeq package. It learns a vector field for a two-dimensional spiral from noisy observations.

import torch
import torch.nn as nn
from torchdiffeq import odeint_adjoint as odeint

class Field(nn.Module):
    def __init__(self, dim=2, hidden=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim, hidden), nn.Tanh(),
            nn.Linear(hidden, hidden), nn.Tanh(),
            nn.Linear(hidden, dim),
        )

    def forward(self, t, h):
        # t is the current time, h is the state
        return self.net(h)

field = Field()
opt = torch.optim.Adam(field.parameters(), lr=1e-3)

t = torch.linspace(0.0, 3.0, 40)          # observation times
h0 = torch.tensor([[2.0, 0.0]])            # start state
true = make_spiral(t)                      # placeholder data helper, shape (40, 1, 2)

for step in range(500):
    opt.zero_grad()
    pred = odeint(field, h0, t, rtol=1e-4, atol=1e-5, method="dopri5")
    loss = ((pred - true) ** 2).mean()
    loss.backward()
    opt.step()
    if step % 100 == 0:
        print(step, float(loss))

What to watch while it trains

• Print the NFE every few steps. If it climbs fast, the dynamics are getting stiff.
• Start with loose tolerances and tighten them later.
• Use smooth activations. ReLU can create corners that make adaptive steps tiny.
• Keep weights small at the start so the field begins near zero.

Using a neural ODE as a layer

You can place the integrator between an encoder and a classifier. The encoder maps the input to a state. The ODE block moves the state for time T. The head reads the result.

class ODEBlock(nn.Module):
    def __init__(self, field, T=1.0):
        super().__init__()
        self.field = field
        self.register_buffer("t", torch.tensor([0.0, T]))

    def forward(self, x):
        return odeint(self.field, x, self.t)[-1]

model = nn.Sequential(
    nn.Linear(20, 32),
    ODEBlock(Field(dim=32, hidden=64)),
    nn.Linear(32, 3),
)

This model has one shared field instead of many separate hidden layers. It often has fewer parameters than a deep stack of the same width.

A second walk-through: stiffness

Consider dh/dt = -50 * h with h(0) = 1. The exact answer decays very fast.

With Euler and a step of 0.05, each step multiplies the state by 1 - 50 * 0.05 = -1.5. The value flips sign and grows. The solver has become unstable, even though the true answer shrinks to zero.

With a step of 0.01, each step multiplies by 0.5, and the result decays as it should. So a fast time scale forces a small step.

This is what stiffness means in practice:

• The system has a fast part and a slow part.
• The fast part forces tiny steps for stability.
• The slow part is what you actually care about.

An adaptive solver will cut its step size and your training slows down. An implicit solver can take large steps here at a higher cost per step.

Variants and relatives

Latent ODE and ODE-RNN

An ODE-RNN reads observations one by one. Between observations, the hidden state follows an ODE. At each observation, a recurrent cell updates the state. A latent ODE goes further: it encodes the sequence into a starting state, runs the ODE, and decodes.

Neural controlled differential equations

A controlled differential equation lets the state respond to the derivative of an input path. The input is interpolated into a continuous path first, for example with cubic splines. The model then processes the path as a whole. This handles streaming, irregular and partly missing data in a principled way.

Continuous normalizing flows

Here the ODE moves points from a simple distribution to a complex one. The change in log density follows its own equation that involves the trace of the field's Jacobian. Trace estimators make this cheap enough to use on larger problems.

Augmented and second-order variants

• Augmented: add extra zero-filled dimensions to the state.
• Second order: model position and velocity, which suits mechanical systems.
• Hamiltonian and Lagrangian forms: build in energy conservation.

Neural SDEs

A stochastic differential equation adds random noise to the drift. Neural SDEs model uncertainty and can act as generative models for time series.

Implementation checklist

• Pick the state dimension first. Everything else depends on it.
• Decide whether time enters the field or not.
• Choose a fixed-step solver for the first run.
• Check that loss decreases on a tiny dataset before scaling.
• Compare adjoint gradients with direct backpropagation on a small case.
• Log NFE per epoch.
• Move to an adaptive solver, then tighten tolerance step by step.
• Add a baseline model that has no ODE.

Strengths and limitations

Strengths

• Handles irregular time naturally, since you can query any time point.
• Constant memory in depth when trained with the adjoint method.
• Parameter efficient, because the field is shared across depth.
• Smooth, invertible trajectories help in density estimation.
• Adaptive compute: simple inputs can need fewer evaluations.
• Links to physics. Prior knowledge can be built into the field.
• Works with missing time points without padding or resampling.

Limitations

• Slow. Each forward and backward pass calls the field many times.
• Cost varies per input, which complicates batching.
• Training can become unstable if the dynamics grow stiff.
• Trajectories cannot cross, which limits expressiveness without augmentation.
• Adjoint gradients can be slightly inexact.
• Fewer ready-made tricks than standard deep learning. Tuning solver tolerance is an extra task.
• On regular, dense data, a plain recurrent or attention model is often as good and much faster.

When the limits matter most

The speed cost matters in large-scale training. A model that takes 20 function evaluations per sample is roughly 20 times the cost of one layer. For very large tables or long sequences this adds up. Regularisation that penalises solver work can reduce the cost, and fixed-step solvers give predictable timing.

How it compares with alternatives

Neural ODE and state-space models

Both are continuous-time models. A linear state-space model, such as S4, has a fixed linear field and discretises it with a step size. A neural ODE uses a nonlinear learned field and a general solver. The first is easier to run in parallel. The second is more flexible but slower.

Neural ODE and physics-informed networks

A physics-informed network uses a known equation as a training constraint. A neural ODE learns an unknown equation from data. They can be combined: you know part of the field and let a network learn the rest.

Role in tabular foundation models

Why this matters for tables

Tabular foundation models learn from raw tables: rows, columns, mixed types, missing cells and often no useful column names. A neural ODE is not a core part of most of them. It is still relevant in several concrete ways, and it is helpful to be honest about where it fits and where it does not.

Row and column structure

A table has no natural order of columns and no depth axis. A neural ODE has a time axis. To use one on a table you must decide what "time" means. Two options are common:

• Treat depth as time, and let a shared field refine the row embedding. This is the continuous version of a residual stack.
• Treat a real timestamp column as the time axis, and model each entity as a trajectory. This suits tables of repeated measurements, such as sensor logs or patient visits.

Mixed types and missing values

Many tables are irregular in time: each entity has a different number of rows at different dates. A latent ODE or neural CDE can read such rows without resampling to a fixed grid. Missing cells become gaps in observation, not values that must be filled. The model carries its state forward through the gap, using the learned dynamics.

This is a natural fit for imputation. You infer the hidden trajectory, then read off the value at any missing time. See Imputation networks and Masked cell modeling for the more common table-wide methods.

No column names

A model that sees only numbers, without names, must learn the meaning of each column from its values and its relation to other columns. A shared vector field can act as a learned "physics" over column embeddings. The field learns how a state should evolve given its own content, not given a label. This is one way continuous-depth layers can stay name-free.

Pre-training on synthetic and real tables

Tabular foundation models such as TabPFN and TabICL are pre-trained on large numbers of synthetic tables drawn from priors. Some of those priors resemble dynamical systems or structural causal models. A neural ODE is a way to generate or learn such dynamics. It can serve as a data generator for time-indexed synthetic tables, which adds variety to pre-training. See Synthetic data generation.

In-context learning

In-context learning means the model reads labelled example rows and predicts for new rows without weight updates. The usual engine is attention. A neural ODE can in principle play the role of a learned inference step over the context, but it is not the standard choice. Attention is faster on modern hardware and easier to batch.

Continual learning

A continuous-time model can absorb new data points at new times without a fixed window. When new rows arrive, the state can be updated from the last known time. This helps in continual or streaming settings where the table keeps growing. It does not remove the usual risk of forgetting, so it is paired with replay or regularisation.

Anomaly detection

Trajectory models give a natural anomaly score. If a new reading is far from the predicted path, it is suspicious. This works well for sensor tables where behaviour is smooth. See Anomaly detection and Autoencoders for reconstruction-based scoring.

Sector and semantic classification

For classifying what a table or column is about, the signal comes mostly from value patterns and distribution shape. A neural ODE is rarely used here. A time-indexed column may still benefit from a trajectory encoder that summarises its dynamics into a fixed vector, which a classification head such as the ones described in Semantic sector heads can then read.

Where neural ODEs do not fit

For ordinary, non-temporal tables with a few hundred to a few hundred thousand rows, attention-based models and gradient boosted trees are the stronger and simpler choices. See XGBoost and LightGBM. Adding an ODE solver to a static table brings cost without clear gain. For the wider picture of how these models are built and tested, see /model/architecture, /model/research and /model/benchmarks.

Practical guidance and best practices

Choosing a setting

• Use a neural ODE when time is irregular, when memory is tight, or when you want a smooth invertible map.
• Use a fixed-step solver first. It is predictable and easy to debug.
• Move to an adaptive solver only when a fixed step cannot meet accuracy.

Tuning the solver

• Start with rtol near 1e-3 and atol near 1e-4. Tighten later.
• Watch NFE across epochs. A steady rise is a warning sign.
• Add a small penalty on the size of the field or on the NFE to keep dynamics simple.

Network design

• Use smooth activations such as tanh, softplus or swish.
• Keep the field network small. Two hidden layers are often enough.
• Avoid batch normalisation inside the field, as it breaks the ODE assumption that f depends only on state and time.
• Add time as an input to the field if the dynamics change with time.

Data handling

• Normalise inputs and, for time series, rescale time to a modest range such as 0 to 1 or 0 to 10.
• Pass the real time stamps. Do not round them to a grid.
• Keep a mask of observed values when data have gaps.

Training habits

• Use gradient clipping.
• Use a learning rate warm-up.
• Compare adjoint and direct backpropagation on a small case to check they agree.
• Log solver statistics along with the loss.

Evaluating the result

• Test at time points not seen in training to check true continuous behaviour.
• Compare with a simple baseline, such as a recurrent network with time features.
• Report both accuracy and wall-clock time.

A short decision guide

Reading solver diagnostics

• A very small mean step size means the dynamics are stiff or the field is not smooth.
• A rising NFE across epochs means the learned field is getting harder to integrate.
• A large gap between forward and backward NFE suggests that the adjoint pass is struggling.
• Sudden loss spikes often line up with NFE spikes.

Extra tips for beginners

• Start with a toy problem such as a spiral or a damped oscillator.

• Plot the learned vector field as arrows. It shows what the model believes.

• Plot trajectories for several start points. Crossing paths signal a bug.

• Try the same data with a plain recurrent model to see the gap.

• Keep the run short at first. A few hundred steps is enough to see if the setup is sound.

• Save solver settings with the model, since the same weights behave differently under a different tolerance.

Common mistakes

• Using ReLU in the field and then wondering why steps become tiny.
• Setting very tight tolerances from the start, which makes training crawl.
• Ignoring NFE growth until training stalls.
• Rounding irregular time stamps to a grid, which throws away the main benefit.
• Applying a neural ODE to a static table with no time axis and expecting a gain.
• Using batch normalisation or dropout inside the field in a way that changes between solver calls.
• Forgetting that adjoint gradients are approximate, then chasing noise.
• Reading the final state without a suitable head, so that scale mismatches hurt the loss.
• Feeding very long time ranges so that trajectories blow up.
• Skipping a simple baseline. Many reported gains vanish against a well tuned recurrent model.

Frequently asked questions

What is a neural ODE in simple terms?

A neural ODE is a network that defines how a hidden state changes over time and uses a numerical solver to follow that change. Instead of a fixed list of layers, you have one learned rule for the rate of change. The output is the state after the solver has followed that rule for a set span of time.

How is a neural ODE different from a residual network?

A residual network adds a learned update at each of a fixed number of layers. A neural ODE is the limit of that process with infinitely many tiny updates, and a solver picks how many steps to use. The ODE also shares one set of weights across all depth, while a residual network has separate weights in each block.

What is the adjoint method and why is it used?

The adjoint method computes gradients by solving a second ODE backwards in time instead of storing every forward step. This keeps memory use roughly constant no matter how many solver steps were taken. The price is more computation and small numerical errors in the gradients.

Are neural ODEs good for time series?

They are a strong choice when the time stamps are irregular or when you need to query the model at times between observations. On regular, dense series, a recurrent or attention model is often just as accurate and much faster, so a neural ODE is best used when its special features matter.

Why are neural ODEs slow?

Each forward pass calls the vector field network many times as the solver steps through time, and the backward pass adds more calls. The number of calls also changes from input to input, which makes batching harder. Fixed-step solvers and penalties on solver work can reduce the cost.

Can a neural ODE handle missing data?

Yes, in the sense that missing observations are simply times at which no data point is given. The hidden state continues to evolve through the gap, so there is no need to fill values before training. If you also need the missing values themselves, you can read them from the trajectory.

Do neural ODEs work on ordinary tabular data?

Not usually. A table without a time axis gives the ODE nothing to integrate over, other than an artificial depth variable. Tree ensembles and tabular foundation models are normally better and faster for such tables. Neural ODEs make more sense for tables that hold repeated measurements over time.

What is a latent ODE?

A latent ODE is a generative model in which a hidden state follows an ODE and the observed data are noisy readings of that state. An encoder infers the starting state from the data, the ODE evolves it, and a decoder maps it back to observations. It is used for irregular time series and for sampling smooth trajectories.

How does a neural ODE relate to state-space models like S4 and Mamba?

All three describe a hidden state that changes over continuous time. S4 and Mamba use a linear field with learned structure and a fast discretisation, while a neural ODE uses a general nonlinear field and a flexible solver. The linear versions are far faster to train in parallel, and the nonlinear version is more expressive.

What software is used to build neural ODEs?

The usual route is a differential equation solver library that works with an automatic differentiation framework, for example one built on PyTorch or JAX. These libraries provide fixed and adaptive solvers and an adjoint option, so you only write the field network and the loss.

Glossary

• Adjoint method: a backward ODE used to get gradients without storing all forward states.
• Adaptive solver: a solver that changes its step size to meet an error target.
• Augmented ODE: an ODE with extra state dimensions that lets trajectories avoid crossing.
• Continuous normalizing flow: a flow model whose transform is defined by an ODE.
• Dormand-Prince: a common adaptive Runge-Kutta method.
• Euler method: the simplest solver, a straight-line step along the field.
• Flow map: the function from the start state to the state at a later time.
• Initial value problem: find the trajectory given a starting state and a rule.
• Latent ODE: a model with a hidden ODE state behind noisy data.
• NFE: number of function evaluations made by the solver.
• Neural CDE: a model where the state is driven by a path of inputs.
• ODE: an equation giving the rate of change of a state.
• Runge-Kutta: a family of higher-order solvers.
• Stiff system: a system with fast and slow parts that forces small steps.
• Tolerance: the allowed solver error per step.
• Vector field: a function giving a direction of change at each point.

Summary

A neural ODE is a network that learns the rate of change of a hidden state and uses a solver to compute outputs. It offers continuous depth, continuous time, constant training memory with the adjoint method and a clean fit for irregularly sampled data. It costs more compute than a plain stack of layers and needs care with solver settings.

For tabular work, neural ODEs matter most when a table holds repeated, irregular measurements per entity, where they support imputation, forecasting and anomaly scoring. For static tables, attention-based tabular foundation models and tree ensembles remain the practical choices. Related reading: Mamba, S4, Autoencoders, /model/research and /model/benchmarks.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: