PINN

Overview

A PINN, short for physics-informed neural network, is a neural network that learns from data and from a physical law at the same time. The law is usually a differential equation. The network is trained so that its output both matches the measurements and satisfies the equation. This lets it make sensible predictions even when data is scarce or noisy.

The core trick is simple. Automatic differentiation gives exact derivatives of the network output with respect to its inputs. Those derivatives are plugged into the equation, and whatever is left over, the residual, becomes part of the loss. A PINN is therefore a regularized regression model where the regularizer is physics. It is widely used for surrogate models, inverse problems and digital twins, and it is a useful mental model for how prior knowledge can be injected into any learner, including models that read raw tables.

What it is

A PINN is a function approximator, almost always a fully connected network, that maps coordinates to a physical quantity. For example, the inputs might be position x and time t, and the output might be temperature u(x, t).

The network has no built-in knowledge of heat flow. Knowledge enters through the loss function. The loss has several parts:

• A data term that compares predictions with observed measurements.
• A residual term that measures how badly the predicted field violates the governing equation.
• A boundary term that enforces conditions at the edges of the domain.
• An initial term that enforces the starting state for time-dependent problems.

Think of a student solving a physics exam. A normal neural network is a student who memorizes past answers. A PINN is a student who memorizes past answers and also checks every result against the formulas printed on the cover of the exam. If an answer contradicts the formula, the student loses marks and corrects it.

What it is not

• It is not a classical numerical solver such as finite elements. It has no mesh and no guaranteed error bounds.
• It is not a pure black box. The structure of the problem is written into the loss.
• It is not a single architecture. The word describes a training idea, not a layer type.
• It is not a tabular model. It works on continuous fields, although the same idea of soft constraints transfers to tables.

Where it came from

The idea of solving differential equations with neural networks is older than modern deep learning. Work from the 1990s already trained small networks to satisfy ordinary and partial differential equations by minimizing a residual at sample points.

The modern formulation was popularized by a 2019 paper in the Journal of Computational Physics by Raissi, Perdikaris and Karniadakis, titled "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations". It combined three ingredients that had matured: cheap automatic differentiation in deep learning frameworks, expressive deep networks, and GPU training.

Several related lines of work followed:

• Neural operators such as DeepONet and the Fourier neural operator, which learn maps between function spaces rather than a single solution.
• Domain decomposition variants that split a large domain into subdomains with separate networks.
• Variational forms that use weak formulations of the equation.
• Hard-constraint designs that build boundary conditions into the network output so they hold exactly.
• Software libraries that package collocation sampling, residual computation and loss weighting.

The field is now often called scientific machine learning. It sits between numerical analysis and deep learning. For the broader context of how this kind of prior knowledge links to modern models, see /model/research.

How it works

Step 1: choose the physics

Write the equation as a residual that should equal zero. For one-dimensional heat flow, the equation says that the rate of temperature change over time equals a diffusion constant times the second derivative in space. Move everything to one side and you have a residual r(x, t).

Everyday analogy: a bathroom scale should read zero when nothing is on it. The residual is the reading on an empty scale. Any non-zero reading means something is off.

Step 2: build the network

Create a network u_theta(x, t). Smooth activations such as tanh are common, because the equation needs derivatives of the output, and a function with a flat second derivative would hide the physics. Networks with ReLU activations have zero second derivatives almost everywhere, which makes them a poor fit for second-order equations.

Step 3: sample collocation points

Pick many points inside the domain where the residual will be checked. These are called collocation points. They do not need measurements. Pick points on the boundary and at the initial time as well.

Analogy: a building inspector does not check every brick. The inspector picks representative spots across the wall and tests those. The more spots, the more confidence.

Step 4: compute derivatives with automatic differentiation

Use reverse-mode automatic differentiation twice (or more) to get derivatives of u_theta with respect to x and t. These are exact derivatives of the network, not finite-difference estimates. A framework such as PyTorch or JAX provides this directly.

Step 5: assemble the loss

The total loss is a weighted sum:

• Data loss on measured points.
• Residual loss at collocation points.
• Boundary and initial loss.

The weights decide which part of the problem the optimizer cares about most.

Step 6: optimize

Train with a first-order optimizer such as Adam for a while, then often switch to a quasi-Newton method such as L-BFGS to polish the result. Physics losses have sharp, ill-conditioned landscapes, and second-order-like methods help at the end.

Step 7: evaluate

Compare against a reference solution if one exists, or against held-out measurements. Check the residual on fresh points the optimizer never saw. A low training residual with a high residual at new points signals overfitting to collocation points.

Forward and inverse problems

• Forward problem: parameters and conditions are known, find the solution field.
• Inverse problem: some observations are known, find unknown parameters such as a diffusion coefficient or a source term.

For inverse problems, the unknown parameters are simply added as trainable variables next to the network weights. This is one of the most practical uses of PINNs, because the physics reduces the amount of data needed to pin down the parameters.

Variants and extensions

The basic recipe has grown into a family of methods. Each one fixes a specific weakness.

• Hard-constraint PINNs multiply or shift the network output so that boundary and initial conditions hold exactly. This removes loss terms and makes training easier.
• Domain decomposition PINNs use one small network per subdomain and add terms that glue the pieces together at the interfaces. They scale to larger domains and to solutions with different behavior in different regions.
• Variational PINNs use a weak form of the equation, which lowers the order of derivatives needed and can be more stable.
• Causal PINNs weight early times more heavily than late times, so the network learns the solution in the order that time flows.
• Adaptive sampling moves collocation points toward regions where the residual is large, much like a teacher spending more time on the hardest topics.
• Gradient-enhanced PINNs also penalize the derivative of the residual, which adds smoothness information.
• Bayesian PINNs attach uncertainty to the solution, which matters when data is noisy.
• Operator-learning hybrids combine a neural operator with a physics residual, so the model needs fewer simulated examples.

Typical application areas

• Heat transfer and diffusion in materials and buildings.
• Fluid flow around simple shapes, including inverse problems that recover a hidden pressure or velocity field from a few sensor readings.
• Structural mechanics, such as estimating stiffness from displacement data.
• Subsurface flow and groundwater models.
• Biomedical flow, for example estimating blood flow patterns from partial imaging.
• Electromagnetics and wave propagation.
• Chemical kinetics and reaction networks, where the parameters of rate equations are unknown.
• Finance and epidemiology, where a differential equation describes growth but its parameters change.

What the optimizer actually sees

It helps to picture the training as a tug of war between ropes. One rope pulls the network toward the measurements. Another pulls it toward satisfying the equation. A third pulls it toward the boundary values. If one rope is much stronger, the others barely matter, and the network lands in the wrong place. This is why loss weights, scaling and sampling dominate practical work, much more than the choice of layer width.

Why smooth activations matter

An equation with a second derivative needs a network whose second derivative is informative. A network made of piecewise linear pieces has a second derivative that is zero almost everywhere, so the residual teaches nothing. Smooth activations such as tanh, sine or softplus give useful higher-order derivatives. This is a small detail with a large effect, and it is a good example of how the physics changes standard deep learning advice.

Key concepts and terms

A worked example

We solve a simple problem: a decaying quantity described by du/dt = -k u with u(0) = 1 and k = 2. The exact answer is u(t) = exp(-2 t). We train a small network to find it using only the equation and the initial condition. This is a good first PINN because we can check the answer by hand.

import torch
import torch.nn as nn

torch.manual_seed(0)

class Net(nn.Module):
    def __init__(self, width=32):
        super().__init__()
        self.f = nn.Sequential(
            nn.Linear(1, width), nn.Tanh(),
            nn.Linear(width, width), nn.Tanh(),
            nn.Linear(width, 1),
        )

    def forward(self, t):
        return self.f(t)

net = Net()
k = 2.0
opt = torch.optim.Adam(net.parameters(), lr=1e-3)

t_col = torch.rand(200, 1, requires_grad=True)   # collocation points in [0, 1]
t0 = torch.zeros(1, 1)                           # initial time

for step in range(5000):
    opt.zero_grad()
    u = net(t_col)
    du_dt = torch.autograd.grad(
        u, t_col, grad_outputs=torch.ones_like(u), create_graph=True
    )[0]
    residual = du_dt + k * u                      # should be zero
    loss_phys = (residual ** 2).mean()
    loss_init = ((net(t0) - 1.0) ** 2).mean()     # u(0) = 1
    loss = loss_phys + loss_init
    loss.backward()
    opt.step()

with torch.no_grad():
    t_test = torch.linspace(0, 1, 5).reshape(-1, 1)
    print(torch.cat([t_test, net(t_test), torch.exp(-k * t_test)], dim=1))

Reading the result

The printed table has three columns: time, the network prediction and the exact answer. After training, the prediction and the exact answer should agree to about two or three decimal places. Note what did not happen: the network never saw a single measurement of u beyond the initial value. The equation alone shaped the curve.

A numeric walk-through of one residual

Suppose at t = 0.5 the network currently outputs u = 0.40 and its slope is du/dt = -0.5.

• Residual = du/dt + k u = -0.5 + 2 x 0.40 = 0.30.
• Squared residual = 0.09.

The exact solution at t = 0.5 is exp(-1), about 0.368, with slope about -0.736. Then the residual is -0.736 + 2 x 0.368 = 0. The optimizer pushes the network toward the state where this number is zero at every collocation point.

Adding noisy data

If we also have a few noisy measurements, add a data term:

t_obs = torch.tensor([[0.1], [0.4], [0.9]])
u_obs = torch.exp(-k * t_obs) + 0.01 * torch.randn_like(t_obs)
loss_data = ((net(t_obs) - u_obs) ** 2).mean()
loss = loss_phys + loss_init + 10.0 * loss_data

The weight 10.0 is a design choice. Too small and the data is ignored. Too large and the physics is ignored. Tuning these weights is among the hardest practical parts of the method.

An inverse variant

Make k a trainable parameter:

k = nn.Parameter(torch.tensor(1.0))
opt = torch.optim.Adam(list(net.parameters()) + [k], lr=1e-3)

With a few observations of u over time, training recovers a value of k close to 2. The physics form (exponential decay) is assumed. Only its rate is learned.

Strengths and limitations

Strengths

• Works with little data because the equation supplies information.
• Mesh-free: no grid generation is needed, which helps with complex shapes.
• Handles inverse problems in the same framework as forward problems.
• Produces a smooth, differentiable solution that can be queried anywhere.
• Easy to combine measurements, equations and constraints in one loss.
• Fast to evaluate after training, so it can act as a surrogate in loops.

Limitations

• Training can be slow and unstable. The loss landscape is hard.
• Loss terms compete, and bad weighting leads to a solution that satisfies one term and ignores another.
• Spectral bias makes high-frequency or multi-scale solutions hard to learn.
• Stiff systems, shocks and chaotic dynamics are difficult.
• No built-in error bound. A low loss does not guarantee an accurate solution.
• For well-posed problems on simple domains, classical solvers are often faster and more reliable.
• Each new geometry or parameter set may need retraining, unless an operator method is used.

Common failure patterns

• The network finds a trivial solution that satisfies the equation but not the physics you meant, such as a constant function when the boundary conditions are too weak.
• The residual is small at training points but large between them.
• Gradients of different loss terms differ by orders of magnitude.
• Long time horizons make the solution drift because errors accumulate.

Mitigations include adaptive loss weights, causal training that respects the flow of time, Fourier feature inputs, domain decomposition and adaptive resampling of collocation points where the residual is large.

A configuration sketch

Many teams keep the settings of a PINN run in a small file so that experiments can be repeated. A plausible layout looks like this.

problem:
  equation: heat_1d
  diffusivity: 0.1
  domain: {x: [0.0, 1.0], t: [0.0, 1.0]}
network:
  hidden_layers: 4
  width: 64
  activation: tanh
  fourier_features: false
sampling:
  n_collocation: 5000
  n_boundary: 400
  n_initial: 400
  resample_every: 200
loss_weights:
  residual: 1.0
  boundary: 10.0
  initial: 10.0
  data: 5.0
optimizer:
  stage_1: {name: adam, lr: 0.001, steps: 20000}
  stage_2: {name: lbfgs, steps: 2000}
seeds: [0, 1, 2, 3, 4]

Keeping seeds in the file is a good habit. Five runs give a feel for the spread of results, and a method that works for only one seed is not reliable.

Reading the loss curves

• If the residual loss falls while the boundary loss stays high, the network is solving the interior and ignoring the edges. Raise the boundary weight.
• If all terms fall slowly and flatten, the learning rate may be too small, or the network may be too small for the solution.
• If the loss jumps up and down, lower the learning rate or switch to a smoother schedule.
• If the training residual is tiny but the test residual is large, add collocation points or resample more often.

How it compares with alternatives

Picking between them

• If you trust the equation and have a clean domain, use a numerical solver.
• If the equation is known but data is sparse and parameters are unknown, a PINN is a good fit.
• If you have many simulations and need instant answers across a family of conditions, a neural operator or a surrogate trained on simulation data is a better fit.
• If you have plenty of data and no clear equation, a standard model such as a transformer or a tree ensemble is simpler.

Role in tabular foundation models

PINNs and tabular foundation models both address the question of what to do when data is limited. They answer it in different ways, and the two ideas meet in several practical places. For the design of a model that reads raw tables, see /model/architecture.

Priors instead of data

A tabular foundation model such as TabPFN is pre-trained on huge numbers of synthetic tables drawn from a prior. After pre-training, it makes predictions on a new table with in-context learning: the training rows are given as context, and no weight update happens. The prior plays the role the equation plays in a PINN. It encodes assumptions about how data tends to behave, so the model needs fewer rows to be accurate.

• In a PINN, the prior is a differential equation, enforced through the loss.
• In a tabular foundation model, the prior is a family of generators (for example, random causal structures and noise), enforced through pre-training.

Physics-based synthetic tables

A simulator that solves an equation can generate tables: each row is a set of parameters and conditions, and the target columns are simulated outputs. Such tables can enlarge the pre-training corpus for models that learn from raw tables. This is an application of synthetic data generation. The benefit is that the rows obey real physical relationships. The risk is that a narrow set of simulators teaches the model only that narrow world.

Soft constraints on tabular targets

Tabular problems in engineering often have known rules: a mass fraction must lie between 0 and 1, energy must balance, a flow cannot be negative, a monotone relationship holds. A penalty term added to the training loss works exactly like a PINN residual. The model is nudged to respect the rule without a hard clamp. This is useful for models that are fine-tuned on a specific industrial table.

Missing values and sensor columns

Industrial tables often have missing sensor readings. If a physical relationship links columns (for example, a conservation law across three flow meters), the relation can help fill a missing value. This is a physics-aware form of imputation. General-purpose imputation networks learn relationships from data alone. A physics residual can add a check: an imputed value that breaks a known balance is suspect.

Anomaly detection by residual

A trained PINN or a physics model gives an expected value. The gap between the observed row and the expected value is a residual. Large residuals flag rows that disagree with known physics, which is a form of anomaly detection. A tabular foundation model can use the same idea without a formula: it predicts a cell from the rest of the row, and a large surprise marks an unusual row.

No column names

A raw table may arrive with anonymous columns. A PINN needs to know which input is time and which is space, because the equation refers to them. A tabular foundation model cannot rely on names either, so it must infer structure from values and distributions. This is a real difference: physics priors need meaning, and meaning is exactly what is missing from anonymous columns. A practical compromise is to supply only the units or roles as metadata when they are known.

Mixed types

PINNs work with continuous inputs. Tables mix numbers, categories, dates and text. A hybrid system can use a tabular foundation model to handle the mixed columns and hand the continuous operating conditions to a physics module, or add physics-derived features (for example, a dimensionless number computed from several columns) as extra columns. Feature columns derived from physics often help a tabular model more than a larger network.

Continual learning

A deployed digital twin gets new data over time. A PINN can be refined with new measurements, but retraining may erase what it learned about earlier regimes. Techniques such as elastic weight consolidation and experience replay address this problem for both PINNs and tabular models. The physics term acts as a stable anchor, since it does not change as the data drifts.

Sector and semantic classification

Classifying a table by its sector or meaning is a different task from solving an equation, yet physics knowledge can help. Columns that follow thermal or fluid relationships hint at a process-industry table. The link is indirect, and it is better to treat PINNs as a source of features and constraints than as a classifier.

Honest summary of the link

• PINNs are not tabular models, and tabular foundation models are not PDE solvers.
• They share the idea that good priors replace data.
• PINN-style penalties, simulator-generated rows and physics features can improve tabular models in engineering settings.
• Results depend heavily on the quality of the physics and the care in weighting.

See /model/benchmarks for how tabular models are measured on ordinary data.

Practical guidance and best practices

Before training

• Write the equation, its units and its boundary conditions on paper first.
• Non-dimensionalize the problem. Scale inputs and outputs to roughly order one. Mismatched scales are a leading cause of bad training.
• Check that the problem is well posed. If a human cannot say what conditions pin down the solution, the network cannot either.
• Get a reference solution for a simplified case, even a crude one, to validate the code.

Architecture choices

• Start with 3 to 6 hidden layers of 32 to 128 units and smooth activations.
• Try Fourier features or sinusoidal inputs when the solution oscillates.
• Use hard constraints for boundary conditions when you can. They remove one loss term and help the optimizer.
• For problems with several scales, consider splitting the domain.

Loss design

• Normalize each term so that they start at similar magnitudes.
• Use adaptive weighting schemes that rebalance terms based on gradient norms or residual size.
• Log each loss term separately. A single total hides which part is failing.
• For time-dependent problems, train early times first, then extend.

Sampling

• Resample collocation points every few hundred steps.
• Put more points where the residual is high.
• Use quasi-random sequences for even coverage.
• Include enough boundary points to define the domain clearly.

Optimization

• Use Adam first, then L-BFGS.
• Use a decaying learning rate.
• Run several seeds. PINN training is sensitive to initialization.
• Use double precision when derivatives are of high order, since rounding can dominate otherwise. Compare with mixed precision trade-offs before using lower precision.

Validation

• Hold out measurements and check the error there.
• Evaluate the residual on fresh points.
• Compare with a trusted solver on at least one case.
• Report uncertainty if decisions depend on the answer. Ensembles or Bayesian deep learning methods can help.

Deployment

• Wrap the trained network as a function that accepts inputs and returns outputs plus a flag for out-of-range queries.
• Monitor the residual in production. A rising residual on new data means the physics no longer matches reality.
• Keep the training code and the solver reference with the model so that the surrogate can be audited.

A short checklist before you start

1. Can you write the governing equation and name every symbol in it?
2. Do you know the boundary and initial conditions?
3. Are all inputs and outputs scaled to a similar range?
4. Do you have a simple reference case to test the code?
5. Is there a reason a classical solver is not enough?
6. Do you know how you will measure success on points the optimizer never saw?

If you can answer all six, the project is well framed. If not, fix the framing before spending time on network design.

A short checklist for tabular use

1. List the physical rules that must hold between columns.
2. Turn each rule into a number that is zero when the rule holds.
3. Add the squared number to the loss with a modest weight.
4. Track how often the model breaks the rule on held-out rows.
5. Add physics-derived columns and see if a plain model improves.

Common mistakes

• Treating a low loss as proof of correctness. Always check against independent data or a solver.
• Skipping scaling. Raw units can differ by many orders of magnitude.
• Using ReLU with a second-order equation.
• Leaving loss weights at one and hoping for the best.
• Too few collocation points, which lets the network satisfy the equation only at those spots.
• Training once and trusting the result. Different seeds can give different answers.
• Forgetting the boundary or initial condition, so the problem is under-determined.
• Using a PINN where a classical solver would be faster and safer.
• Extrapolating beyond the sampled domain. The network has no physics guarantee there.
• Ignoring measurement noise. Fitting noisy data too tightly can fight against a correct equation.
• Mixing up the forward and the inverse problem, and then reading a fitted parameter as a certain fact.

Frequently asked questions

What does PINN stand for?

PINN stands for physics-informed neural network. It is a neural network whose training loss includes the residual of a physical law, usually a differential equation, so that predictions agree with both the data and the law.

How is a PINN different from a normal neural network?

A normal network only minimizes the gap between predictions and labeled examples. A PINN adds a second goal, which is to satisfy a known equation at many points where no labels exist. This extra information makes it useful when measurements are sparse or expensive.

Do PINNs need labeled data?

Not always. A forward problem can be solved with only the equation, the boundary conditions and the initial conditions. Labeled measurements become important for inverse problems, where unknown parameters must be found, and for cases where the equation is only partly known.

Are PINNs better than finite element methods?

Not in general. For forward problems on well-defined domains, mature numerical solvers are usually faster, more accurate and come with error estimates. PINNs shine when data must be combined with physics, when parameters are unknown, or when meshing is awkward.

Why do PINNs fail on some problems?

The usual reasons are competing loss terms, stiff or multi-scale solutions, shocks, long time horizons and poor scaling. The optimization problem is harder than ordinary regression, and small changes in weights or sampling can change the outcome a lot.

What are collocation points?

Collocation points are locations in the domain where the equation residual is evaluated during training. They do not need measured values. The network is encouraged to satisfy the equation at each of them, and good coverage matters for a reliable solution.

Can a PINN be used on tabular data?

The pure method works on continuous fields, not on general tables. The idea still carries over: a penalty for breaking a known rule can be added to a tabular model, and physics-derived columns can be added as features. Tabular foundation models can also be pre-trained on tables produced by physical simulators.

How does a PINN relate to a digital twin?

A digital twin is a live model of a physical asset. A PINN can serve as part of it, because it blends measured data with the governing equations and can be updated as new readings arrive. See digital twins for the wider picture.

What is the difference between a PINN and a neural operator?

A PINN learns one solution for one set of conditions, and usually needs retraining for new ones. A neural operator learns the mapping from conditions to solutions, so a single trained model answers many cases quickly, but it normally needs many simulated examples to train.

Which software is used to build PINNs?

Any framework with automatic differentiation will do, such as PyTorch or JAX. Specialized libraries exist that handle sampling, residual assembly and loss weighting, but the core method fits in a few dozen lines, as the example above shows.

Glossary

• Automatic differentiation: A technique that computes exact derivatives of a program by tracking operations.
• Boundary condition: A rule that the solution must obey on the edge of the domain.
• Collocation point: A sampled location where the equation residual is checked.
• Data loss: The part of the loss that compares predictions with measurements.
• Digital twin: A live virtual copy of a physical system, updated with real data.
• Forward problem: Find the solution when the equation and conditions are known.
• Hard constraint: A condition built into the network form so it holds exactly.
• Initial condition: The state of the system at the start time.
• Inverse problem: Find unknown parameters from observed outcomes.
• L-BFGS: A quasi-Newton optimizer often used to finish PINN training.
• Neural operator: A model that maps input functions to output functions.
• PDE: Partial differential equation, an equation relating a function and its derivatives in several variables.
• Residual: The amount by which a prediction fails to satisfy an equation.
• Soft constraint: A condition enforced by adding a penalty to the loss.
• Spectral bias: The tendency of networks to learn low-frequency patterns before high-frequency ones.
• Surrogate model: A fast approximation of an expensive simulation.
• Synthetic table: A table generated by a simulator or a statistical prior, not collected from the world.

Summary

A PINN trains a neural network on measurements and on a physical law at once. The law enters as a residual in the loss, computed with automatic differentiation at collocation points. This gives useful predictions with little data, supports inverse problems and yields smooth, queryable surrogates.

The method is sensitive to scaling, loss weights and sampling, and it does not replace classical solvers for easy forward problems. For tabular foundation models, the main lessons are about priors: physics penalties, simulator-generated tables and physics-derived features can all help a model that reads raw rows, while anomaly checks by residual and physics-aware imputation add safety. Explore TabPFN, anomaly detection and /model/architecture next.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: