NeRF

Overview

NeRF, short for neural radiance field, is a method that learns a 3D scene from a set of ordinary photographs and then renders the scene from new camera positions. A small neural network takes a 3D point and a viewing direction as input. It returns a color and a density. By sampling many points along each camera ray and blending them, the method paints a picture of the scene that was never photographed.

The key idea is that the scene lives inside the weights of a network instead of in a mesh or a grid of voxels. Training compares rendered pixels with real pixels and nudges the weights until they agree. The result is a compact, continuous model of a scene that handles fine detail, soft shadows and shiny surfaces well. The price is slow training and slow rendering in the original form, which later work such as Gaussian splatting and grid-based variants set out to fix.

What it is

A neural radiance field is a function. You give it a position in space, written as three numbers, and a direction from which you are looking, written as two angles. It answers with two things.

• A volume density, which says how much "stuff" is at that point. Empty air has density near zero. A solid wall has high density.
• A color, which can change with the viewing direction. This lets the model capture glossy highlights and reflections.

That function is stored in a multilayer perceptron, a plain feed-forward network. It is not a convolutional network and it does not look at images directly. It only knows about points in space. The images are used as the training signal.

An everyday analogy

Imagine a room filled with faint colored fog. At each spot the fog has a thickness and a tint. If you shine a flashlight along a line, the light picks up tint from every patch of fog it passes through and gets blocked where the fog is thick. A camera ray works the same way. NeRF learns the fog, one spot at a time, so that rays through it reproduce your photos.

What goes in and what comes out

• Input at training time: a set of images, plus the camera pose for each image (where the camera stood and where it pointed) and the camera intrinsics (focal length and sensor details).
• Output after training: a function you can query anywhere, and from it, a rendered image for any new camera pose.

Implicit versus explicit scenes

Classic 3D data is explicit. A mesh lists triangles. A point cloud lists points. A voxel grid lists cells. A radiance field is implicit. The shape is not listed anywhere. It is defined by what the network returns when you ask. Implicit models are smooth and compact, but you have to run the network to see anything.

Where it came from

The roots lie in older ideas from graphics and vision.

• Image-based rendering tried to make new views by blending nearby photos.
• Structure from motion recovered camera positions and sparse 3D points from photo sets.
• Multi-view stereo built denser surfaces from those poses.
• Volume rendering, long used in medical imaging and film effects, described how to integrate color and opacity along a ray.
• Neural implicit representations showed that a network could encode a shape as a signed distance or an occupancy field.

The paper that named and popularized the method, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", appeared in 2020. It combined a coordinate-based network, classic volume rendering and a positional encoding trick, and it produced strikingly sharp novel views from a modest number of photos.

A wave of follow-up work arrived within a couple of years.

• Mip-NeRF (2021) handled scale and aliasing by representing cones of light instead of thin rays.
• Instant-NGP (2022) used a multiresolution hash grid of learned features, which cut training from hours to minutes.
• Many papers handled unbounded scenes, dynamic scenes, sparse input views, relighting, and editing.
• Surface-focused variants recovered clean meshes from radiance fields.

In 2023 a different representation based on many small 3D Gaussians reached real-time rendering speeds and shifted much attention away from pure network-based fields. NeRF remains the reference point for the whole family, and its ideas still run through newer methods.

How it works

This section walks through the pipeline in the order the data flows.

Step 1: collect photos and recover cameras

You take many photos of a scene from different angles. Overlap matters. Each point should appear in several images. A separate tool, often a structure-from-motion program, finds matching features across images and solves for each camera pose. If the poses are wrong, the radiance field will be blurry, because the training signal contradicts itself.

Step 2: turn pixels into rays

Every pixel in every image defines a ray. The ray starts at the camera center and passes through the pixel on the image plane. Given the camera pose and focal length, you can compute the ray origin and direction for any pixel.

Step 3: sample points along each ray

Along the ray, between a near and a far limit, you pick many sample positions. Early versions picked them evenly with some random jitter. Later versions used a two-stage plan: first a coarse pass to find where the density is, then a fine pass that places more samples where the scene actually has content. Think of it as a first rough survey followed by a closer look at the interesting parts.

Step 4: encode the positions

A plain network struggles to produce sharp detail from raw coordinates. It tends to give smooth, blurry output. The fix is positional encoding. Each coordinate is mapped to a set of sine and cosine values at several frequencies. This gives the network easy access to both coarse and fine patterns.

An analogy: telling someone a street address by giving only the house number is hard to use. Giving the number, the block, the district and the city at different levels of detail makes it easy to place.

Grid-based variants replace the fixed sine and cosine features with learned features stored in a table indexed by position, which is much faster.

Step 5: query the network

At each sample point, the network receives the encoded position and the encoded viewing direction. A common design lets density depend only on position, while color depends on position and direction. That rule keeps the geometry the same from every view, while letting highlights shift as you move.

Step 6: volume rendering

Now you have, along one ray, a list of colors and densities. To get one pixel color you integrate.

• For each sample, compute how much light survives up to that point. This is called transmittance. It drops as the ray passes through dense regions.
• Compute the sample's opacity from its density and the gap to the next sample.
• Weight each sample color by transmittance times opacity.
• Sum the weighted colors.

This is a differentiable formula. That is the heart of the method. Because every step can be differentiated, a gradient can flow from the pixel error all the way back to the network weights.

Step 7: compare with the real pixel and update

The rendered pixel is compared with the true pixel, usually by squared error. A batch of rays from random images is processed together, the loss is averaged, and an optimizer updates the weights. Repeating this many thousands of times drives the field toward a scene that explains all the photos at once.

Step 8: render new views

After training, pick any camera pose, build rays for every pixel, run the same sample, query and render steps, and you get an image from a viewpoint no one photographed.

A tiny sketch in code

The following is a simplified outline in PyTorch style. It leaves out batching details, hierarchical sampling and many practical tricks.

import torch
import torch.nn as nn

def encode(x, n_freqs=10):
    out = [x]
    for k in range(n_freqs):
        out.append(torch.sin((2.0 ** k) * x))
        out.append(torch.cos((2.0 ** k) * x))
    return torch.cat(out, dim=-1)

class TinyNeRF(nn.Module):
    def __init__(self, width=128):
        super().__init__()
        in_pos = 3 + 3 * 2 * 10
        in_dir = 3 + 3 * 2 * 4
        self.trunk = nn.Sequential(
            nn.Linear(in_pos, width), nn.ReLU(),
            nn.Linear(width, width), nn.ReLU(),
            nn.Linear(width, width), nn.ReLU(),
        )
        self.density = nn.Linear(width, 1)
        self.color = nn.Sequential(
            nn.Linear(width + in_dir, width // 2), nn.ReLU(),
            nn.Linear(width // 2, 3), nn.Sigmoid(),
        )

    def forward(self, pos, direction):
        h = self.trunk(encode(pos, 10))
        sigma = torch.relu(self.density(h))
        d = encode(direction, 4)
        rgb = self.color(torch.cat([h, d], dim=-1))
        return rgb, sigma

def render_ray(model, origin, direction, near=2.0, far=6.0, n=64):
    t = torch.linspace(near, far, n)
    pts = origin + t[:, None] * direction
    dirs = direction.expand_as(pts)
    rgb, sigma = model(pts, dirs)
    delta = torch.cat([t[1:] - t[:-1], torch.tensor([1e10])])
    alpha = 1.0 - torch.exp(-sigma.squeeze(-1) * delta)
    trans = torch.cumprod(torch.cat([torch.ones(1), 1.0 - alpha + 1e-10]), 0)[:-1]
    weights = alpha * trans
    return (weights[:, None] * rgb).sum(dim=0)

Key concepts and terms

A worked example

Suppose you photograph a small garden statue with a phone. You walk in a half circle and take 80 photos. This example shows the workflow and the numbers that matter.

Preparation

1. Photograph the object with steady lighting and consistent exposure. Turn off auto exposure changes if you can.
2. Run a structure-from-motion tool on the photos. It outputs a pose for each image and a sparse cloud.
3. Check how many images were registered. If only half were, retake photos with more overlap.
4. Hold out about ten percent of the images. You will not train on them. They serve as a test.

Training

Here is a small configuration in YAML, using placeholder names.

dataset:
  path: data/statue
  image_downscale: 4
  holdout_every: 10
model:
  type: grid_nerf
  hash_levels: 16
  features_per_level: 2
  mlp_width: 64
render:
  samples_per_ray: 64
  near: 0.1
  far: 8.0
train:
  rays_per_batch: 4096
  steps: 20000
  lr: 0.01
  loss: mse

During training, a log may look like the following. The numbers are only an illustration of the typical shape of progress, not a measured result.

step 1000   loss 0.0210   psnr 17
step 5000   loss 0.0060   psnr 22
step 10000  loss 0.0031   psnr 25
step 20000  loss 0.0022   psnr 27

The pattern is the point. Early steps reveal coarse color and shape. Later steps sharpen edges and textures. Past a certain point, the training score keeps rising but the held-out score flattens, which signals overfitting to the training views.

A numeric walk-through of one ray

Consider a ray with four samples. The densities and gaps are as follows.

Alpha is one minus the exponential of minus density times gap. Transmittance at each sample is the product of one minus alpha for all earlier samples.

• Sample 1 transmittance is 1.00, weight is 0.00.
• Sample 2 transmittance is 1.00, weight is 0.39.
• Sample 3 transmittance is 0.61, weight is about 0.52.
• Sample 4 transmittance is about 0.085, weight is about 0.08.

The final pixel is the weighted sum of colors, dominated by samples 2 and 3. Sample 4 sits behind a dense region, so it barely contributes. This is exactly how occlusion emerges without any explicit geometry.

Evaluation

Render the held-out views and compare them to the real photos using PSNR, SSIM and LPIPS. Look at the images too. A good score with visible floaters or blur near edges should not be ignored.

Export

Many toolkits let you export a camera path as a video, extract a mesh with a marching-cubes step, or export a point cloud sampled from the density. Mesh quality depends on how sharp the learned density is.

Strengths and limitations

Strengths

• Photo-realistic novel views from ordinary photos.
• Handles view-dependent effects such as gloss and reflection.
• Compact: the scene is a small set of weights or features.
• Continuous: no fixed resolution of a mesh or grid.
• Fully differentiable, so it fits into larger learning pipelines.
• Works with a wide range of scene types, from objects to rooms to streets.

Limitations

• The original form trains slowly, often hours per scene.
• Rendering requires many network queries per pixel, so it is slow for real-time use.
• It needs accurate camera poses. Errors blur the result.
• Each scene is trained separately. The basic method does not generalize to new scenes without training, although generalizable variants exist.
• Moving objects, changing light and auto exposure break the static scene assumption.
• Unbounded outdoor scenes and thin structures are hard.
• Extracting precise geometry is harder than getting pretty images.
• Editing a scene is awkward because the content is hidden in weights.

How it compares with alternatives

A good rule: if you need real-time viewing, look at splats or baked meshes. If you need a research baseline or a smooth continuous field, NeRF-style methods remain a clear choice.

Role in tabular foundation models

NeRF is a method for images and 3D space, not for tables. Even so, it teaches several lessons that carry over to models that learn from raw tables, and it touches spatial tabular data directly.

A coordinate network is a function approximator

A radiance field is a small network that maps coordinates to values. A table can be viewed the same way. A row index and a column index map to a cell value. This is the idea behind implicit representations of data: store a smooth function instead of a list of numbers. Tabular foundation models do something related when they learn a function from row features to a target, as described on the architecture page.

Positional encoding and column identity

NeRF showed that raw coordinates are poor inputs and that sine and cosine features help a network see detail. Tables face a parallel issue. A raw number such as "37" carries little meaning without context, and a model with no column names has to infer what a column is. Tabular models use learned embeddings of numeric values and column positions, in the same spirit. See FT-Transformer and masked cell modeling for how cells become tokens.

Reconstruction from partial observations

NeRF fits a scene from a sparse set of views and fills in what no photo shows. That is imputation in another setting. A tabular model with missing values does the same thing: it uses the observed cells to predict the missing ones. The loss is computed only where data exists, just as a NeRF loss is computed only on pixels that were photographed. Related topics are imputation networks and denoising autoencoders.

Per-scene fitting versus in-context learning

The basic NeRF is trained from scratch for every scene. A tabular foundation model works differently. It is pre-trained on a large number of tables, many of them synthetic, so that at use time it can read a new table in context and predict without gradient updates. This contrast is useful. NeRF shows what you gain from per-instance optimization (high fidelity) and what you pay (time). In-context models such as TabPFN and TabICL give up some per-dataset fitting in return for speed. Generalizable NeRF variants, which learn priors across many scenes, follow the same path as foundation models.

Spatial tables and geography

Many real tables carry coordinates: latitude and longitude, building footprints, sensor locations, parcel boundaries. A radiance field can supply 3D structure for such rows, for example turning photos of a site into a model that links to a table of assets. The geographic side of the work is covered under GIS. A tabular model can then use derived columns such as height, volume or visibility as features.

Anomaly detection and change

Comparing a rendered view with a new photo highlights differences. Pixels that disagree with the learned field can signal change, damage or an object that was not there before. That is anomaly detection on a spatial signal. Tabular anomaly methods ask a similar question about rows: does this row fit what the model expects? See anomaly detection.

Continual learning

A radiance field trained on one set of photos can forget if you fine-tune it on new ones. Continual learning methods for tabular models, such as replay or regularization, face the same problem. The lesson is to keep a small memory of old data or to constrain how far weights may move. See experience replay and EWC.

Sector and semantic classification

Scene understanding adds semantic labels to a radiance field, so each point carries a class such as wall, tree or road. Tabular models assign semantic classes to whole tables or columns, for example telling whether a table is about retail or logistics from its values alone. The mechanisms differ, but the goal of attaching meaning to raw structure is shared. See semantic sector heads.

What not to claim

NeRF does not replace a tabular model, and a tabular foundation model does not replace NeRF. They live in different data worlds. The value of the comparison is conceptual: coordinate encodings, fitting from partial data, and the trade between per-instance fitting and pre-trained priors. For details of this site's own approach, see /model/research and /model/benchmarks.

Practical guidance and best practices

Capture

• Use sharp images. Motion blur harms pose estimation and training.
• Keep exposure, white balance and focus consistent.
• Capture in a pattern that covers the object from many angles with strong overlap.
• Avoid moving objects, people and changing light.
• Include some close views for detail and some wide views for context.
• Avoid large featureless areas such as blank walls, which break pose estimation.

Poses

• Verify that nearly all images were registered.
• Inspect the recovered camera path. It should look like what you did.
• If poses are noisy, consider methods that refine them during training.

Training

• Start from a fast grid-based implementation for iteration, then try higher quality settings once the pipeline works.
• Downscale images for early tests.
• Hold out views and track a held-out score.
• Use a distortion or floater penalty if the toolkit offers one.
• Set near and far bounds sensibly. Too wide wastes samples, too tight cuts content.
• Use background modelling for outdoor scenes.

Rendering and delivery

• Decide early whether the end goal is a video, an interactive viewer, or a mesh. That choice shapes the method.
• For interactive web viewing, consider converting to a representation built for real-time rendering.
• Keep the original photos and poses. They let you retrain with better settings later.

Hardware

• A modern GPU with decent memory is nearly required. Training on a CPU is impractical.
• Memory use grows with the size of the feature grid and the batch of rays.

Reproducibility

• Fix random seeds and log the configuration.
• Record the software version of the pose tool and the training toolkit.
• Save the held-out split so you compare methods fairly.

Common variants at a glance

Reading a result critically

A pretty video can hide problems. Before you trust a radiance field, run these checks.

• Orbit the camera far from the training path. Does the scene fall apart?
• Look at thin objects such as wires and leaves. They are often smeared.
• Look at mirrors and glass. Reflections may be baked in as fake geometry.
• Check the empty space around the object for floaters.
• Compare depth maps with what you know about the scene.
• Check the edges of the covered area. Quality usually drops sharply there.

A simple project plan

1. Define the goal: a video, a viewer, a mesh, or an input to analysis.
2. Capture with the checklist above and keep a log of conditions.
3. Estimate poses and reject images that failed to register.
4. Train a fast baseline at low resolution.
5. Inspect held-out renders and fix capture or pose issues.
6. Train the final model at higher quality.
7. Export, document the settings and archive the inputs.

Cost and effort

Costs fall mostly in capture time and GPU time. A quick object capture takes minutes to shoot and minutes to train with a grid-based method. A large site may need several capture sessions and long training runs. Plan for retakes, since a bad capture is cheaper to fix early than to hide with settings later.

Data governance for captured scenes

Photos of places can include faces, license plates and private property. Treat captured data with the same care as any other personal or sensitive data. Blur or remove people where required, store raw photos securely, and keep a record of who may use the trained model.

Questions to ask before you start

• Is the scene static for the whole capture?
• Can you control or at least record the lighting?
• Do you need geometry, or only good-looking views?
• Who will view the result, and on what device?
• How will you store and version the raw photos and poses?
• What is the acceptable error, and how will you measure it?

Where the output can feed other systems

A trained field can produce depth maps, point clouds, meshes and rendered images. Each of these is a plain file that other tools can read. Depth maps and point clouds can be loaded into spatial databases or analysis software, and rendered images can be fed to detection or classification models. Treat the radiance field as one stage in a pipeline, not the end of it.

Common mistakes

• Skipping pose checks and then blaming the network for blur.
• Training with auto exposure on, which makes the same point look different across photos.
• Using too few images or too little overlap.
• Judging quality only by the training score. Always check held-out views.
• Ignoring floaters and calling the render finished.
• Forgetting that the scene must be static.
• Setting near and far bounds that cut off the scene.
• Treating a radiance field as an accurate measuring tool. Its geometry is a by-product, not a guarantee.
• Comparing methods with different image resolutions or splits.
• Expecting real-time playback from a slow original-style model.

Frequently asked questions

What does NeRF stand for?

NeRF stands for neural radiance field. It is a neural network that stores a 3D scene by returning a color and a density for any point and viewing direction, which lets a renderer produce images from camera positions that were never photographed.

How many photos does NeRF need?

There is no fixed number. Small objects often work with several dozen well-spread photos, while large scenes may need hundreds. What matters most is overlap, sharpness and accurate camera poses. Some methods are designed for very few views, but they lean on learned priors and give up some fidelity.

Is NeRF the same as photogrammetry?

No. Photogrammetry builds explicit surfaces such as meshes and textures from photos. NeRF learns an implicit field and renders images from it. Both start from multiple photos and camera poses, but NeRF is better at view-dependent effects while photogrammetry gives cleaner standard geometry.

Why is the original NeRF slow?

Each pixel needs many samples along its ray, and each sample requires a pass through a neural network. Training repeats this for millions of rays. Faster variants cut cost by replacing most of the network with learned feature grids or by using a different representation.

Can NeRF handle moving objects?

The basic method assumes a static scene. Extensions add a time input or a deformation field to model motion, but they need more data and are harder to train. For casual capture, it is best to keep the scene still.

How is NeRF different from Gaussian splatting?

NeRF stores a scene in a network or feature grid and renders by sampling along rays. Gaussian splatting stores a scene as many explicit 3D Gaussians and renders by projecting them onto the image. The second approach usually renders much faster, while the first is more compact and continuous.

Do I need a GPU to train a NeRF?

In practice, yes. Training involves millions of ray samples and many gradient steps, which a graphics card handles far better than a CPU. A CPU can run tiny demos, but real scenes are impractical without a GPU.

What does NeRF have to do with tabular data?

Directly, little. Conceptually, it shares ideas with tabular models: encoding raw inputs so a network can see detail, fitting from partial observations and filling gaps, and the trade between per-instance training and pre-trained priors. It also links to spatial tables through coordinates and geography.

How do I judge the quality of a NeRF?

Render held-out views and compare them with the real photos using PSNR, SSIM and LPIPS, then inspect the images by eye. Watch for blur, floaters and warped geometry. A single number rarely tells the whole story.

Can NeRF give me a 3D mesh?

Yes, with extra steps. You can sample the density on a grid and extract a surface with a marching-cubes style method. The quality depends on how crisp the learned density is, so surface-oriented variants usually give cleaner results than the original.

Glossary

• Alpha: The opacity of a sample, computed from density and the gap to the next sample.
• Camera pose: Position and orientation of a camera in space.
• Coordinate network: A network that takes coordinates as input and returns values at those coordinates.
• Density: A measure of how much a point blocks or emits light.
• Floaters: Unwanted blobs in empty space in a render.
• Hash grid: A table of learned features addressed by hashed positions.
• Hierarchical sampling: Choosing ray samples in two passes, coarse then fine.
• Implicit representation: A shape or scene defined by a function rather than listed data.
• Intrinsics: Internal camera parameters such as focal length.
• LPIPS: A learned perceptual image distance.
• MLP: Multilayer perceptron, a feed-forward network.
• Novel view synthesis: Making images from new camera positions.
• Positional encoding: Sine and cosine features applied to coordinates.
• PSNR: Peak signal-to-noise ratio, a basic image fidelity score.
• Radiance: Light traveling in a given direction from a point.
• Ray: A line from the camera through a pixel.
• SSIM: Structural similarity index, an image quality score.
• Structure from motion: Estimating camera poses and sparse structure from photos.
• Transmittance: The fraction of light that survives to a point along a ray.
• Volume rendering: Integrating color and density along a ray to get a pixel.

Summary

NeRF learns a 3D scene as a function. A small network, fed with encoded positions and view directions, returns color and density. A differentiable volume rendering step turns those values into pixels, and training matches them to real photos. The result is a continuous, compact scene that gives high-quality novel views, at the cost of slow training and rendering in the original form and a need for accurate camera poses.

For tabular work, NeRF is a conceptual relative rather than a direct tool. Its coordinate encodings, learning from partial observations, and the trade between per-scene fitting and pre-trained priors echo themes in tabular foundation models, and its spatial outputs link to geographic tables. Read next about Gaussian splatting, GIS, the architecture, research and benchmarks pages.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: