Gaussian splatting

Overview

Gaussian splatting is a way to capture a real scene from photos and display it in 3D at interactive speed. The scene is stored as a large cloud of small, soft, colored blobs called 3D Gaussians. To draw a view, each blob is projected onto the screen as a flat soft spot, and the spots are blended from back to front. Because this is closer to ordinary graphics rasterization than to ray marching through a network, rendering is fast and can reach real-time frame rates on a good GPU.

The method is trained by comparing rendered images with the original photos and adjusting the position, size, orientation, color and opacity of every blob. It reached wide attention in 2023 as a fast alternative to NeRF, keeping similar visual quality while training in minutes and rendering in real time. The trade-off is file size and a less tidy surface: a scene may hold millions of blobs, and the blobs do not form a clean mesh by themselves.

What it is

3D Gaussian splatting is a point-based scene representation. Each primitive is an anisotropic 3D Gaussian, which you can picture as a soft ellipsoid, like a tiny fuzzy egg that can be stretched, squashed and rotated. Each Gaussian carries these attributes.

• A center position in 3D space.
• A scale along each of three axes, which sets how big and how stretched it is.
• A rotation, usually stored as a quaternion, which sets its orientation.
• An opacity, which sets how solid it is.
• A color, often stored as spherical harmonic coefficients so that the color can change with viewing direction.

A whole scene is a long list of such records, much like a table with one row per Gaussian and a fixed set of columns. That analogy to a table matters later in this article.

An everyday analogy

Think of a painting made of millions of translucent sticker dots. Each dot is oval, has its own tilt and tint, and fades at the edges. If you stand in one place and look, the dots in front cover the dots behind, and the blend looks like a photograph. If you move, the dots shift in your view and you see the scene from the new angle. Training is the process of placing, resizing and tinting the stickers until the view from each original photo matches that photo.

What goes in and what comes out

• Input: a set of photos, plus camera poses and usually a sparse point cloud from a structure-from-motion step.
• Output: a list of Gaussians, which can be stored in a single file and shown in a viewer.

Explicit versus implicit

Unlike a radiance field stored in network weights, a Gaussian scene is explicit. You can open the list, count the primitives, delete the ones in a region, and move or crop them. This makes editing and inspection much easier, although the numbers themselves are not easy to read.

Where it came from

Gaussian splatting grew from older point-based graphics.

• Point splatting in classic rendering drew surfaces as many small oriented discs.
• Elliptical weighted average splatting handled anti-aliasing for such discs.
• Structure from motion and multi-view stereo provided sparse points and camera poses from photos.
• Differentiable rendering made it possible to optimize scene parameters by gradient descent.
• Radiance fields such as NeRF (2020) showed that photo-realistic view synthesis could be learned from images, but they were slow.

The paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering" appeared in 2023. It combined three ideas: a Gaussian primitive that can be projected analytically, an optimization that adds and removes Gaussians as it trains, and a fast tile-based GPU rasterizer that is also differentiable. The combination matched the quality of leading radiance field methods while rendering at real-time speeds.

Many follow-ups appeared quickly.

• Anti-aliasing and scale-aware variants to avoid artifacts at different zoom levels.
• Compression methods to shrink file sizes.
• Dynamic scene versions that add motion over time.
• Surface-oriented versions that flatten Gaussians to recover cleaner geometry.
• Large-scene methods that split a city-sized capture into blocks.
• Viewers that run in a web browser or on mobile devices.
• Generation methods that produce Gaussians from text or a single image.

How it works

Step 1: collect photos and camera poses

As with NeRF, you take many overlapping photos of the scene and run a structure-from-motion tool. It returns a pose for each photo and a sparse set of 3D points that match features across images. Good overlap and sharp photos make a large difference.

Step 2: initialize the Gaussians

The sparse points become the first Gaussians. Each starts with a small isotropic size, a neutral color and moderate opacity. This starting cloud is crude, but it sits roughly where the scene surfaces are, which gives the optimizer a head start.

Step 3: project Gaussians to the image

For a given camera, each 3D Gaussian is projected onto the image plane. The projection of a 3D Gaussian is approximately a 2D Gaussian, so the math stays simple. The result is a soft elliptical spot with a center, a shape and a depth.

An analogy: shining a flashlight through a fuzzy glass bead onto a wall makes a soft oval of light. Where you stand and which way the bead is turned decide the shape of that oval.

Step 4: sort and blend

Spots are sorted by depth and blended from front to back. Each spot contributes color weighted by its opacity and by how much light has not yet been blocked by the spots in front. This is the same compositing formula used in volume rendering, but evaluated over a sorted list of splats instead of dense samples along a ray.

Step 5: tile-based rasterization

To be fast, the image is divided into small tiles, for example 16 by 16 pixels. Each Gaussian is assigned to the tiles it touches. Gaussians are sorted per tile, and each tile is processed in parallel by a block of GPU threads. This is why the method reaches real-time speed. It avoids evaluating a neural network for millions of sample points.

Step 6: compare with the photo and update

The rendered image is compared with the training photo. The loss is usually a mix of an L1 pixel loss and a structural similarity term. Because every step is differentiable, gradients flow back to every attribute of every Gaussian that was visible: position, scale, rotation, opacity and color. An optimizer updates them.

Step 7: adaptive density control

During training, the number of Gaussians does not stay fixed. Every so often the method adjusts the cloud.

• Clone: where a small Gaussian sits in an under-reconstructed region and its position gradient is large, it is duplicated to fill in detail.
• Split: where a large Gaussian covers a region that needs finer detail, it is replaced by two smaller ones.
• Prune: Gaussians that are nearly transparent or far too large are removed.
• Opacity reset: opacity is periodically lowered so that useless Gaussians can fade out and be pruned.

Think of a sculptor adding clay where detail is missing, carving off clay where there is too much, and occasionally wiping the surface to see what is truly needed.

Step 8: view-dependent color

To capture gloss and reflections, color is stored as spherical harmonics. These are a compact set of smooth functions on a sphere. A low order gives mostly diffuse color. A higher order lets the color change as you look from different directions. Training often starts with low order terms and adds higher ones gradually.

Step 9: render new views

At display time, pick any camera, project all Gaussians, sort, and blend. Scenes with a few million Gaussians can be drawn at interactive speed on a decent GPU.

A simplified sketch in code

The following outlines the data layout and the blending formula in Python style. Real systems use fused GPU kernels, so this is for understanding only.

import numpy as np

class Gaussians:
    def __init__(self, n):
        self.pos = np.zeros((n, 3))        # centers
        self.log_scale = np.zeros((n, 3))  # log of axis lengths
        self.quat = np.tile([1.0, 0, 0, 0], (n, 1))  # rotations
        self.opacity_logit = np.zeros(n)   # before sigmoid
        self.sh = np.zeros((n, 16, 3))     # spherical harmonic coefficients

    def opacity(self):
        return 1.0 / (1.0 + np.exp(-self.opacity_logit))

def composite(colors, alphas):
    """Front-to-back blending for one pixel.
    colors: (k, 3) sorted near to far, alphas: (k,) in [0, 1]."""
    out = np.zeros(3)
    transmittance = 1.0
    for c, a in zip(colors, alphas):
        out += transmittance * a * c
        transmittance *= (1.0 - a)
        if transmittance < 1e-3:
            break
    return out

Storing log-scale and an opacity logit keeps scale positive and opacity between zero and one without extra constraints.

Key concepts and terms

A worked example

Suppose you want to turn a short phone video of a market stall into a viewable 3D scene.

Preparation

1. Record a slow video that circles the stall at two heights. Avoid fast motion to limit blur.
2. Extract frames at a steady rate, for example a few per second, and drop blurry ones.
3. Run structure from motion to get camera poses and a sparse cloud. Check that most frames registered.
4. Hold out some frames for testing.

Configuration

A configuration with placeholder names might look like this.

data:
  images: data/stall/images
  sparse: data/stall/sparse
  test_every: 8
model:
  sh_degree: 3
  init_scale: auto
train:
  iterations: 30000
  position_lr: 0.00016
  opacity_reset_every: 3000
  densify_from: 500
  densify_until: 15000
  densify_every: 100
  densify_grad_threshold: 0.0002
  lambda_ssim: 0.2
export:
  format: ply
  path: out/stall.ply

What training looks like

The numbers below illustrate the typical pattern and are not measured results.

iter 1000    gaussians 120k    psnr 21
iter 7000    gaussians 900k    psnr 26
iter 15000   gaussians 1.8M    psnr 28
iter 30000   gaussians 1.9M    psnr 29

The number of Gaussians climbs during the densification phase, then stabilizes after it ends. Quality rises quickly early on and then slowly.

A numeric walk-through of blending

Take one pixel covered by three splats sorted near to far.

• Start with transmittance 1.0.
• A adds 1.0 times 0.50 times red, so (0.50, 0, 0). Transmittance becomes 0.50.
• B adds 0.50 times 0.60 times green, so (0, 0.30, 0). Transmittance becomes 0.20.
• C adds 0.20 times 0.90 times blue, so (0, 0, 0.18). Transmittance becomes 0.02.

The pixel color is about (0.50, 0.30, 0.18), a muted red with some green and a touch of blue. Splat order matters. If A and C were swapped, the blue would dominate. This is why sorting is essential, and why wrong sorting causes popping artifacts when the camera moves.

Evaluation and export

Render the held-out frames and compute PSNR, SSIM and LPIPS. Inspect the images for floaters and blur. Export the Gaussians to a file and open it in a viewer. A file with a couple of million Gaussians can be hundreds of megabytes, so consider compression before sharing.

Strengths and limitations

Strengths

• Real-time rendering on a capable GPU.
• Training in minutes for many scenes.
• High visual quality with fine detail and view-dependent color.
• Explicit primitives that can be cropped, moved, deleted or combined.
• Fits naturally into graphics engines because it uses rasterization.
• Easy to inspect: the scene is a list you can count and filter.
• Many ecosystems and viewers now exist, including web viewers.

Limitations

• Large files, since each of millions of Gaussians has dozens of numbers.
• The cloud of blobs is not a watertight surface. Mesh extraction needs extra work.
• Quality drops away from the training views, and the scene can look odd from unseen angles.
• Thin structures, mirrors and transparent objects are still hard.
• Initialization from sparse points can fail in textureless regions.
• Popping and aliasing can appear when zooming or moving the camera far from the training path.
• Static scene assumption in the basic form.
• Memory use on the GPU grows with the number of Gaussians.
• Licensing and patent questions can matter for commercial use, so check the terms of any implementation you rely on.

How it compares with alternatives

The practical rule: choose splatting when interactive display matters, choose photogrammetry or lidar when measured geometry matters, and choose a radiance field when you want a compact continuous model or are doing research.

Role in tabular foundation models

Gaussian splatting works on photos and 3D space, not on spreadsheets. Still, its structure resembles tabular data in ways that help explain how tabular foundation models think, and it connects to spatial data work in practice.

A scene is a table

A trained splat scene is a table. Each Gaussian is a row. The columns are position x, y and z, three scales, four quaternion values, an opacity, and a block of color coefficients. All columns are numeric, with very different ranges and meanings. Position is in scene units, opacity is in zero to one, and rotations must be normalized. This is the classic situation of a tabular model: mixed scales, columns with different semantics, and no column names unless you add them. Models that learn from raw tables must infer what a column means from its values alone. See the architecture page for how a tabular foundation model handles that.

Row order does not matter

The Gaussians in a scene have no natural order. Shuffling the rows gives the same picture, since sorting by depth happens at render time. A good tabular model also treats rows as an unordered set, and many tabular foundation models are built so that permuting rows does not change the answer. This permutation invariance is a shared design principle.

Analysis of splat tables

Because a scene is a table, ordinary tabular methods apply to it.

• Anomaly detection can flag floaters: Gaussians that are large, isolated and low in opacity.
• Clustering can group Gaussians into objects or regions.
• Dimensionality reduction such as UMAP can reveal structure in color coefficients.
• Regression or classification can predict a semantic label per Gaussian, for example ground, building or vegetation.
• Imputation can fill in attributes that are missing for parts of the scene.

These uses are conceptual illustrations of how tabular tools could be applied, not claims about any specific product.

Compression as tabular modeling

Shrinking a splat file is a table compression problem: reduce the precision of columns, remove near-duplicate rows, and learn shared codes for color blocks. Ideas such as quantization, clustering of similar rows and learned codebooks apply directly. Tabular models that capture column dependencies could, in principle, predict one attribute from others and store only the residual.

Missing values and partial observation

Splatting reconstructs a scene from partial views. Regions never seen have no Gaussians or poor ones. This parallels missing values in a table: the model must decide what to do with cells that are unknown. Tabular foundation models use masking during pre-training to learn this skill, as in masked cell modeling and imputation networks.

Optimization per instance versus in-context learning

Splatting fits each scene by gradient descent for tens of thousands of steps. A tabular foundation model such as TabPFN or TabICL works in a different way: it is pre-trained once on many tables, many of them synthetic, then reads a new table in context and predicts in a single forward pass. Splatting is a good example of the other end of the spectrum, where all the learning happens per instance. Both ends have a place. Per-instance fitting gives high fidelity for one case. Pre-trained priors give speed and generality.

Spatial tables and geography

Real-world splat scenes can be geo-referenced. A captured site can be tied to coordinates and joined with tables of assets, parcels or sensors. Spatial analysis of such data is the work of GIS. A model that reads tabular columns can then use quantities derived from the scene, such as height above ground or visible area, as features. The same applies to smart-city and urban uses, such as urban intelligence and digital twins.

Continual learning

A captured place changes: new buildings, new signs, seasonal changes. Updating a splat scene with fresh photos while keeping old content mirrors the continual learning problem. Replay of old views and constraints on how far Gaussians may move are the practical answers, much as in experience replay and EWC for tabular models.

Sector and semantic classification

Adding semantic labels to Gaussians, for example by lifting labels from a segmentation model such as SAM or features from CLIP, is a classification task on a table. It is similar in spirit to assigning a semantic sector to a table based on its content, as described under semantic sector heads.

What not to claim

Gaussian splatting is not a tabular foundation model, and a tabular foundation model does not render scenes. The comparison is useful for ideas: tables of heterogeneous numeric columns, unordered rows, partial observation, and the contrast between per-instance fitting and pre-training. For this site's approach and results, see /model/research and /model/benchmarks.

Practical guidance and best practices

Capture

• Use sharp, well-exposed frames. Blur harms both pose estimation and training.
• Lock exposure and white balance when possible.
• Cover the scene from many angles with strong overlap.
• Keep people and vehicles out of the scene or accept that they will become artifacts.
• Capture at several distances. Close views give detail, far views give context.
• Avoid textureless surfaces and heavy reflections if you can.

Poses and initialization

• Check how many images registered and view the camera path.
• If the sparse cloud is poor, consider random initialization with a bounded region, but expect more floaters.
• For large scenes, split the capture into blocks and merge later.

Training

• Start at reduced resolution to test the pipeline.
• Watch the Gaussian count. A runaway count wastes memory.
• Tune the densification threshold: lower values add more detail but also more size.
• Use a held-out set and a perceptual score as well as PSNR.
• Consider anti-aliasing or scale-aware options if you will view at several zoom levels.

Cleaning

• Remove floaters by cropping to a region of interest.
• Filter out Gaussians with very low opacity or extreme scale.
• Use anomaly detection on the attribute table to find outliers systematically.

Delivery

• Compress before sharing: quantize attributes, lower the spherical harmonic order and prune.
• Use a viewer matched to the target device. Web and mobile viewers may need a smaller scene.
• Keep the source photos, poses and configuration so you can retrain.

Hardware

• A modern GPU is needed for training and for smooth viewing of large scenes.
• GPU memory is the common limit. Reduce image size or the Gaussian count if you run out.

Governance

• Photos can contain faces, plates and private property. Blur or remove them as required, store raw images securely, and record who may use the output.

Reproducibility

• Fix seeds, log configuration, record software versions and keep the test split fixed across experiments.

Common variants at a glance

Reading a result critically

A smooth demo can hide weak spots. Before you trust a splat scene, run these checks.

• Move the camera far from the capture path. Does the scene break up?
• Zoom in and out. Do edges shimmer or fatten?
• Look at thin objects, glass and mirrors.
• Look for floaters in the sky and near the edges of the covered area.
• Compare a few measured distances with known values if accuracy matters.
• Check the file size against the target device.

A simple project plan

1. Define the goal: a viewer, a video, an input to analysis, or a base for a mesh.
2. Capture with a checklist and keep notes on conditions.
3. Estimate poses and drop images that failed to register.
4. Train a quick baseline at low resolution.
5. Inspect held-out renders and fix capture or pose problems.
6. Train the final scene and clean it.
7. Compress, export and archive the inputs and settings.

Questions to ask before you start

• Is the scene static for the whole capture?
• Do you need measured geometry or only a convincing look?
• Who will view the result, and on what device?
• How large a file can you deliver?
• How will you store and version the raw photos and poses?

Where the output can feed other systems

A splat file is a list of attributes, so it can be read by scripts and loaded into data tools. Depth maps and rendered images can feed detection and classification models. Positions can be placed in a geographic frame and joined with tables of assets, which is the kind of work covered under GIS.

Cost and effort

Costs fall mostly in capture time and GPU time. A small scene may need minutes to shoot and minutes to train. A large site may need several sessions, block-wise training and a lot of storage. Plan for retakes, since a bad capture is cheaper to redo early than to hide with settings later.

Data governance for captured scenes

Photos of places can include faces, plates and private property. Treat captured data like any other sensitive data. Blur or remove people where required, store raw photos securely and record who may use the trained scene.

Common mistakes

• Training on poorly registered images and then blaming the method.
• Letting the Gaussian count grow unchecked.
• Judging quality only from training views.
• Ignoring floaters and background clutter.
• Sharing huge uncompressed files and expecting smooth viewing on a phone.
• Assuming the cloud is a clean surface suitable for measurement.
• Comparing methods at different resolutions or with different splits.
• Capturing with moving people or changing light and expecting a clean result.
• Using too high a spherical harmonic order for scenes that are mostly diffuse, which adds size for little gain.
• Treating the output as ground truth geometry without checking it against measurements.

Frequently asked questions

What is Gaussian splatting in simple terms?

Gaussian splatting captures a scene from photos as a huge cloud of small soft colored ovals in 3D. To show the scene, the ovals are flattened onto the screen and blended from back to front, which is fast enough for real-time viewing from any angle.

How is Gaussian splatting different from NeRF?

NeRF stores the scene in a neural network and renders by sampling points along rays. Gaussian splatting stores the scene as explicit Gaussians and renders by projecting them onto the image. Splatting is usually much faster to render, while a radiance field is typically more compact.

Is Gaussian splatting better than photogrammetry?

It depends on the goal. Splatting often looks more realistic, especially for fine detail and shiny surfaces, and it renders in real time. Photogrammetry gives meshes with clean surfaces that suit measurement and standard 3D tools. Many projects use both.

Do I need special equipment to capture a scene?

No. A phone or camera is enough, provided the photos are sharp and overlap well. A steady, slow capture path matters more than the device. Extra hardware such as lidar can help with geometry but is not required.

Why are splat files so large?

Each Gaussian stores position, scale, rotation, opacity and many color coefficients, and a scene can hold millions of them. Compression methods reduce size by lowering precision, removing near-duplicates and sharing codes, at some cost in quality.

Can I edit a Gaussian splat scene?

Yes, more easily than a neural field. Since the scene is an explicit list, you can select Gaussians in a region, delete them, move them, or merge scenes. Precise edits such as changing materials are harder.

Can Gaussian splatting produce a mesh?

Not directly. The Gaussians are soft blobs rather than surfaces. Several methods flatten or regularize the Gaussians so that a surface can be extracted, but the quality varies and extra processing is required.

Does Gaussian splatting work for moving scenes?

The basic form assumes a static scene. Extensions add a time dimension or deformation fields so that Gaussians move over time. They need more data, typically synchronized video from several cameras, and are harder to train.

What does Gaussian splatting have to do with tabular data?

A splat scene is itself a table with one row per Gaussian and numeric columns, so tabular tools such as anomaly detection, clustering and compression apply to it. Conceptually it also shares themes with tabular foundation models: unordered rows, partial observation and the contrast between per-instance fitting and pre-training.

Is it safe to use captured scenes of public places?

Captured photos can include people, license plates and private property, so follow local rules and your own data policy. Blur or remove personal details, secure the raw images and keep a record of permitted uses.

Glossary

• Adaptive density control: The process of cloning, splitting and pruning Gaussians while training.
• Alpha compositing: Blending semi-transparent layers using their opacity.
• Anisotropic: Having different size or spread along different directions.
• Covariance matrix: A matrix that describes the shape and orientation of a Gaussian.
• Differentiable rasterization: A fast drawing method that still allows gradients to flow.
• Floaters: Stray Gaussians in empty space.
• Gaussian: A bell-shaped function; in 3D, a soft ellipsoid.
• Opacity: How solid a Gaussian is.
• Pruning: Removing Gaussians that add little.
• Quaternion: Four numbers that encode a 3D rotation.
• Rasterization: Turning geometric shapes into pixels.
• Spherical harmonics: Basis functions on a sphere used to model view-dependent color.
• Splat: The projected 2D footprint of a Gaussian on the screen.
• Structure from motion: Estimating camera poses and sparse points from photos.
• Tile: A small block of the image processed together on the GPU.
• Transmittance: The fraction of light not yet blocked while blending.

Summary

Gaussian splatting represents a scene as a cloud of millions of soft 3D ellipsoids, each with position, size, orientation, opacity and view-dependent color. Training adjusts them to match photos, using a differentiable tile-based rasterizer and adaptive cloning, splitting and pruning. The payoff is real-time rendering and fast training with high visual quality. The costs are large files, no built-in clean surface and weaker quality away from the captured views.

For tabular work, a splat scene is literally a table of numeric rows, so tabular analysis, compression and anomaly checks apply, and its themes of unordered rows, partial observation and per-instance fitting versus pre-training echo tabular foundation models. Continue with NeRF, GIS, the architecture, research and benchmarks pages.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: