What is Gaussian splatting?
Definition
The name aptly describes the technique: “splashes represented by small spots whose color distribution from the center to the edges follows a Gaussian distribution.” 3D Gaussian Splatting, as outlined in the technical paper, allows for real-time rendering of photorealistic scenes learned from small image samples.
Comparison with Other Approaches
Photogrammetry was the primary scanning technique until recently, generating 3D polygons from multiple viewpoint images. However, it struggles with sceneless edges like skies and cannot accurately handle reflections or mirrors.
The Rise of NeRF
Neural Radiance Fields emerged to overcome photogrammetry’s limitations by training models to implicitly represent scenes as volumetric density and color data. While effective for open scenes and reflective surfaces, NeRF rendering is relatively slow.
What is 3D Gaussian Splatting?
Announced in August 2023, this technique enables real-time rendering using only a few multi-viewpoint images. The 3D space consists of Gaussians, with each defined by position (XYZ), covariance (scale/stretch), color (RGB), and alpha (transparency).
How It Works in Detail
Structure from Motion: The COLMAP library generates point clouds from input images in seconds, estimating camera parameters by matching pixels across images. Conversion to Gaussians: Each point converts into a Gaussian for rasterization. Training: Using Stochastic Gradient Descent — rasterizes Gaussians into images via differentiable rendering, computes loss comparing rendered versus ground truth images, adjusts Gaussian parameters, and automates densification and pruning.
Differentiable Gaussian Rasterization
The technique is fast and differentiable, projecting each Gaussian to 2D, sorting by depth, then blending them pixel-by-pixel front to back.
Implications for 3D Scene Representation
Advantages: High-quality photorealistic scenes, real-time rasterization, relatively fast training. Disadvantages: High VRAM usage (4GB viewing, 12GB training), large disk requirements (1GB+ per scene), incompatible with existing rendering pipelines, currently static.
Current Adaptations: Remote viewer, WebGPU viewer, WebGL viewer, Unity viewer, Optimized WebGL viewer.
Will We See Full Production Implementation?
Probably yes. The main bottleneck — sorting millions of Gaussians — is efficiently handled by CUB radix sort in the original CUDA implementation, and similar performance could be achievable in other pipelines with sufficient development effort.