Radiance Field

At its heart, a radiance field is a 5D neural field: a continuous function, typically a neural network, that maps a 3D spatial coordinate (x, y, z) and a 2D viewing direction (θ, φ) to a color (RGB) and volume density (σ). This implicit neural representation encodes the entire scene within the network's weights, allowing for the query of light properties from any point in space and any direction.

Volume density is a scalar field output by the neural network that defines the scene's geometry. At any 3D point, density (σ) represents the differential probability of a ray of light terminating (being absorbed or scattered) at that location. It determines:

• Opacity: High density regions appear solid.
• Transparency: Low or zero density regions are empty space.
• Geometry: The surface of an object is defined by the region where density transitions from low to high values.

The directional radiance is the color (RGB) output by the neural network, conditioned on both the 3D location and the 2D viewing direction. This allows the model to capture view-dependent effects such as:

• Specular highlights: The shine on glossy surfaces that changes with viewpoint.
• Reflections: Mirror-like surfaces that show different content from different angles.
• Subsurface scattering: The soft glow of light passing through translucent materials like wax or skin.

To generate a 2D image, the radiance field is rendered using the volume rendering integral. This physically-based equation accumulates color and density along each camera ray:

• Ray marching: The ray is sampled at discrete points.
• Transmittance (T): Computes how much light reaches a sample point, based on accumulated density from earlier samples.
• Alpha compositing: The final pixel color is a weighted sum of colors from all samples along the ray, where weights are a product of transmittance and local density. This integral is differentiable, enabling end-to-end training via gradient descent.

Positional encoding is a critical preprocessing step that transforms low-dimensional input coordinates (3D location, 2D direction) into a higher-dimensional space using a set of sinusoidal functions. This technique, based on the neural tangent kernel theory, enables a standard Multi-Layer Perceptron (MLP) to learn high-frequency details in the scene that it would otherwise fail to represent, such as fine textures, sharp edges, and complex geometric structures.

A radiance field is learned from 2D images via a differentiable optimizer. The core training loop involves:

1. Sampling camera rays from training images with known poses.
2. Querying the neural field to predict color/density for points along each ray.
3. Integrating via volume rendering to synthesize a pixel color.
4. Computing photometric loss (e.g., Mean Squared Error) between the rendered and ground-truth pixel.
5. Backpropagating gradients through the entire rendering pipeline to update the neural network's weights, refining both geometry (density) and appearance (color).

The original and most direct application of a radiance field is photorealistic view synthesis. By learning a continuous 5D function of radiance, a trained model can render novel, unseen camera views of a scene with correct parallax, occlusion, and specular highlights. This is the core capability that enables:

• Virtual cinematography and pre-visualization.
• Free-viewpoint video for sports and entertainment.
• Scene exploration from image collections, such as turning a set of tourist photos into an interactive 3D experience.

Radiance fields provide a powerful implicit representation for 3D scene reconstruction. Unlike traditional multi-view stereo which outputs explicit meshes or point clouds, a NeRF represents geometry as a continuous volume density field. This enables the creation of highly detailed digital twins for:

• Architectural visualization and virtual property tours.
• Industrial asset management, allowing engineers to inspect facilities remotely.
• Cultural heritage preservation, creating permanent, interactive records of historical sites and artifacts.

Radiance fields are a key enabling technology for next-generation AR and VR. They allow for realistic occlusion of virtual objects by real-world geometry and enable persistent AR experiences that remember the layout of a room. Applications include:

• Mixed reality collaboration, where remote participants are rendered into a shared 3D space.
• AR navigation with precise, scene-aware visual overlays.
• Realistic avatars and objects that interact correctly with environmental lighting.

In robotics, radiance fields serve as a rich world model for planning and simulation. A robot can use a NeRF to understand the 3D structure of its environment beyond what is provided by traditional LiDAR or depth sensors. This supports:

• Sim-to-real transfer: Training robotic policies in highly realistic neural simulations before deployment.
• NeRF-SLAM: Performing Simultaneous Localization and Mapping (SLAM) by jointly optimizing a NeRF and camera poses in real-time.
• Safe navigation by predicting occluded areas and understanding material properties from appearance.

The generative extension of radiance fields, known as generative radiance fields, allows for the creation of novel 3D content. This is primarily driven by text-to-3D pipelines that use 2D diffusion models and Score Distillation Sampling (SDS) to optimize a NeRF from a text prompt. This enables:

• Rapid 3D asset creation for games, film, and marketing.
• Creative exploration of 3D shapes and scenes guided by natural language.
• Stylization and editing of existing 3D captures by applying new visual themes.

The volumetric nature of radiance fields makes them suitable for scientific domains where data is inherently 3D. They can be adapted to visualize and interact with complex volumetric data. Potential applications include:

• Medical imaging: Fusing multi-view MRI or CT scans into a continuous, explorable 3D volume.
• Computational fluid dynamics: Visualizing simulation results like smoke or plasma as a continuous field.
• Astrophysics: Modeling and exploring nebular clouds or cosmological simulations with realistic light scattering.

The plenoptic function is the theoretical, complete description of light in a scene. It is a 7D function: P(θ, φ, λ, t, Vx, Vy, Vz) representing the intensity of light of wavelength (λ) traveling in direction (θ, φ) at a viewpoint (V) and time (t). A radiance field is a practical, simplified 5D slice of this function (3D position + 2D direction), which a NeRF learns to approximate. This function underpins all light field and image-based rendering techniques.

An implicit neural representation is a method of encoding a continuous signal (like a 3D shape, image, or audio wave) as the weights of a neural network. Instead of storing explicit data (e.g., a polygon mesh or voxel grid), the signal is defined by a function f(x) → y where f is a neural network (like an MLP). A radiance field is a specific type of implicit neural representation where the network maps 3D coordinates and viewing directions to color and density. This approach offers infinite resolution and memory efficiency.

A 5D neural field is the precise architecture of the function learned in a standard NeRF. It is a continuous mapping: F(x, y, z, θ, φ) → (c, σ) where:

• (x, y, z) is the 3D spatial coordinate.
• (θ, φ) is the 2D viewing direction (often represented as a 3D unit vector).
• c = (r, g, b) is the emitted RGB color (radiance).
• σ is the volume density at that point. This field is differentiable, allowing it to be optimized from 2D images via gradient descent.

Volume density (denoted σ, sigma) is a scalar field output by the radiance field network alongside color. It represents the differential probability per unit length that a ray traveling through space will be absorbed or scattered at a given 3D point. In practice:

• σ = 0: The point is empty space; the ray passes through.
• σ > 0: The point contains some matter; the ray may be blocked. This density field implicitly defines the 3D geometry of the scene. During rendering, densities are integrated along rays to compute pixel colors and occlusion.

Volume rendering is the computer graphics technique used to create a 2D image from a 3D volumetric data field (like a radiance field). For NeRF, it involves:

• Casting a ray from the camera through each pixel.
• Sampling points along the ray.
• Querying the neural network for color (c) and density (σ) at each sample.
• Integrating these values using the volume rendering equation to compute the final pixel color. This process is naturally differentiable, connecting 2D pixel losses to 3D scene parameters.

Ray marching is the specific numerical algorithm used to perform volume rendering in NeRF. It approximates the continuous integral along a ray by discretely sampling and accumulating properties. The core steps are:

1. Stratified Sampling: Initially sample N points evenly along the ray within bounds.
2. Network Query: Evaluate the radiance field at each sample point.
3. Alpha Compositing: Accumulate color and opacity using the formula: C = Σ Ti * αi * ci, where Ti is transmittance and αi = 1 - exp(-σi * δi). Hierarchical sampling improves efficiency by sampling more points in dense regions.