Relightable Neural Radiance Field

The core function of a Re-NeRF is to decompose the observed scene into its intrinsic, lighting-invariant properties. This is typically modeled by a neural network that outputs:

• Albedo: The base color or reflectance of a surface point, independent of lighting.
• Surface Normals: The orientation of the geometry at each point.
• Roughness/Metallic: Material properties that define how light interacts with the surface (e.g., via a neural BRDF). These properties, combined with a separate lighting model, are used to synthesize the final rendered color for any given novel light source.

To enable relighting, the system must incorporate a model of illumination. Common approaches include:

• Spherical Harmonics (SH): A compact basis for representing low-frequency environmental lighting.
• Learnable Latent Lighting Codes: A neural network encodes the illumination condition of each input image into a latent vector; novel lighting is achieved by interpolating or providing a new vector.
• Explicit Light Probes: Using High Dynamic Range (HDR) environment maps as direct inputs to a differentiable rendering equation. The lighting model is combined with the intrinsic properties via a shading function (e.g., based on Physically Based Rendering principles) to compute the final pixel color.

Training a Re-NeRF requires a differentiable rendering pipeline. This allows gradients to flow from the final rendered image back to the network parameters for the geometry, materials, and lighting.

• The network predicts density (for geometry) and the intrinsic material properties for any 3D coordinate.
• A rendering integral (like in standard NeRF) accumulates these properties along camera rays.
• A shading model (e.g., a microfacet model) is applied during this integration, using the predicted lighting, to produce the final pixel color.
• The loss is computed between this rendered color and the ground-truth input image, enabling joint optimization of all components.

Advanced Re-NeRFs model global illumination effects, not just direct lighting. This includes:

• Shadows: Cast shadows are modeled by having the rendering integral account for light occlusion from the scene's own learned geometry.
• Indirect Lighting: Light bouncing off surfaces (color bleeding) can be approximated. Some methods use a second radiance network that encodes the view-dependent outgoing radiance, which is itself a function of the global illumination.
• Precomputed Radiance Transfer (PRT): Inspired by classical graphics, some neural methods precompute or co-learn how light is transferred within the scene to enable real-time relighting with complex effects.

Training a robust Re-NeRF requires a dataset with varied lighting. Common capture setups include:

• Fixed Object, Moving Lights: The object remains static while a single light source (or a light stage) moves around it. This provides clean signal for disentangling appearance from lighting.
• In-the-Wild Images: Methods using internet photos must handle inconsistent lighting, requiring robust optimization to find a consensus geometry and albedo. The network must maintain multi-view consistency for geometry and albedo across all lighting conditions, which is a key challenge that separates Re-NeRF from standard view synthesis.

The primary output of a Re-NeRF is a scene representation that enables:

• Novel View Synthesis under Novel Lighting: Render the scene from any viewpoint under any new HDRI environment map.
• Material Editing: Modify the extracted albedo or roughness and see the change consistently under all lighting.
• Lighting Design: Virtually place new light sources (e.g., a lamp) in the captured scene and see realistic shadows and highlights.
• Asset Creation: The extracted texture maps (albedo, normal, roughness) can be exported to standard 3D formats for use in game engines or digital twin pipelines.

This technology drives next-generation AR/VR experiences by allowing virtual objects to be convincingly composited into real environments with consistent lighting.

• Dynamic Relighting: Virtual objects cast and receive shadows correctly as the user moves a light source or the environment lighting changes.
• Consistent Material Appearance: Ensures virtual materials (metal, plastic, fabric) react believably to the captured environment's lighting, crucial for product visualization and virtual try-on.
• Enables the creation of persistent, relightable AR scenes that can be revisited under different conditions.

Online retail leverages Relightable NeRFs to provide customers with hyper-realistic, interactive product views.

• View-Anywhere, Light-Anywhere: Customers can rotate a product and change its lighting environment (e.g., 'see this sofa in daylight' or 'under warm lamp light') from a single capture session.
• Material Accuracy: Accurately represents how different materials (glossy leather, brushed metal, matte fabric) interact with light, reducing purchase uncertainty.
• Dramatically reduces the cost and time associated with traditional studio photography, which requires reshoots for every lighting setup.

Architects and lighting designers use Relightable NeRFs for predictive design and client presentations.

• Lighting Design Validation: Import a captured NeRF of a space and simulate new lighting fixtures, window placements, or material changes to preview their impact in the exact context.
• Historical Preservation: Create a relightable archive of heritage sites, allowing study and presentation under various illuminations without risking damage to the original.
• Integrates with Building Information Modeling (BIM) workflows to provide photorealistic context for new designs within existing conditions.

For robotics and autonomous vehicles, Relightable NeRFs provide a critical source of synthetic training data with perfect ground truth.

• Lighting-Robust Perception: Training perception models (for object detection, segmentation) on a scene rendered under thousands of novel, physically plausible lighting conditions improves real-world robustness.
• Simulation Fidelity: Creates highly realistic simulation environments from real-world scans where lighting (time of day, weather) is a controllable parameter, reducing the sim-to-real gap.
• Enables testing of sensor systems (cameras, lidar) under extreme or rare lighting scenarios that are difficult or dangerous to capture physically.

The process of estimating the underlying scene properties—such as geometry, materials, and lighting—from a set of 2D images. This inverts the traditional graphics pipeline to solve for the inputs (scene parameters) given the outputs (rendered images). It is the core computational problem that relightable NeRFs address.

• Key Inputs: Multi-view images, sometimes with known or estimated camera poses.
• Estimated Outputs: 3D geometry, Bidirectional Reflectance Distribution Function (BRDF) parameters, and environmental lighting.
• Applications: Digital asset creation, visual effects, and augmented reality.

A mathematical function that defines how light is reflected at an opaque surface. It describes the ratio of reflected radiance to incident irradiance as a function of the illumination angle and viewing angle.

• Core Model: Physically-based BRDFs, like the microfacet model, are energy-conserving and used in Physically Based Rendering (PBR).
• Parameters: Typically includes albedo (base color), roughness, and metallic properties.
• In Relightable NeRFs: The scene's appearance is often decomposed into a neural representation of geometry and a BRDF-like function, allowing for physics-based relighting.

A Bidirectional Reflectance Distribution Function that varies across the surface of a material. This allows for the representation of complex, non-uniform appearance properties like scratches, stains, rust, fabric weave, or wood grain.

• Representation: Often stored as a set of texture maps (albedo, normal, roughness, etc.).
• Capture: Requires specialized setups like gonioreflectometers or multi-light photographic techniques such as photometric stereo.
• Relation to NeRF: A relightable NeRF can be seen as modeling an implicit, continuous SVBRDF across a neural scene representation.

A controlled illumination system, typically a dome or sphere equipped with hundreds of programmable LED light sources and synchronized cameras. It is used to capture the complete reflectance field of an object or human face.

• Function: By sequentially illuminating the subject from many known directions and capturing images, it records how the subject scatters light from every angle.
• Output: Enables ultra-realistic relighting and appearance modeling for digital doubles in film and games.
• Data for AI: Light stage captures provide the high-quality, ground-truth multi-illumination data used to train and evaluate relightable neural representations.

A rendering framework that allows the calculation of gradients with respect to scene parameters (like vertex positions, material properties, or lighting). This enables the use of gradient-based optimization (e.g., backpropagation) for inverse rendering tasks.

• Mechanism: Provides a mathematical bridge between pixels in a 2D image and the 3D scene parameters that generated them.
• Enabling Technology: Fundamental to optimizing Neural Radiance Fields (NeRF) and relightable variants, as it allows the model to learn from image reconstruction loss.
• Libraries: Frameworks like PyTorch3D and Mitsuba 2 implement differentiable versions of rasterization and ray tracing.

The computer vision task of separating an image of an object into its intrinsic components, most commonly albedo (base surface color), shading (effects of lighting and geometry), and often surface normals. This disentanglement is crucial for relighting.

• Goal: To isolate the invariant material properties from the variable lighting conditions.
• Challenges: Inherently ill-posed (one image, many possible decompositions). Solved using priors, multi-view images, or known lighting.
• In Neural Fields: Relightable NeRFs perform a 3D, continuous version of appearance decomposition, learning separate functions for geometry, reflectance, and lighting.