NeRF in the Wild (NeRF-W)

A per-image latent vector that captures variable global scene properties, allowing a single static NeRF model to represent a collection of images under different conditions.

• Purpose: Models changes in global illumination, weather, and post-processing (e.g., white balance, exposure) across the input photo set.
• Mechanism: Each training image is assigned a learnable embedding vector. This vector is concatenated with the spatial coordinate and viewing direction before being fed into the radiance (color) MLP.
• Result: The network learns to modulate its color output based on this embedding, enabling the reconstruction of all input photos without conflating lighting changes with permanent geometry.

An auxiliary neural output that models per-ray uncertainty, used to identify and separate transient objects (e.g., people, vehicles) from the static scene.

• Purpose: Robustly reconstruct the permanent, static background in the presence of occluders and moving objects that are inconsistent across images.
• Mechanism: Alongside density (σ) and color (c), the network outputs a transient radiance and transient beta (uncertainty) value for each sampled point. High beta values indicate regions likely to contain transient content.
• Loss Function: Uses a negative log-likelihood loss under a Laplacian distribution. This down-weights the contribution of high-uncertainty (transient) pixels during training, preventing them from corrupting the static scene model.

The core capability to factor a scene into a canonical static model and variable transient elements, which are not baked into the permanent geometry.

• Outputs: For any 3D point, the model produces:
  • Static density & radiance: The permanent scene.
  • Transient density & radiance: Temporary objects.
  • Transient uncertainty: Confidence in the transient model.
• Rendering: The final pixel color is a composite of the rendered static and transient components.
• Application: Enables the creation of clean 3D reconstructions of landmarks from tourist photos, removing crowds and vehicles to reveal the underlying structure.

The training process that works with in-the-wild photo collections, which lack controlled lighting, consistent timing, and accurate camera parameters.

• Input Data: Internet photo collections (e.g., Flickr images of a landmark) with approximate SfM camera poses. No requirement for fixed lighting or a cleared scene.
• Joint Optimization: Simultaneously learns:
  1. The static 5D radiance field.
  2. Per-image appearance embeddings.
  3. Per-point transient fields.
• Robustness: The uncertainty-weighted loss provides inherent robustness to outliers and inconsistencies in the training data, which are common in unstructured sets.

The separation of lighting and color from underlying shape, resulting in a 3D geometry model that is consistent despite varying photographic conditions.

• Achieved By: The appearance embedding absorbs variability that is correlated across an entire image (like time of day), while the density field (σ) is optimized using all images and must explain geometry consistently.
• Benefit: Produces a more accurate and coherent 3D structure than a standard NeRF trained on the same data, which would often create "floaters" or blurry geometry to explain lighting changes.
• Verification: The static density field can be extracted as a point cloud or mesh that represents the scene without lighting artifacts.

The ability to render novel views of the scene with user-specified appearance or with transient objects removed.

• Novel View Synthesis: Generate images from new camera positions.
• Appearance Editing: By manipulating the appearance embedding vector, one can simulate different lighting conditions (e.g., sunset vs. noon) on the same static geometry.
• Transient Removal: By rendering only the static component, the output shows a clean view of the scene without people, cars, or other occlusions.
• Uncertainty Visualization: The rendered beta (uncertainty) map highlights image regions where the model encountered inconsistent data, providing insight into scene dynamics.

NeRF-W is ideal for digitally preserving historical sites and monuments from tourist photos or archival imagery, where conditions are never controlled.

• Appearance embeddings separate the permanent structure from variable illumination (sunlight, shadows) and weather effects.
• Uncertainty estimation filters out transient objects like visitors, scaffolding, or modern signage, leaving a clean 3D model of the static heritage asset.
• Enables the creation of digital twins for sites that cannot be closed to the public or meticulously scanned with professional equipment.

Building large-scale, navigable 3D maps from dashcam or street-view imagery requires handling dynamic elements and changing time-of-day.

• Models the static urban geometry (buildings, roads, signs) while isolating moving cars, pedestrians, and seasonal changes like fallen leaves.
• The decomposed representation provides a stable world model for simulation and testing of autonomous vehicles and drones.
• Appearance latent codes can represent different lighting conditions (day, night, dusk), allowing planners to visualize a location under any scenario.

In visual effects, artists often need to integrate CGI into footage shot under inconsistent lighting or with unwanted foreground elements.

• NeRF-W can reconstruct the 3D set or location from the raw footage itself.
• The static scene model allows for perfect camera tracking and realistic CGI lighting integration.
• Transient objects (like a crew member accidentally in shot) are identified and can be removed, with the model plausibly inpainting the occluded background.

For persistent AR experiences that must work across different times and weather, NeRF-W provides a stable geometric anchor.

• Creates a persistent 3D map of a location (e.g., a city square, a store interior) from many user-contributed photos.
• AR content can be attached to the recovered static geometry, ensuring it stays locked in place regardless of current lighting or temporary decorations.
• The system is more reliable than traditional visual SLAM in highly dynamic environments because it explicitly reasons about transience.

Platforms like Google Earth or Mapillary benefit from algorithms that can fuse billions of unstructured photos into coherent 3D models.

• NeRF-W's architecture is designed for the 'in the wild' data problem: photos from different devices, seasons, and times of day.
• It outperforms traditional Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines in cases with high appearance variation, as it explicitly models and disentangles these factors.
• The output is a clean, textured 3D model with metadata about lighting conditions present in the source imagery.

NeRF-W provides an intuitive pipeline for editing real-world scenes by manipulating its decomposed components.

• Because the model separates static radiance, appearance, and transient uncertainty, editors can:
  • Re-light the scene by interpolating appearance embeddings.
  • Remove objects flagged as transient (e.g., litter, a parked car) by rendering only the static components.
  • Insert new objects with consistent lighting by using the recovered appearance parameters for shading.
• This moves beyond simple 2D inpainting to a semantically meaningful 3D editing workflow.

A per-image latent vector learned during NeRF training that captures variable scene properties like lighting, weather conditions, and post-processing effects. In NeRF-W, these embeddings allow a single static model to reconstruct a collection of photos taken under different illumination, separating consistent geometry from transient appearance.

• Purpose: Encodes the photometric style of each input image.
• Mechanism: A small vector (e.g., 32-dimensional) is optimized alongside the main network weights.
• Output: Modulates the radiance output of the MLP to match the specific look of each training photo.

Non-static elements within a scene that are present in only some input images, such as moving people, vehicles, or shadows. NeRF-W explicitly models these using a secondary MLP head that predicts:

• Transient Density & Color: For occluding or foreground objects.
• Transient Uncertainty: A per-ray variance estimate.
• Impact: The model learns to down-weight the photometric loss from uncertain, transient regions, preventing them from corrupting the underlying static scene reconstruction. This is key to handling 'in the wild' photo collections.

The process by which a model predicts its own confidence or potential error for a given output. In NeRF-W, an uncertainty MLP head outputs a per-sample variance σ² along each ray.

• Role in Training: The photometric loss is scaled by the inverse of this uncertainty (a form of heteroscedastic loss).
• Result: The model learns high uncertainty for regions containing transient objects or noise, effectively ignoring them when reconstructing the permanent scene.
• Differentiation: This differs from aleatoric uncertainty in Bayesian deep learning, as it's a learned, task-specific output.

A foundational method that enables the joint optimization of a NeRF scene representation and imperfect camera poses. This is a critical precursor capability for NeRF-W.

• Problem Solved: 'In the wild' photos often lack accurate, pre-computed camera parameters (from COLMAP).
• Mechanism: BARF gradually introduces high-frequency positional encodings during training, allowing it to coarse-align cameras first, then refine details.
• Connection to NeRF-W: While NeRF-W handles varying appearance, BARF handles pose inaccuracy; they address complementary challenges of unstructured data.

A model architecture (e.g., MVSNeRF, PixelNeRF) trained on a multi-scene dataset to learn strong priors. This enables sparse view synthesis on novel scenes without per-scene optimization.

• Contrast with NeRF-W: NeRF-W is designed for per-scene optimization on a challenging single scene. Generalizable NeRFs are feed-forward networks for rapid inference.
• Trade-off: Generalizable models are faster but often yield lower fidelity than a per-scene optimized model like NeRF-W.
• Hybrid Potential: Techniques from NeRF-W, like appearance embeddings, can be incorporated into generalizable architectures for robustness.

Extensions of NeRF that model scenes changing over time. They take a 4D input (3D space + time) and output time-varying density and radiance.

• Key Difference from NeRF-W: NeRF-W separates static scenes from transient occluders. Dynamic NeRF models coherent deformations or changes of the entire scene geometry (e.g., a melting ice cube).
• Representation: Often uses a canonical space plus a deformation field network.
• Application Scope: While NeRF-W handles messy photo collections, Dynamic NeRF is for controlled, sequential capture of non-rigid motion.