Appearance Embedding

An appearance embedding is a low-dimensional latent vector (e.g., 32-128 dimensions) assigned to each training image. It conditions the NeRF's radiance output, allowing the same underlying geometry to be rendered under different illumination, weather, or seasonal conditions. This separates intrinsic scene properties from transient, image-specific appearance.

• Key Mechanism: The embedding vector is concatenated with the spatial coordinate and viewing direction before being fed into the radiance MLP.
• Result: A single unified model can represent a scene at noon, dusk, or under cloudy skies.

In models like NeRF in the Wild (NeRF-W), appearance embeddings work alongside an uncertainty field. This architecture explicitly disentangles the static, permanent scene geometry from transient objects like moving people or cars.

• The appearance embedding models consistent but variable lighting.
• A separate transient embedding and predicted uncertainty per ray allow the model to downweight the photometric loss for pixels containing non-static objects.
• This prevents the model from baking temporary elements into the permanent scene reconstruction.

Appearance embeddings are learnable parameters optimized via gradient descent alongside the NeRF's MLP weights during training.

• Training: Each image's unique embedding is optimized to minimize the photometric loss between the rendered and actual image.
• Inference for Novel Views: To render a novel view under a specific condition (e.g., 'sunset'), the system uses the appearance embedding from the training image that best matches that condition.
• Interpolation: Embeddings can often be interpolated to synthesize smooth transitions between appearances, like a time-lapse of lighting changes.

Appearance embeddings differ from classical intrinsic image decomposition (separating albedo and shading). They are a data-driven, non-physical latent representation.

• Not Explicit Shading: The embedding captures all appearance variation in a compressed form, which may conflate lighting, white balance, and even sensor effects.
• Advantage: This flexibility allows the model to handle complex, real-world appearance changes without requiring strict physical constraints or ground truth decompositions for training.

This technique is essential for building practical NeRF systems from unstructured photo collections, such as internet tourism photos.

• NeRF-W: The canonical example, enabling 3D reconstruction from online photo albums where every picture has different lighting.
• Time-Lapse & Seasonal Modeling: Capturing a landmark across different times of day or years.
• Foundation for Dynamic NeRF: The concept extends to dynamic scenes, where embeddings can be conditioned on a time index to model appearance changes over a video sequence.

While powerful, appearance embeddings have specific constraints that system designers must consider.

• Per-Scene Optimization: Embeddings are typically learned per scene; they do not provide a prior for generalizing to new scenes.
• Ambiguity Risk: There is a risk of the embedding absorbing geometric or view-dependent effects if not properly regularized, leading to blurry geometry.
• Fixed Vocabulary: At inference, you are limited to the set of appearances seen during training or interpolations between them; you cannot generate arbitrarily new lighting conditions without additional generative modeling.

This is the canonical application introduced by NeRF in the Wild (NeRF-W). When training on unstructured internet photos of a landmark (e.g., the Trevi Fountain) taken at different times of day and seasons, a per-image appearance embedding captures the variable illumination conditions (sun position, cloud cover) and atmospheric effects. The core NeRF model learns the static geometry and albedo, while the embedding modulates the output color to match each specific input image's lighting. This cleanly separates persistent scene structure from transient appearance.

In industrial digital twin creation, the same physical asset (e.g., a factory machine) may need to be visualized with different material finishes or under various lighting setups for design review. A per-scenario appearance embedding can be used to condition the NeRF's rendering to simulate:

• Different paint colors or metallic finishes.
• The effect of turning specific area lights on or off.
• The visual impact of different clean/dirty states. This allows for rapid, photorealistic A/B testing of visual designs without re-capturing the 3D scene.

For scenes that change slowly over long periods, such as a construction site or a growing plant, appearance embeddings can be assigned per time-step in a captured sequence. Instead of modeling geometry change (which requires a dynamic NeRF), the embedding captures the evolving visual texture and lighting. This is efficient for scenarios where the 3D structure is largely static, but the surface properties (e.g., wet vs. dry ground, snow cover, leaf color) change systematically over time.

Advanced extensions use appearance embeddings not as monolithic vectors but as parameters for a learned lighting model. For instance, an embedding can be decomposed to represent coefficients of a spherical harmonics basis. This allows for explicit scene relighting: after training, the embedding can be interpolated or edited to synthesize novel lighting conditions (e.g., 'sunset' to 'blue hour') on the pre-learned geometry. This moves beyond simple reconstruction towards controllable scene editing.

In generative radiance fields (e.g., for text-to-3D), appearance embeddings act as a control mechanism. A single generative model can produce a variety of stylized 3D assets by sampling different appearance codes. For example, a model trained on 3D cars could use one embedding subspace for paint color and another for environment map lighting, allowing independent control over these attributes during 3D asset synthesis. This provides a compact latent space for navigating visual variations.

In multi-modal capture setups, images may come from different sensors (e.g., a standard RGB camera and a multispectral imager). A per-sensor or per-spectral-band appearance embedding can learn the sensor-specific response function and color characteristics. This allows a unified NeRF to fuse data from heterogeneous sources, reconstructing a 3D scene that can later be rendered to simulate the output of any of the training sensors, enabling applications in remote sensing and scientific imaging.