Neural Scene Graph

A Neural Scene Graph decomposes a complex environment into a tree-like hierarchy of nodes, where each node represents a distinct, semantically meaningful object or background element. This is a fundamental shift from monolithic NeRF representations.

• Root Node: Typically represents the static background or global scene context.
• Child Nodes: Represent individual, movable objects (e.g., a car, a chair).
• Transform Edges: Connect nodes via spatial transformations (rotation, translation, scale), defining each object's pose relative to its parent. This structure mirrors classic scene graphs in computer graphics but uses neural fields as the underlying object representation.

Each node in the graph is instantiated as an independent neural representation, most commonly a small Neural Radiance Field (NeRF) or a similar implicit model (like a Signed Distance Function).

• Local Coordinates: Each object-NeRF is defined in its own canonical, object-centric coordinate system.
• Specialization: Individual NeRFs can be optimized to capture fine details of their specific object, improving overall fidelity.
• Efficiency: Rendering can be accelerated by using simpler or faster representations for distant or less important objects. This object-wise decomposition is key for compositional generalization and editing.

To render a novel view, the system composites the scene by evaluating each object-NeRF and applying its learned or known spatial transformation.

1. Ray Transformation: For each pixel's ray, the ray is transformed from world coordinates into the local coordinate system of each object node using the inverse of the node's transformation matrix.
2. Local Sampling: The object's NeRF is queried along the transformed ray to obtain local density and color values.
3. Alpha Compositing: The outputs from all objects are alpha-composited in depth order (typically using the classic volume rendering equation) to produce the final pixel color. This process enables correct occlusion and interaction between neural objects.

The explicit graph structure enables powerful editing operations that are intractable for a single, entangled NeRF.

• Object-Level Manipulation: Objects can be translated, rotated, scaled, or removed by simply editing their node's transformation matrix or pruning the node from the graph. The object's neural representation remains intact.
• Instance Swapping: A node's neural field can be replaced with another compatible neural field (e.g., swapping one car model for another).
• Animation: By defining trajectories for transformation matrices over time, dynamic sequences can be created. This is foundational for applications in digital twins and interactive 3D content creation.

The graph structure allows for rendering optimizations borrowed from traditional graphics pipelines.

• Frustum Culling: If an object's bounding volume (often derived from its NeRF's density field) is outside the camera's view frustum, its entire sub-graph can be skipped during rendering.
• Level of Detail (LOD): Different neural representations of the same object with varying complexity (e.g., a high-detail and a low-detail NeRF) can be attached to a node and selected based on distance from the camera.
• Selective Updates: Only parts of the scene graph that have changed (e.g., a moved object) need to be re-optimized, saving computational cost during test-time optimization.

Advanced NSG frameworks disentangle appearance into intrinsic properties, moving towards inverse rendering.

• Neural Reflectance Fields: An object node can be modeled as a neural reflectance field, separating its Bidirectional Reflectance Distribution Function (BRDF) from lighting.
• Shared Lighting Model: A global lighting node (e.g., an environment map or a set of virtual light sources) can be connected to object nodes, allowing for scene relighting where lighting changes are applied consistently across all objects.
• Material Consistency: This structure enforces that the same material, if used on multiple objects, has consistent reflectance properties across the graph.

The foundational technique upon which Neural Scene Graphs are built. A NeRF represents a continuous 3D scene as a volumetric function, parameterized by a multilayer perceptron (MLP). It maps a 3D coordinate (x, y, z) and viewing direction (θ, φ) to a volume density and view-dependent RGB color. This implicit representation is optimized via differentiable volume rendering to synthesize photorealistic novel views from a sparse set of 2D images.

The critical mathematical framework that enables the optimization of 3D scene representations from 2D images. Differentiable rendering allows gradients to flow from a loss computed on rendered pixels back to the underlying scene parameters (like density, color, or object pose). This is the engine that makes training Neural Scene Graphs possible, as it allows the backpropagation of error through the entire rendering pipeline to update the individual neural fields and their spatial transformations.

An alternative implicit representation for geometry, often used in place of a density field in modern neural rendering. An SDF defines a surface by the signed distance from any point in space to the nearest surface, with the sign indicating inside (negative) or outside (positive). In a Neural Scene Graph context, individual objects might be represented by neural implicit surfaces defined by SDFs, which offer precise, watertight geometry that is easier to extract as a mesh than a NeRF's density field.

The broader problem of estimating underlying physical scene properties from images. While a standard NeRF learns an entangled representation of geometry and appearance, inverse rendering aims to disentangle components like:

• Geometry (mesh or SDF)
• Material (via a Bidirectional Reflectance Distribution Function - BRDF)
• Lighting (environment maps or light probes) Neural Scene Graphs advanced this by adding a hierarchical object structure to the inverse rendering problem, enabling per-object material and lighting editing.

A classical data structure from computer graphics and vision that represents a scene as a directed graph. Nodes represent entities (objects, lights, cameras), and edges represent relationships between them (spatial transformations like translation/rotation, semantic links like 'holds', or hierarchical 'parent-child' links). A Neural Scene Graph injects neural representations into this structured framework, where each node contains a neural field (NeRF or SDF) and edges are parameterized transformations optimized alongside the representations.

A class of generative models that learn to represent complex data as a combination of simpler, reusable parts or objects. This principle is central to Neural Scene Graphs. Key aspects include:

• Disentanglement: Separating object identity, pose, and appearance.
• Compositionality: Assembling novel scenes by recombining learned object representations.
• Hierarchy: Representing parts within whole objects (e.g., wheels on a car). This approach provides strong inductive biases for data efficiency, generalization, and intuitive scene editing compared to monolithic scene representations.