Occupancy Network

An occupancy network defines a 3D shape as a continuous scalar field rather than a discrete grid. The neural network, typically a Multilayer Perceptron (MLP), learns a function (f(\mathbf{x}) \rightarrow [0,1]) that outputs the occupancy probability for any continuous 3D coordinate (\mathbf{x} = (x, y, z)). This bypasses the memory and resolution constraints of voxel grids, enabling theoretically infinite resolution and smooth surfaces.

• Key Benefit: Memory efficiency is decoupled from output resolution.
• Core Mechanism: The network generalizes from training samples to infer occupancy for unseen 3D points.

The core function of an occupancy network is differentiable with respect to its input coordinates and network parameters. This allows the model to be trained end-to-end using gradient descent from 2D image supervision or 3D ground truth.

• Training Signal: Gradients flow from a reconstruction loss (e.g., Binary Cross-Entropy) back through the network to update weights.
• Surface Extraction: The actual surface is defined as the decision boundary of the classifier, specifically the 0.5-level set where (f(\mathbf{x}) = 0.5). This boundary can be extracted using algorithms like Marching Cubes.

Occupancy networks are inherently conditional generative models. The network architecture is extended to accept a shape latent code (\mathbf{z}) as an additional input: (f(\mathbf{x}, \mathbf{z}) \rightarrow [0,1]). This enables powerful applications:

• Shape Completion: Given a partial scan (encoded into (\mathbf{z})), the network predicts occupancy for the full volume.
• Learning from Images: A 2D encoder network can process an input image to produce (\mathbf{z}), allowing 3D reconstruction from a single view.
• Variational Formulations: Models like Occupancy Networks (ONet) use a Variational Autoencoder (VAE) framework to learn a structured latent space for shape generation and interpolation.

To train from 2D images without 3D supervision, occupancy networks are integrated into a differentiable rendering pipeline. A differentiable ray marcher casts rays from a camera. At sampled 3D points along each ray, the occupancy network is queried. The sequence of occupancy values is used to compute a depth map or silhouette that can be compared to the ground truth image.

• Loss Function: Uses reprojection loss (e.g., L1 on depth) or silhouette loss (Binary Cross-Entropy).
• Challenge: Occupancy is a binary concept, but the rendering process requires a soft, differentiable approximation for gradient flow.

Occupancy networks are a primary type of implicit neural representation, closely related to Neural Signed Distance Functions (SDFs). The key distinction is the output space and the resulting surface definition.

• Occupancy Network: Outputs a probability (p \in [0,1]). Surface is the 0.5-level set. Training uses a classification loss.
• Neural SDF: Outputs a signed distance (d \in \mathbb{R}). Surface is the zero-level set. Training requires Eikonal regularization to ensure valid distance fields.
• Trade-off: Occupancy is often easier to learn from data but provides less inherent geometric information (like surface normals) than an SDF.

The flexibility of occupancy networks makes them foundational for several advanced 3D tasks:

• Single-View 3D Reconstruction: Predicting a full 3D model from a single RGB image (e.g., DISN, ONet).
• Learning from Unstructured Point Clouds: Fitting an occupancy field to a sparse, unordered point cloud to create a watertight surface.
• Generative 3D Modeling: As the decoder in 3D Generative Adversarial Networks (GANs) or VAEs to synthesize novel shapes.
• 4D Reconstruction: Modeling deforming objects over time by conditioning the network on a time parameter (t): (f(\mathbf{x}, t, \mathbf{z})).

A Signed Distance Function (SDF) is a mathematical representation of a 3D surface where the value at any point in space is the shortest distance to the surface. The sign indicates whether the point is inside (negative) or outside (positive) the object. Unlike an occupancy network which outputs a binary probability, an SDF provides a continuous, smooth distance field.

• Key Advantage: Enables high-quality surface extraction via the zero-level set.
• Use Case: The foundation for models like DeepSDF and Neural SDF.

An Implicit Neural Representation (INR) is a foundational method of representing a continuous signal—such as a 3D shape, image, or radiance field—using a neural network. Typically, a coordinate-based network (like an MLP) maps spatial coordinates directly to the signal's value at that location.

• Core Principle: The network itself acts as a compressed, queryable database for the signal.
• Relation to Occupancy Networks: An occupancy network is a specific type of INR where the output is an occupancy probability.

The Zero-Level Set is the set of points where an implicit function's value is zero, defining the reconstructed surface. For an SDF, this is the surface; for an occupancy network, it's typically the 0.5 probability threshold.

Marching Cubes is the seminal algorithm for converting this implicit boundary into an explicit polygonal mesh. It processes a 3D grid, examining small cubes (voxels) to determine how the surface passes through them, generating triangles.

• Critical Step: The final stage for converting a trained occupancy network or SDF into a usable watertight mesh.

Differentiable Rendering is a framework that makes the process of generating a 2D image from a 3D scene representation mathematically differentiable. This allows gradients to flow from 2D pixel errors back to 3D scene parameters.

• For Volumetric Fields: Differentiable volumetric rendering (as used in NeRF) and differentiable ray marching are key for optimizing neural fields from images.
• For Surfaces: Differentiable rasterization enables optimizing mesh properties from image losses.
• Connection: While occupancy networks are often trained on 3D data, differentiable rendering allows them to be learned from 2D image collections alone.

A Neural Radiance Field (NeRF) is an implicit neural representation that models a 3D scene volumetrically. Instead of occupancy, it maps a 3D coordinate and viewing direction to a color and density (opacity). The scene is rendered using the volume rendering equation and alpha compositing.

• Key Difference: NeRF models appearance and density for novel view synthesis; occupancy networks model geometry for shape reconstruction.
• Hybrid Approaches: Modern methods often combine geometric representations (like occupancy/SDF) with radiance fields for high-fidelity reconstruction and rendering.

Quantitative evaluation of 3D shape reconstruction, including outputs from occupancy networks, relies on specific metrics:

• Chamfer Distance: Measures the average nearest-neighbor distance between two point clouds. It's sensitive to local details but can be fooled by non-uniform distributions.
• Earth Mover's Distance (EMD): A more rigorous metric that computes the minimum cost to transform one point distribution into another, considering global structure.
• Volumetric IoU: For occupancy grids, this measures the overlap between predicted and ground-truth occupied voxels. A high IoU indicates accurate binary occupancy classification.