Occupancy Networks

An Occupancy Network is an implicit neural representation that defines geometry as a continuous decision boundary in 3D space. Unlike explicit representations (e.g., a triangle mesh with a fixed number of vertices), it is a function, $f_\theta(\mathbf{x})$, parameterized by a neural network. This function maps any 3D coordinate $\mathbf{x}$ to a scalar occupancy probability between 0 and 1. The surface is defined as the isosurface where $f_\theta(\mathbf{x}) = 0.5$. This provides infinite resolution and compact storage, as the network's weights encode the entire shape.

The core task of the network is a binary classification problem at every point in space. For a given 3D coordinate:

• Output ~1.0: The point is inside the object.
• Output ~0.0: The point is outside the object.

The network is typically trained with a binary cross-entropy loss using ground-truth occupancy values. This formulation is simpler than regressing a signed distance, as it only requires knowing if a point is inside or outside, not its exact distance to the surface. Inference involves querying the network at many points to find the 0.5 isosurface, often using algorithms like Marching Cubes.

A key advantage is differentiability. The occupancy probability $f_\theta(\mathbf{x})$ is a differentiable function of both the input coordinate and the network parameters $\theta$. This enables:

• Learning from 2D images via differentiable rendering techniques.
• Direct optimization of the shape based on image-based losses (e.g., silhouette matching).
• Generative modeling where shapes can be smoothly interpolated in the latent space of an autoencoder. The surface itself (the isosurface) is not directly differentiable, but the underlying occupancy field is, allowing gradients to flow through the 3D volume during training.

Occupancy Networks provide significant memory efficiency compared to dense voxel grids. A high-resolution 512³ voxel grid requires storing 134 million binary values. An Occupancy Network can represent the same detailed surface using only the weights of a moderately sized MLP (e.g., a few megabytes). This efficiency comes from the network's ability to act as a compressed, learned prior for 3D shapes. However, this trades off memory for compute: evaluating the network at millions of query points for surface extraction is computationally intensive, a challenge addressed by subsequent hybrid methods like Instant NGP.

Occupancy Networks are rarely used for a single shape. They are typically conditioned on a latent vector $\mathbf{z}$ that encodes a specific shape, enabling a variety of tasks:

• 3D Reconstruction: $\mathbf{z}$ is encoded from input images (e.g., 2D silhouettes, depth maps).
• Generative Modeling: $\mathbf{z}$ is sampled from a prior distribution in models like Occupancy Networks autoencoders or GANs.
• Shape Completion: $\mathbf{z}$ is inferred from a partial observation (e.g., a single depth scan). The network architecture becomes $f_\theta(\mathbf{x}, \mathbf{z})$, allowing a single model to represent an entire shape space.

Occupancy Networks are closely related to Neural Signed Distance Functions (SDFs). Both are implicit representations using coordinate-based MLPs.

• SDF: Outputs a signed, continuous distance to the surface. The surface is the zero-level set. Provides exact distance, useful for physics.
• Occupancy: Outputs a discrete probability. The surface is the 0.5-level set. Simpler loss formulation. Key Difference: An SDF contains more information (distance and direction), making it more expressive but potentially harder to learn from sparse supervision. Some advanced networks, like DeepSDF, successfully regress SDFs, while Occupancy Networks offer a robust alternative for pure shape classification tasks.

Occupancy Networks excel at converting sparse or noisy 3D point clouds into watertight, high-fidelity surface meshes. By querying the network at millions of spatial coordinates, it produces a continuous occupancy field that can be extracted into a mesh using the Marching Cubes algorithm. This is critical for LiDAR processing in autonomous vehicles and 3D scanning pipelines, where raw sensor data must be converted into usable CAD or simulation-ready models.

When conditioned on a single 2D image, an Occupancy Network can infer the complete 3D geometry of an object. The network learns priors about object categories (e.g., chairs, cars) to hallucinate plausible occluded geometry. This capability is foundational for:

• Augmented Reality (AR): Instantly generating 3D models from phone camera feeds.
• Content Creation: Rapid 3D asset generation for games and films from reference images.
• Robotics: Enabling robots to reason about object shape and grasp points from a monocular camera.

Occupancy Networks can complete partial 3D scans by predicting occupancy for unobserved regions. This shape completion is vital in digital archaeology (reconstructing broken artifacts) and medical imaging (reconstructing organs from partial MRI slices). The network acts as a powerful geometric prior, filling holes with semantically plausible geometry rather than simple interpolation, making it superior to traditional methods for handling significant missing data.

By learning a latent space of 3D shapes, Occupancy Networks can function as generative models (e.g., Occupancy GANs). This allows for:

• Sampling diverse, novel 3D shapes from a learned category.
• Interpolating smoothly between two shapes in latent space.
• Conditional generation of 3D geometry from class labels or text prompts. This enables automated design exploration and provides a rich dataset of synthetic 3D models for training other vision systems.

The continuous decision boundary defined by an Occupancy Network (where occupancy = 0.5) provides an exact, differentiable surface. This is directly usable for physics engines and robotic motion planning to perform accurate collision queries. Unlike discrete voxel grids, the implicit representation allows for efficient, precise distance queries and contact normal calculations, which are essential for simulating rigid-body dynamics and planning safe robot trajectories in cluttered environments.

Occupancy Networks are often combined with differentiable rendering pipelines. The occupancy field can be used to define a Signed Distance Function (SDF) or to inform a neural radiance field, enabling end-to-end optimization from 2D images. This hybrid approach allows for reconstructing not only geometry but also photorealistic appearance, supporting applications in digital twin creation and high-quality novel view synthesis where accurate geometry is a prerequisite.

A Neural Signed Distance Function (Neural SDF) is an implicit neural representation that maps a 3D coordinate to its signed distance from the nearest object surface. A value of zero defines the surface, positive values indicate points outside the object, and negative values indicate points inside.

• Key Distinction from Occupancy: While an occupancy network outputs a binary classification (inside/outside), an SDF provides a continuous, metric distance to the surface. This allows for higher geometric fidelity and easier extraction of triangle meshes via algorithms like Marching Cubes.
• Applications: High-quality surface reconstruction, 3D printing, and physical simulation where precise distances are critical.

An Implicit Neural Representation (INR) is the foundational paradigm for Occupancy Networks. It is a continuous, parameterized function—typically a coordinate-based Multilayer Perceptron (MLP)—that maps spatial (or spatio-temporal) coordinates directly to an output signal value.

• Core Concept: The network acts as a "neural memory" that stores a continuous signal (e.g., occupancy probability, signed distance, color, density) without discrete voxels or meshes.
• Advantages: Provides infinite resolution, compact storage, and differentiable properties. Occupancy Networks, Neural Radiance Fields (NeRF), and Neural SDFs are all specific types of INRs.

Differentiable rendering is a framework that formulates the image synthesis (rendering) process as a differentiable function of scene parameters. This enables gradient-based optimization of geometry, appearance, and lighting directly from 2D image observations.

• Connection to Occupancy Networks: To train an occupancy network from images, a differentiable renderer is required. A common approach is to use Differentiable Ray Marching, where occupancy probabilities along a ray are integrated to compute a silhouette loss, allowing gradients to flow back to update the network's weights.
• Key Enabler: This technique is what allows implicit representations like Occupancy Networks to be learned from standard 2D datasets without explicit 3D supervision.

Neural Radiance Fields (NeRF) is a coordinate-based neural representation that encodes a continuous volumetric scene by mapping a 3D location and viewing direction to an emitted color and volume density.

• Comparison with Occupancy Networks: Both are INRs, but they model different scene properties. NeRF focuses on appearance and density for photorealistic view synthesis. Occupancy Networks focus purely on binary geometry (occupied vs. empty space).
• Hybrid Approaches: Modern systems often combine them, using an occupancy network to define geometry and a separate network or field to model view-dependent appearance, improving efficiency and quality.

3D Gaussian Splatting is an explicit, point-based scene representation where the scene is modeled as a collection of anisotropic 3D Gaussians with associated opacity and spherical harmonics for color, rendered via a differentiable rasterization pipeline.

• Paradigm Contrast: This is an explicit representation, in contrast to the implicit representation of Occupancy Networks. Geometry is directly defined by millions of discrete, optimizable primitives (Gaussians).
• Trade-offs: 3D Gaussian Splatting achieves extremely fast, real-time rendering quality but typically requires more memory than a compact implicit network. Occupancy Networks offer a more memory-efficient, continuous boundary but can be slower to render at high fidelity.

Positional encoding is a critical technique for training coordinate-based MLPs in implicit neural representations. It projects low-dimensional input coordinates (e.g., [x, y, z]) into a higher-dimensional space using a set of sinusoidal functions.

• Purpose: Standard MLPs are biased towards learning low-frequency functions. Directly feeding (x,y,z) coordinates results in blurry, low-detail reconstructions. Positional encoding allows the network to more easily model the high-frequency details of object surfaces and textures.
• Formula: A common encoding uses γ(v) = [sin(2^0 π v), cos(2^0 π v), ..., sin(2^{L-1} π v), cos(2^{L-1} π v)] where v is a coordinate and L is the number of frequency bands. This is applied to each coordinate dimension before being fed into the occupancy network.