Occupancy Network

Unlike a discrete voxel grid, an occupancy network models space as a continuous implicit function. A neural network, typically a Multilayer Perceptron (MLP), learns a function f(x, y, z) → [0, 1] that maps any 3D coordinate (x, y, z) to a probability of occupancy. This provides infinite resolution and memory efficiency, as the model is defined by its weights, not by a pre-defined grid size.

• Key Benefit: Enables querying occupancy at arbitrary precision without being constrained by a fixed grid resolution.
• Contrast: Traditional methods like Truncated Signed Distance Functions (TSDFs) in voxel grids require dense storage and suffer from discretization artifacts.

The network is conditioned on input sensor data (e.g., images, point clouds) to predict occupancy specific to a scene. This is achieved by encoding the observations into a global latent code or using a cross-attention mechanism. For example, given a set of images, a convolutional encoder extracts features that are pooled into a context vector, which the MLP uses to modulate its predictions.

• Architecture: Often follows an encoder-decoder pattern: Encoder(Observations) → Latent Code → MLP(Coordinates, Latent Code) → Occupancy Probability.
• Flexibility: Can be conditioned on diverse inputs: single images, multi-view images, sparse point clouds, or LiDAR scans.

To extract a usable mesh surface from the learned continuous field, the network leverages the differentiable nature of the Marching Cubes algorithm. The occupancy probabilities are evaluated on a dense 3D grid, and the classic Marching Cubes algorithm finds the iso-surface where occupancy equals 0.5. Because the network's forward pass is differentiable, gradients can flow from the final mesh or rendering loss back through the sampling process to update the network weights.

• Workflow: Network → Occupancy Field → Differentiable Sampling → Marching Cubes → Triangle Mesh.
• Advantage: Enables end-to-end training with losses defined on the final 3D geometry or 2D renderings.

The network outputs a probability of occupancy, not a binary classification. This models inherent uncertainty from ambiguous observations (e.g., occluded regions, sensor noise). A value of 0.5 represents the decision boundary for the surface. This probabilistic formulation is natural for training with a binary cross-entropy loss against ground truth data.

• Interpretation: A value of 0.1 indicates a point is very likely empty (free space), 0.9 indicates very likely occupied, and 0.5 indicates the predicted surface boundary.
• Application: Crucial for robotic path planning, where understanding uncertain regions (neither clearly free nor occupied) is vital for safe navigation.

Training does not require evaluating the network on every point in 3D space—an intractable task. Instead, training uses strategic point sampling. For each training scene, points are sampled:

• Near surfaces: To learn precise geometry.
• In free space: To learn empty regions.
• At random: For general scene coverage.

This focused sampling makes training computationally feasible and forces the network to learn a sharp decision boundary at surfaces. The loss is computed only on these sampled points.

Occupancy networks are closely related to Neural Implicit Representations learning Signed Distance Functions (SDFs). Both are continuous functions parameterized by a neural network.

• Key Difference: An SDF network f_sdf(x, y, z) → d outputs a signed distance to the nearest surface. The surface is at the zero-level set (where d=0). An occupancy network outputs a probability p with the surface at the 0.5-level set.
• Trade-off: SDFs provide direct distance information useful for physics simulation, but occupancy probabilities can be easier to learn from data and are more directly related to probabilistic robotics frameworks.

A voxel grid is the fundamental 3D data structure used by occupancy networks. It discretizes a 3D volume into a regular lattice of cubic elements (voxels), analogous to pixels in a 2D image. Each voxel stores attributes like:

• Occupancy probability (the core output of an occupancy network)
• Semantic class (for 3D semantic segmentation)
• Color or feature vectors While memory-intensive for high resolutions, voxel grids provide a structured, easily processable format for 3D convolutional neural networks.

A Signed Distance Function (SDF) is a continuous, implicit alternative to a discrete occupancy grid. For any 3D point, an SDF outputs its signed distance to the nearest object surface (negative inside, positive outside, zero on the surface). Neural implicit representations often learn an SDF via a multilayer perceptron (MLP). Compared to occupancy networks, SDFs offer:

• Exact surface geometry at arbitrary resolution.
• Inherent smoothness beneficial for robotics tasks like grasping.
• A more memory-efficient continuous representation, though querying can be slower.

3D semantic segmentation is the task of assigning a categorical label (e.g., 'chair', 'wall', 'floor') to every element in a 3D representation. Occupancy networks are frequently extended to perform this task by predicting a class distribution per voxel instead of a single occupancy probability. This is critical for embodied AI agents, enabling them to understand not just where objects are, but what they are for task planning and navigation. The output is a semantically labeled 3D map of the environment.

Neural Radiance Fields (NeRF) is a technique for novel view synthesis that represents a scene as a continuous volumetric function, mapping a 3D location and viewing direction to color and density. While occupancy networks predict binary geometry, NeRFs model view-dependent appearance and continuous density for photorealistic rendering. The two approaches are complementary: occupancy provides clear, actionable geometry for robotics, while NeRF provides high-fidelity visual reconstruction. Hybrid models now exist that combine geometric occupancy with neural radiance fields.

Bird's-Eye View (BEV) perception is a dominant paradigm in autonomous driving where sensor data (e.g., from multiple cameras) is transformed into a unified top-down 2D representation. A BEV feature map is analogous to a 2D slice of a 3D occupancy grid at ground level. Modern occupancy forecasting models for autonomous vehicles predict future states in BEV space, estimating where dynamic objects (cars, pedestrians) will occupy in subsequent timesteps, which is crucial for safe motion planning.

Scene flow is the complete 3D motion vector field of every point in a scene between two time frames. It is the 3D equivalent of 2D optical flow. For dynamic scene understanding, occupancy networks are often coupled with scene flow estimation. This allows a system to not only know the current 3D occupancy of the environment but also to predict how each occupied region is moving. This is essential for motion planning in environments with other moving agents, enabling prediction of future occupancy states.