Coordinate-Based Network

Unlike explicit representations like voxel grids or meshes, a coordinate-based network defines a scene as a continuous function. The network f(x, y, z) -> property can be queried at infinite spatial resolution. This eliminates the memory constraints of discrete grids and allows for the modeling of smooth, detailed surfaces and volumetric fields without predefined topology.

The core network is typically a simple, fully-connected Multilayer Perceptron (MLP). This architecture is chosen for its universal approximation capabilities and ability to process arbitrary, unordered coordinate inputs. Key design considerations include:

• Depth and Width: Balancing representational capacity with overfitting risk.
• Activation Functions: Critical for learning high-frequency details (e.g., ReLU, sine functions in SIRENs).
• Parameter Efficiency: A small MLP can represent complex scenes, making the representation highly compact compared to explicit 3D data.

Raw spatial coordinates are low-dimensional and poorly suited for MLPs to learn high-frequency scene details, a problem known as spectral bias. To overcome this, coordinates are transformed via an input encoding before being fed to the MLP. Common techniques include:

• Positional Encoding: Uses sinusoidal functions (e.g., sin(2^k π x), cos(2^k π x)) to project coordinates into a higher-dimensional space.
• Fourier Features: Similar projection using random frequency bands.
• Hash Encoding: A multi-resolution lookup table for extremely fast, memory-efficient encoding, as used in Instant NGP.

The network is a differentiable function of its input coordinates and weights. This enables two fundamental operations:

• Gradient-Based Optimization: Scene parameters (network weights) can be learned from 2D images via differentiable rendering, where gradients flow from pixel errors back through the rendering equation and into the network.
• Analytical Queries: Beyond simple value prediction, one can compute spatial derivatives (e.g., the surface normal as the gradient of an SDF) via automatic differentiation, which is essential for lighting, physics, and mesh extraction.

The network's output defines the type of implicit representation and its application:

• NeRF: Outputs a density (σ) and view-dependent color (RGB).
• Neural SDF: Outputs a signed distance value, where the zero-level set defines the surface.
• Occupancy Network: Outputs a probability (0 to 1) of occupancy.
• Single-Property Fields: Can output other scalar or vector fields, such as material attributes, semantic labels, or velocity (for dynamic scenes).

Coordinate-based networks exemplify a classic systems trade-off. They store scene information implicitly within the network's weights, leading to a very compact memory footprint (often just a few megabytes). However, querying the scene (inference) is computationally intensive, as it requires a full forward pass of the MLP for each 3D point. This contrasts with explicit representations (like point clouds), which are memory-heavy but permit constant-time lookups. Advances like hash encoding and specialized MLP kernels aim to optimize this compute cost.

An Implicit Neural Representation (INR) is the overarching paradigm where a continuous signal (image, audio, 3D shape) is represented by a neural network that maps coordinates to signal values. A coordinate-based network is a specific implementation of an INR.

• Core Principle: The network itself is the compact, differentiable storage format.
• Contrast with Explicit: Unlike voxel grids or meshes, INRs do not store values for every discrete location; they are queried on-demand.
• Applications: Beyond 3D shapes, INRs are used for images, videos, and scientific data.

A Signed Distance Function (SDF) is a specific type of scalar field output by a coordinate-based network. For any 3D point (x, y, z), it outputs the shortest distance to a surface, with sign indicating inside (negative) or outside (positive).

• Mathematical Definition: f(x, y, z) = d, where |d| is the distance, sign indicates interior/exterior.
• Surface Definition: The zero-level set (where SDF=0) defines the reconstructed surface.
• Advantages: Provides a smooth, differentiable field ideal for physics simulation and robust mesh extraction.

An Occupancy Network is a coordinate-based network that outputs a binary or probabilistic classification for any 3D coordinate: occupied (inside the object) or empty (outside).

• Output Range: Typically a value between 0 and 1, representing occupancy probability.
• Contrast with SDF: Less precise geometric information than a distance value, but often easier for a network to learn for complex topology.
• Training: Uses a binary cross-entropy loss against ground-truth occupancy labels.

Positional Encoding is a critical preprocessing step where low-dimensional input coordinates are transformed into a higher-dimensional space using a set of sinusoidal functions before being fed to the network.

• Purpose: Enables standard MLPs (which suffer from spectral bias) to learn high-frequency details in images and geometry.
• Function: γ(p) = (sin(2^0 π p), cos(2^0 π p), ..., sin(2^{L-1} π p), cos(2^{L-1} π p))
• Result: Without this, neural networks tend to produce overly smooth, blurry reconstructions.

Differentiable Rendering is the set of techniques that allow gradients to flow from a 2D image loss back to the 3D scene parameters modeled by a coordinate-based network.

• Mechanism: Makes the rendering pipeline (ray marching, shading, compositing) a differentiable function.
• Key for Optimization: Enables learning a 3D scene from only 2D images without 3D supervision.
• Two Primary Types: Differentiable Ray Marching (for volumetric fields like NeRF) and Differentiable Rasterization (for meshes).

The Zero-Level Set is the set of points {x, y, z : f(x, y, z) = 0} for an SDF, defining the extracted surface. Mesh Extraction is the algorithm to convert this implicit surface into an explicit polygon mesh.

• Primary Algorithm: Marching Cubes processes a discrete 3D grid of SDF values to generate a mesh approximating the zero-level set.
• Goal: Produce a watertight mesh (a closed, manifold surface without holes).
• Challenge: Balancing mesh fidelity with triangle count and ensuring topological correctness.