Coordinate-Based MLP

A coordinate-based MLP is a Multilayer Perceptron (MLP) composed of stacked, fully-connected layers. It takes a low-dimensional coordinate vector (e.g., [x, y, z]) as input and outputs a property of the scene at that location, such as color, density, or signed distance. This simple, continuous function approximator can represent complex signals without any convolutional or recurrent layers, making it a universal function approximator for spatial data.

• Core Operation: f_θ(x) → y, where x is a coordinate, θ are network weights, and y is the output signal.
• Typical Structure: 4-8 hidden layers with 64-256 neurons each, using non-linear activation functions like ReLU or sine.

A raw MLP with ReLU activations suffers from spectral bias, tending to learn low-frequency functions and blurring fine details. To overcome this, input coordinates are transformed into a higher-dimensional space before being fed to the network. Two primary techniques are used:

• Positional Encoding (PE): Maps coordinates using a set of sinusoidal functions: γ(p) = [sin(2^0 π p), cos(2^0 π p), ..., sin(2^(L-1) π p), cos(2^(L-1) π p)]. This allows the MLP to easily represent high-frequency scene content.
• Feature Grid/Hash Encoding: Stores learnable feature vectors in a multi-resolution grid or hash table. The coordinate is used to interpolate these features, which are then passed to a small MLP. This is the core of Instant NGP, drastically reducing the required network size and training time.

The MLP's output defines the type of implicit neural representation being modeled. The network architecture is largely consistent; it is the interpretation of the output that changes:

• NeRF (Neural Radiance Field): Outputs a volume density (σ) and RGB color c. The density defines geometry, and the view-dependent color enables photorealistic rendering.
• Neural SDF (Signed Distance Function): Outputs a signed distance d to the nearest surface. A value of zero defines the surface, negative is inside, positive is outside.
• Occupancy Network: Outputs a probability (0 to 1) of a coordinate being inside an object.
• Neural Reflectance Field: Outputs decomposed material properties like albedo, roughness, and normal, enabling relighting.

The coordinate-based MLP is not trained on paired (coordinate, value) data. Instead, it is optimized end-to-end through a differentiable renderer. A rendering loss between synthesized and ground-truth images drives the learning.

• Process: For a training image, camera rays are cast. Points along each ray are sampled, and the MLP is queried at those 3D coordinates.
• Volume Rendering Integral: The sampled densities and colors are composited into a final pixel color using the differentiable volume rendering equation.
• Loss: The mean squared error (MSE) between the rendered pixel colors and the actual image pixels is backpropagated through the renderer and into the MLP's weights, teaching it the correct 3D scene structure.

This architecture offers distinct benefits and limitations compared to explicit 3D representations like meshes or point clouds.

Advantages:

• Continuous Resolution: Infinitely queryable at any spatial resolution, avoiding discretization artifacts.
• Memory Efficiency: Can represent complex scenes with a relatively small number of network parameters compared to a high-resolution voxel grid.
• Differentiability: Seamlessly integrates with gradient-based optimization for reconstruction from images.

Trade-offs:

• Computational Cost: Rendering requires thousands of network queries per ray, which is slow for vanilla MLPs.
• Editability: The scene is encoded in a black-box weight distribution, making localized editing non-trivial.
• Initialization Sensitivity: Training can be unstable and prone to local minima without careful initialization and regularization.

The core coordinate-MLP concept has been extended and accelerated for practical use:

• Hybrid Representations (e.g., Instant NGP): Replace the deep MLP with a tiny network (1-2 layers) fed by features from a fast, multi-resolution hash table. This decouples detail (stored in the grid) from smooth interpolation (handled by the MLP).
• Explicit Primitive Integration (e.g., 3D Gaussians): While not an MLP, 3D Gaussian Splatting represents a shift towards explicit, differentiable primitives for real-time performance, addressing the primary speed bottleneck of pure coordinate-MLP rendering.
• Modular & Structured Networks: Architectures like Neural Scene Graphs decompose the scene into object-level MLPs for independent manipulation, moving beyond a single monolithic function.

Coordinate-based MLPs serve as the 3D representation in generative models. A generative model (like a GAN or diffusion model) learns to produce the weights of a coordinate-MLP that represents a coherent 3D object.

• pi-GAN: Uses a periodic positional encoding and a StyleGAN2-inspired generator to produce MLP parameters that render consistent 3D objects from any viewpoint.
• Score Distillation Sampling (SDS): In text-to-3D systems like DreamFusion, a coordinate-based MLP (a NeRF) is optimized. Its 2D renders are scored by a pretrained 2D diffusion model (e.g., Stable Diffusion), providing gradients to update the 3D representation without any 3D training data.

Beyond computer graphics, coordinate-MLPs are used as continuous models for complex scientific data defined over space.

• Climate Modeling: Representing continuous temperature or pressure fields from sparse sensor data.
• Medical Imaging: Modeling MRI or CT scan data as a continuous intensity function I(x,y,z), allowing for super-resolution and smooth interpolation between slices.
• Physics-Informed Neural Networks (PINNs): The coordinate-MLP is trained not just on data points but also to satisfy the underlying partial differential equations (PDEs) governing the system, by incorporating the PDE residuals into the loss function.

An Implicit Neural Representation (INR) is the overarching paradigm where a continuous, parameterized function—most commonly a coordinate-based MLP—maps spatial (or spatio-temporal) coordinates directly to an output signal value. This output can represent color, volume density, signed distance, or acoustic pressure. INRs provide a resolution-independent, memory-efficient alternative to discrete grids (voxels, point clouds) for representing complex signals like 3D shapes, images, and radiance fields.

• Core Principle: The scene is a continuous function, not a discrete data structure.
• Key Benefit: Infinite resolution and differentiable optimization.
• Common Outputs: RGB color, signed distance, occupancy probability, volume density.

Positional Encoding (or Fourier Feature mapping) is a critical preprocessing technique for coordinate-based MLPs. It transforms low-dimensional input coordinates (e.g., [x, y, z]) into a higher-dimensional vector using a set of sinusoidal functions. This mapping helps standard MLPs, which have a spectral bias towards learning low-frequency functions, to effectively represent the high-frequency details present in images and 3D geometry.

• Function: γ(v) = [sin(2^0 π v), cos(2^0 π v), ..., sin(2^{L-1} π v), cos(2^{L-1} π v)]
• Purpose: Alleviates the spectral bias of ReLU MLPs.
• Result: Enables the network to learn fine details like texture and sharp edges, which is essential for photorealistic NeRF synthesis.

Hash Encoding is a memory-efficient, multi-resolution feature indexing technique that accelerates coordinate-based MLPs. Instead of using a large, dense feature grid, it employs a trainable multi-resolution hash table. For a given 3D coordinate, the encoding performs a fast hash lookup to retrieve and interpolate stored feature vectors from multiple resolution levels, which are then fed into a tiny MLP. This is the core innovation behind Instant Neural Graphics Primitives (Instant NGP).

• Mechanism: Uses a spatial hash function to index a compact table of learnable features.
• Advantage: Enables real-time training and rendering with a fraction of the parameters.
• Trade-off: Handles hash collisions via gradient-based training, which averages features in overlapping cells.

SIREN is a specialized type of coordinate-based MLP that uses periodic sine functions as activation functions for all layers (including the first). This architectural choice allows SIRENs to model complex natural signals and their derivatives with remarkable accuracy and detail, without requiring explicit positional encoding. The sinusoidal activations create an inherent bias for representing signals with strong higher-frequency components.

• Architecture: Φ(x) = W_n ( sin( W_{n-1} ( ... sin(W_1 x + b_1) ... ) + b_{n-1} ) ) + b_n
• Strength: Excellent for representing signals with fine details, such as images, audio waves, and 3D shapes defined by Signed Distance Functions (SDFs).
• Property: The derivative of a SIREN is itself a SIREN, making it ideal for solving differential equations.

Differentiable Rendering is the framework that makes training coordinate-based MLPs for 3D scenes possible. It formulates the classical graphics rendering pipeline—the process of generating a 2D image from 3D scene parameters—as a differentiable function. This allows gradients to flow from a loss computed on a rendered image back through the rendering equation to update the underlying scene parameters stored in the MLP's weights.

• Core Link: Connects the coordinate-MLP's outputs (color/density) to pixel-level image loss.
• Key Techniques: Includes differentiable ray marching and the Volume Rendering Integral.
• Application: Enables optimization of geometry, appearance, and lighting from only 2D image observations.

Ray Marching is the iterative sampling algorithm used to render images from a coordinate-based MLP representing a volumetric scene (like a NeRF). For each pixel, a camera ray is cast into the scene. The algorithm then samples points along this ray, queries the MLP at each point for color and density, and numerically integrates these values using the volume rendering integral to compute the final pixel color.

• Process: 1. Cast ray. 2. Sample points. 3. Query MLP. 4. Composite results.
• Role: The physical bridge between the neural scene representation (MLP) and the 2D image plane.
• Optimization: Hierarchical or importance sampling is used to focus computation on relevant parts of the ray.