Positional Encoding

The original formulation, introduced in the Transformer architecture 'Attention Is All You Need,' uses a set of sine and cosine functions at geometrically increasing frequencies. For a position pos and dimension i, the encoding is:

• Even indices: sin(pos / 10000^(2i/d_model))
• Odd indices: cos(pos / 10000^(2i/d_model)) This design ensures that each position has a unique encoding and that the model can learn to attend to relative positions through linear transformations of these sinusoidal patterns.

A core motivation in neural graphics is to overcome spectral bias or the 'frequency principle,' where standard MLPs preferentially learn low-frequency components of a target signal. By projecting coordinates into a high-dimensional space with explicit high-frequency components, positional encoding enables the MLP to approximate fine details like sharp edges, textures, and high-frequency color variations in images and 3D scenes that it would otherwise smooth over.

Non-learned (Fixed) Encoding: Uses a deterministic function (e.g., sinusoids, random Fourier features). It is simple, generalizes to any sequence length, and has theoretical properties for representing relative position.

Learned (Positional) Embeddings: Treats each position index as a token and learns a dense embedding vector, as in BERT's initial token embeddings. While flexible, it is limited to a predefined maximum sequence length and lacks inherent structural biases for extrapolation.

In neural scene representations like NeRF, fixed encodings are standard as they must handle continuous, unseen 3D coordinates at inference.

For 3D neural fields (NeRF, SDFs), positional encoding is applied independently to each coordinate (x, y, z). A 3D point (x, y, z) is encoded by concatenating the sinusoidal encodings for each axis. This allows the network to resolve high-frequency spatial structure in all dimensions. Viewing direction (θ, φ) is often encoded separately at a different frequency band to capture view-dependent effects like specular highlights.

Sinusoidal positional encoding is a specific case of random Fourier feature mapping. According to the neural tangent kernel theory, passing inputs through a random Fourier feature layer shifts the network's effective kernel towards a stationary kernel, improving convergence for learning high-frequency functions. This formalizes why the technique is so effective: it pre-conditiones the model's inductive bias to better fit complex signals.

Modern neural graphics primitives often replace pure sinusoidal encoding with more efficient parametric mappings:

• Hash Encoding (InstantNGP): Uses a multi-resolution hash table of learnable feature vectors. Provides extremely fast lookup and high-quality reconstruction.
• Spherical Harmonics: Used for encoding viewing directions to compactly represent low-frequency, view-dependent appearance.
• Integrated Positional Encoding (IPE): Used in Mip-NeRF, it encodes not a point but the expected statistics of a conical frustum, enabling anti-aliased rendering by modeling volumetric sampling. These methods retain the core goal of mapping coordinates to a richer feature space but do so with learned, memory-efficient data structures.

An Implicit Neural Representation (INR) is a continuous, parameterized function—typically a coordinate-based MLP—that maps spatial (or spatio-temporal) coordinates directly to an output signal value, such as color, density, or signed distance. Positional encoding is a critical pre-processing step for INRs, as it helps the network overcome spectral bias and learn high-frequency details in the target signal.

• Core Architecture: The foundation for NeRF, Neural SDFs, and Neural Light Fields.
• Continuity: Represents scenes as continuous functions, enabling infinite resolution.
• Application: Used for 3D reconstruction, novel view synthesis, and signal compression.

Hash encoding is a memory-efficient, multi-resolution feature indexing technique that uses a trainable hash table to store feature vectors for spatial coordinates. It serves as a highly efficient alternative to sinusoidal positional encoding, enabling near-instantaneous lookup and dramatically faster training (e.g., in Instant NGP).

• Multi-Resolution Grid: Stores features at multiple levels of detail for adaptive fidelity.
• Collision Handling: Uses a hash function that tolerates collisions, controlled by gradient updates.
• Performance: Enables training of high-quality neural scenes in seconds or minutes, rather than hours.

SIREN is a specific type of coordinate-based MLP that uses periodic sine functions as activation functions throughout the network. Unlike standard ReLU MLPs that require explicit positional encoding, SIRENs inherently excel at modeling complex natural signals and their derivatives, making them highly effective for representing detailed shapes and textures.

• Inherent High Frequencies: The sine activations allow the network to learn fine details without explicit Fourier feature mapping.
• Derivative Continuity: Excellent for physics-informed tasks where smooth gradients are required.
• Alternative Paradigm: Represents a different architectural solution to the same problem positional encoding addresses for ReLU networks.

Differentiable rendering is a framework that formulates the image synthesis process (like the volume rendering integral) as a differentiable function of scene parameters. This allows gradient-based optimization of geometry, appearance, and lighting from 2D image observations. Positional encoding is often applied before the differentiable renderer, within the neural scene representation being optimized.

• Gradient Flow: Enables backpropagation of image loss through the rendering equation to scene parameters.
• Foundation for Learning: The key technique that allows models like NeRF to be trained from photographs.
• Unified Pipeline: Connects neural representations (the scene model) with supervision signals (2D images).

Ray marching is the core iterative sampling algorithm used to render implicit and volumetric neural representations like NeRF and Neural SDFs. It numerically evaluates integrals by sampling points along camera rays. Positional encoding is applied to each sampled 3D coordinate before it is fed into the neural network to query color and density.

• Numerical Integration: Approximates the continuous volume rendering integral with discrete samples.
• Hierarchical Sampling: Often uses a two-pass approach (coarse and fine sampling) for efficiency.
• Rendering Primitive: The essential computational procedure that translates a neural field into a 2D image.

A coordinate-based MLP is a fully-connected neural network that takes spatial coordinates (x, y, z) and possibly a viewing direction as input, and outputs a property of the scene at that location. This is the fundamental neural architecture for most implicit neural representations. Positional encoding is applied directly to the input coordinates of this MLP to overcome its tendency to learn low-frequency functions.

• Universal Approximator: The MLP serves as a compact, continuous function approximator for the scene.
• Input-Output Mapping: Directly models the mapping from a query location to a visual property.
• Bottleneck Architecture: Without encoding, this network architecture suffers from spectral bias, favoring smooth interpolations.