Canonical Space Mapping

The core principle of canonical space mapping is the separation of concerns. A scene's intrinsic properties—its canonical shape, material, and texture—are learned in a static, unchanging 3D space. Its extrinsic motion is modeled separately via a learned deformation field. This separation simplifies optimization, as the network does not need to learn appearance from scratch at every timestep, leading to more stable training and higher-fidelity reconstructions of the base object.

A deformation field is a continuous, learned function that maps observed 3D points at a given time back to their corresponding locations in the canonical space. It is typically implemented as a Multi-Layer Perceptron (MLP) that takes as input a 3D coordinate and a time code, and outputs a 3D displacement vector. This field must be bijective (invertible) to ensure consistent mapping. For articulated objects like humans, this is often combined with skinning weight networks that predict blend weights for a skeletal rig.

At the heart of the system is a standard Neural Radiance Field (NeRF) defined in the canonical space. This NeRF is queried only with canonical coordinates to predict volume density and view-dependent color. Because it operates in a static space, it can learn highly detailed geometry and complex view-dependent effects (like specular highlights) without the confounding variable of motion, which is handled entirely by the deformation field. This is the key advantage over monolithic Dynamic NeRF models.

Learning the deformation field from monocular or sparse-view video is an ill-posed problem. To prevent degenerate solutions (e.g., collapsing all geometry to a point), strong regularization losses are essential:

• As-Rigid-As-Possible (ARAP) Loss: Encourages local deformations to be close to a rigid transformation.
• Cycle Consistency Loss: Ensures that deforming a point from canonical to observed space and back returns to the original location.
• Temporal Smoothness Loss: Penalizes large, abrupt changes in the deformation field between consecutive frames. These constraints enforce physically plausible motion.

Canonical space mapping is particularly powerful for:

• Human & Animal Performance Capture: Learning a single, high-quality canonical model of an actor that can be animated via deformation.
• Long-Term Dynamic Scenes: Modeling scenes where objects move but their intrinsic appearance remains stable (e.g., a car driving, a person walking).
• Data Efficiency: Once a canonical model is learned, it can generalize to new motions with less data.
• Editing & Re-animation: The disentangled representation allows artists to edit the canonical shape or apply novel motion sequences to the learned model.

Canonical mapping is a pivotal concept with several specialized implementations:

• Deformable NeRF (D-NeRF): The seminal work that popularized this paradigm for monocular video.
• Neural Scene Flow Fields (NSFF): Jointly learns scene flow (3D motion) alongside the radiance field.
• 4D Gaussian Splatting: Uses explicit, time-varying 3D Gaussians, where a canonical set of Gaussians is deformed over time.
• Articulated Models: For human bodies, methods like A-NeRF or HumanNeRF use a statistical body model (SMPL) to provide a strong kinematic prior for the deformation field, drastically improving robustness.

A dynamic Neural Radiance Field that models non-rigid motion by learning a continuous deformation field. This field maps observed 3D points at any given time back to a single, static canonical space. This separation simplifies learning, as the canonical space holds a consistent representation of shape and appearance, while the deformation field accounts for all temporal change.

A continuous vector field that defines the mapping between a canonical 3D coordinate and its observed, deformed position at a specific time. It is the core mathematical object learned in canonical space mapping.

• Function: Takes a canonical point and a time, outputs a 3D displacement vector.
• Properties: Often constrained to be smooth and invertible to ensure physically plausible motion.
• Implementation: Typically parameterized by a multilayer perceptron (MLP).

The overarching goal of creating a time-varying 3D model (3D + time = 4D) of a scene from image or video data. Canonical space mapping is a dominant strategy for achieving this. The output is a model that can be queried for geometry, appearance, and motion at any spatial point and any moment within the captured sequence.

A critical regularization term used when training deformable reconstruction models like those using canonical space mapping. It penalizes unrealistic, jerky, or discontinuous motion between consecutive frames. This loss enforces the prior that real-world motion is generally smooth and continuous, guiding the optimization of the deformation field to produce more physically accurate results.

A specific, structured prior for deformation, often used for reconstructing humans, animals, or robots. Instead of a general deformation field, motion is modeled as a kinematic chain of rigid parts connected by joints.

• Relation to Canonical Mapping: The canonical space often corresponds to a rest pose or T-pose.
• Skinning: A skinning weight network may be used to predict how much each joint influences a canonical 3D point's deformation, analogous to linear blend skinning in computer graphics.

A seminal method that jointly learns a dynamic radiance field and a 3D scene flow field from monocular video. It represents a closely related paradigm to canonical space mapping. Instead of mapping to a single canonical frame, NSFF estimates the 3D motion vector (scene flow) for every point between consecutive times, enabling both novel view synthesis and motion estimation.