Deformable NeRF

A canonical space is a fixed, normalized 3D reference frame where the scene's static geometry and appearance are defined. The core innovation of Deformable NeRF is learning a continuous deformation field that maps points from this canonical space to their observed 3D positions at each timestep t. This separation disentangles the learning of a high-quality static model from the complex, time-varying motion, leading to more stable optimization and higher-fidelity reconstructions of the base object or scene.

The deformation field is a neural network (often an MLP) that takes a 3D point x and a time t as input and outputs a displacement vector Δx. The observed coordinate is x_observed = x + Δx(x, t). This field is typically regularized to be smooth in both space and time to prevent unrealistic, discontinuous motions. Some advanced implementations also predict a warping of view directions, accounting for how non-rigid motion affects the local surface orientation and shading.

To model dynamics, the network must be conditioned on time. Common approaches include:

• Direct Input: Time t is concatenated with spatial coordinates as a direct input to the deformation and/or radiance networks.
• Temporal Latent Codes: A compact vector embedding z_t is learned for each timestep or frame. This code is fed into the networks, allowing the model to capture complex, frame-specific appearance changes without overfitting to noise. Latent codes can be interpolated to generate novel, in-between timesteps.

Deformable NeRF is trained end-to-end from multi-view video sequences. The system jointly optimizes:

• The canonical NeRF's density and color fields.
• The parameters of the deformation field network. The primary loss is the standard photometric reconstruction loss, comparing rendered pixels to ground truth images. This forces the model to discover a deformation that, when applied, makes the canonical NeRF match all observed frames across time and viewpoints.

Learning from video alone is an ill-posed problem. To avoid degenerate solutions (e.g., collapsing geometry), strong regularization is essential:

• Cycle Consistency: A point deformed from canonical to time t and back should return to its original position.
• Spatial & Temporal Smoothness: Penalizes large gradients in the deformation field.
• Rigidity Loss: Encourages local regions to deform as rigid bodies where appropriate.
• As-Rigid-As-Possible (ARAP) Priors: A common geometric prior that penalizes non-rigid distortions of local neighborhoods.

The entire pipeline remains differentiable, enabling gradient-based learning from images. For a target viewpoint and time t:

1. A ray is cast into the scene.
2. Sample points along the ray are inversely deformed from the observed space back into the canonical space using the estimated deformation field.
3. The canonical NeRF evaluates density and color at these canonical points.
4. Colors are composited via volume rendering. Gradients flow backward through the rendering, canonical NeRF, and deformation networks to update all parameters simultaneously.

The process of creating a time-varying, dynamic 3D model of a scene from a sequence of images or videos. It captures both the scene's geometry and its evolution over time, forming the overarching goal that deformable NeRF addresses. Unlike static 3D models, 4D reconstructions enable replay, analysis, and novel view synthesis of past events.

A continuous vector field that defines the mapping from points in a canonical, static 3D space to their observed positions at a given timestep. This is the core mathematical object learned by a deformable NeRF.

• Function: Takes a canonical 3D coordinate and a time t, outputs a 3D displacement vector.
• Purpose: Separates non-rigid motion from static appearance, allowing the model to learn a single canonical radiance field that is warped over time.

A foundational strategy where all observations of a deforming object are projected back to a single, fixed reference pose. In deformable NeRF, the neural network learns the scene's color and density in this canonical space. The deformation field then maps these canonical points to their correct position at each observed time. This disentanglement simplifies learning and improves generalization.

A specific method that jointly learns a time-varying neural radiance field and a 3D scene flow field from monocular video. It explicitly models the 3D motion vector (scene flow) of every point in space, in addition to its color and density. This provides a more interpretable motion representation compared to a monolithic deformation field and enables tasks like future frame prediction.

A regularization term used during the training of dynamic NeRFs to enforce physically plausible motion. It penalizes the model for predicting unrealistic or abrupt changes in geometry or appearance between consecutive timesteps.

• Example: A loss that minimizes the difference in the predicted deformation field for a point between time t and t+Δt, encouraging smooth motion.
• Impact: Critical for reducing flickering artifacts and producing stable, coherent 4D reconstructions from sparse or noisy video data.

The end-user application enabled by technologies like deformable NeRF. It is an interactive visual media format that allows a viewer to control both the viewpoint and the playback time within a reconstructed dynamic event. The user can navigate a virtual camera as if they were physically present in the 4D scene, a key capability for sports broadcasting, virtual production, and immersive archives.