Multi-View Video Processing

The foundational step of determining the intrinsic parameters (focal length, lens distortion) and extrinsic parameters (position and orientation) for each camera in the rig. Hardware or software genlock ensures microsecond-level synchronization of video frames, which is critical for capturing instantaneous 3D geometry of fast-moving subjects. Without precise calibration, the triangulation of 3D points becomes unreliable, leading to reconstruction artifacts.

The mathematical framework that relates corresponding points across two or more synchronized images. The fundamental matrix and essential matrix encode the epipolar constraint, which states that a point in one image must lie along a corresponding line (the epipolar line) in another image. Dense correspondence matching (e.g., via optical flow or feature matching) followed by triangulation is used to calculate the precise 3D world coordinates of each point, forming the point cloud or mesh for each frame.

The principle that the reconstructed 3D geometry and appearance of a scene should change smoothly over time. Scene flow estimation extends optical flow into 3D, calculating a motion vector for every reconstructed point between frames. Enforcing temporal coherence is a key challenge; methods use regularization losses or recurrent networks (like RNNs or LSTMs) to prevent flickering and ensure physically plausible motion in the final 4D output.

Modern approaches use neural networks to implicitly represent a 4D scene. Dynamic NeRF models treat time as an additional input to a Multi-Layer Perceptron (MLP). Core architectures include:

• Deformable Fields: An MLP learns a continuous deformation field that warps points from a canonical 3D space to their observed position at each time t.
• Time-conditioned Models: The MLP's weights are modulated by a temporal latent code for each frame.
• Explicit 4D Gaussians: Methods like 4D Gaussian Splatting model each point's position, rotation, scale, and opacity as continuous functions of time for real-time rendering.

A class of algorithms that estimate a depth map for each camera view by finding pixel correspondences across images. Patch-based matching and cost volume filtering are traditional techniques. Modern learning-based MVS (e.g., MVSNet) constructs a 3D cost volume from feature maps and regularizes it with 3D CNNs to predict depth. The output is a set of aligned depth maps that are fused (e.g., via Poisson surface reconstruction or TSDF fusion) into a unified 3D mesh per frame.

The primary output of the pipeline. Free-viewpoint video allows a virtual camera to be placed at any point in space and time within the captured volume. Human performance capture is a specialized application requiring high-fidelity reconstruction of non-rigid deformation, often using articulated skeleton models or template meshes as priors. These technologies are foundational for broadcast sports (e.g., the NFL's "Next Gen Stats" 360° replays), visual effects, and immersive telepresence in VR/AR.

4D reconstruction is the process of creating a time-varying, dynamic 3D model of a scene from a sequence of images or videos. It captures both the scene's geometry and its evolution over time, adding the temporal dimension to traditional 3D modeling.

• Core Objective: To produce a coherent spatio-temporal model from multi-view observations.
• Key Challenge: Maintaining temporal consistency while handling occlusions and non-rigid deformations.
• Primary Data Source: Synchronized video streams from calibrated camera arrays.
• Output: A continuous 4D representation (3D space + time) usable for novel view synthesis at any timestamp.

Dynamic NeRF (Neural Radiance Field) extends the standard NeRF framework to model scenes with non-rigid motion and time-varying appearance. It incorporates time as an additional input coordinate to the neural network.

• Architecture: A multilayer perceptron (MLP) that maps a 5D input (3D spatial coordinates, 2D viewing direction, and time) to color and density.
• Deformation Modeling: Often uses a two-stage process: a canonical NeRF for static appearance and a deformation field to warp points to each timestep.
• Training Data: Requires multi-view video of the dynamic scene.
• Application: Enables free-viewpoint video and temporal interpolation from sparse captures.

Scene flow estimation is the computer vision task of calculating the 3D motion vector field for every point in a scene. It describes the complete 3D velocity of observed geometry between consecutive frames, extending optical flow into the spatial domain.

• Input: Typically consecutive pairs of depth maps or point clouds.
• Output: A dense 3D vector field where each vector represents (ΔX, ΔY, ΔZ) motion.
• Role in Reconstruction: Provides strong motion priors for dynamic NeRF methods, constraining the learned deformation to be physically plausible.
• Challenge: Differentiating between object motion, camera motion, and non-rigid deformations.

Non-rigid registration is the process of aligning two or more 3D scans or point clouds of a deforming object or scene. It estimates a smooth, continuous spatial transformation that accounts for elastic or articulated motion, rather than simple translation or rotation.

• Core Function: Finds correspondence between points in a source cloud and a target cloud under deformation.
• Mathematical Basis: Often modeled using thin-plate splines or Gaussian process deformations.
• Use Case in 4D Capture: Aligning per-frame 3D reconstructions from multi-view video into a temporally consistent sequence.
• Key Differentiator: Unlike rigid registration, it can model bending, stretching, and compression.

Canonical space mapping is a strategy in deformable reconstruction where observations of a deforming object across time are mapped back to a single, fixed reference configuration. This simplifies learning by disentangling appearance from motion.

• Canonical Space: A static, neutral 3D coordinate frame representing the object's base shape.
• Deformation Field: A learned function that maps points from this canonical space to their observed positions at any given time t.
• Benefit: Allows a single neural network to model the object's intrinsic color and density, while a separate network handles time-dependent deformation.
• Analogy: Similar to the concept of a rest pose in skeletal animation.

A temporal coherence loss is a regularization term used when training dynamic neural scene representations. It penalizes unrealistic or abrupt changes in geometry or appearance between consecutive timesteps, enforcing smoothness over time.

• Purpose: To prevent flickering artifacts and ensure the reconstructed motion is physically plausible.
• Common Formulation: Often implemented as an L1 or L2 loss on the difference of radiance field properties (e.g., density, color) for the same spatial point across nearby times.
• Necessity: Without this constraint, models may overfit to individual frames, resulting in a jittery, inconsistent 4D reconstruction.
• Extension: Can be combined with scene flow estimation to create stronger motion-aware penalties.