Video-Based Reconstruction

Structure from Motion (SfM) is the foundational geometric pipeline for estimating 3D scene structure and simultaneous camera poses from a collection of 2D images or video frames. It operates by detecting and matching distinctive keypoints (e.g., using SIFT or ORB) across frames, then solving a bundle adjustment problem to minimize reprojection error.

• Output: A sparse 3D point cloud and camera trajectory.
• Key Challenge: Requires sufficient parallax (camera motion) and textured surfaces for reliable matching.
• Example: Building a 3D map of a landmark from a tourist's smartphone video.

Multi-View Stereo (MVS) is a dense reconstruction technique that follows SfM. It uses the known camera poses to estimate depth for every pixel, generating a dense point cloud or mesh. Unlike SfM, MVS matches local image patches to find correspondences.

• Core Process: For each pixel in a reference image, MVS searches along its epipolar line in other views to find the depth that maximizes photo-consistency.
• Output: A dense 3D model (point cloud, mesh, or depth maps).
• Common Algorithms: PatchMatch Stereo, Plane Sweep Stereo. It is computationally intensive and sensitive to lighting changes and untextured regions.

Neural Radiance Fields (NeRF) provide a neural, implicit representation of a static scene. For video, this is extended to Dynamic NeRF or Neural Scene Flow Fields (NSFF), which model scenes changing over time.

• Core Idea: A multilayer perceptron (MLP) maps a 3D spatial coordinate and a time parameter to a volume density and view-dependent color.
• Training: The network is optimized via differentiable volume rendering to minimize the error between rendered and observed video frames.
• Advantage: Produces photorealistic novel views at arbitrary times, handling complex effects like reflections and semi-transparency. It learns a continuous 4D function: F(x, y, z, t) → (σ, rgb).

Scene Flow Estimation is the 3D equivalent of optical flow. It calculates a 3D motion vector for every point in the scene, describing how the geometry moves between frames. This is critical for separating dynamic objects from static backgrounds.

• Definition: A vector field V(x, y, z) = (u, v, w) representing 3D displacement.
• Methods: Can be estimated from multi-view video using geometric constraints or learned directly by neural networks (e.g., RAFT-3D).
• Application: Essential for dynamic object segmentation, motion prediction, and providing temporal coherence in 4D reconstruction pipelines.

For scenes with elastic or articulated motion (e.g., people, clothing), non-rigid registration aligns scans across time. In neural methods, this is often achieved via a deformation field.

• Deformable NeRF Approach: Learns a continuous function that maps a point from a canonical, static 3D space to its observed, deformed position at each timestep: T(x_canonical, t) → x_observed.
• Benefit: Separates learning of appearance (in canonical space) from complex motion, improving generalization and reducing artifacts.
• Related Concept: Skinning weight networks predict how bones in an articulated model influence point deformation, similar to linear blend skinning in CGI.

As an alternative to implicit NeRFs, 4D Gaussian Splatting uses an explicit, point-based representation for dynamic scenes. Each point is modeled as a 3D Gaussian whose attributes are functions of time.

• Core Element: A set of 4D Gaussians with parameters: position (x,y,z,t), rotation, scale, opacity, and spherical harmonics for color.
• Rendering: Uses tile-based rasterization and alpha blending, enabling real-time performance.
• Advantage: Extremely fast training and rendering compared to vanilla NeRF, suitable for interactive applications. It captures motion by interpolating Gaussian attributes over time.

Video-based reconstruction provides ego-motion estimation and dynamic scene understanding for robots and autonomous vehicles. By processing onboard camera feeds, systems can:

• Build local 3D maps of changing environments in real-time.
• Identify and track moving obstacles (dynamic object segmentation).
• Estimate scene flow to predict the future motion of pedestrians and vehicles. This enables safe path planning in unstructured, non-static worlds without relying solely on pre-built maps or lidar.

This application focuses on creating and maintaining high-fidelity digital replicas of physical assets (factories, buildings, infrastructure) from routine inspection or surveillance video. Key processes include:

• 4D reconstruction to model not just geometry but also wear, thermal changes, or assembly progress over time.
• Change detection by comparing reconstructions from different dates.
• Integration with Building Information Modeling (BIM) systems. It allows for remote monitoring, predictive maintenance, and virtual training on an always-current model.

Enables persistent and context-aware AR experiences by understanding the user's environment as a dynamic 4D model. Core uses are:

• Occlusion handling: Virtual objects correctly pass behind and in front of real, moving people.
• Physics-based interaction: Virtual content can collide with and respond to reconstructed real-world geometry.
• Multi-user persistence: A 3D map of a changing room can be shared and updated across devices. On-device 3D reconstruction is critical for low-latency, privacy-preserving AR on mobile and head-worn devices.

Aims to transmit a user's full 3D presence, including gestures and expressions, over a network. Video-based reconstruction enables:

• 3D video conferencing: Participants are rendered as volumetric assets viewable from any angle in a shared virtual space.
• Holographic communication: Using devices like the Microsoft HoloLens to project a life-like 3D model of a remote participant into a local environment.
• Light field displays: Providing motion parallax without needing headsets. This relies heavily on facial performance capture and real-time neural rendering pipelines.

Uses crowd-sourced or municipal video (traffic cameras, drones) to model city-scale dynamics for analysis and simulation. Specific tasks include:

• Traffic flow analysis and optimization by reconstructing vehicle and pedestrian movement patterns.
• Crowd simulation validation and calibration using real-world 4D data.
• Environmental impact studies, such as visualizing shadow propagation or wind flow around new buildings over a full day. This application often involves large-scale 4D semantic segmentation to classify moving entities (cars, bikes, people).

The process of creating a time-varying, dynamic 3D model of a scene from a sequence of images or videos. Unlike static 3D reconstruction, it captures both geometry and its evolution over time, producing a spatio-temporal volume. Key applications include:

• Free-viewpoint video for sports and entertainment
• Digital twins of manufacturing or logistical processes
• Long-term scene understanding for autonomous systems

An extension of the Neural Radiance Field (NeRF) framework designed to model scenes with non-rigid motion and time-varying appearance. It incorporates temporal parameters into the neural representation, allowing the model to output different density and color values for the same spatial coordinate at different times. Core innovations include:

• Deformation fields that map points from a canonical space to observed positions
• Temporal latent codes that compactly encode the scene state at a given moment
• Temporal coherence losses to ensure smooth motion

The computer vision task of calculating the 3D motion vector field for every point in a scene. It describes how the observed geometry moves between consecutive frames, providing a dense, per-point 3D displacement. This is a more complete description than 2D optical flow, as it accounts for depth. It is critical for:

• Dynamic object tracking in autonomous driving
• Motion segmentation to separate independently moving objects
• Providing supervision for learning-based 4D reconstruction methods

The process of aligning two or more 3D scans or point clouds of a deforming object or scene. It estimates a smooth, continuous spatial transformation (unlike rigid registration) that accounts for elastic, articulated, or topological changes. Common techniques involve:

• Iterative Closest Point (ICP) variants with deformation models
• Embedded deformation graphs for efficient surface manipulation
• Neural deformation fields learned from data Essential for merging sequential scans in dynamic 3D reconstruction pipelines.

An explicit, point-based representation for dynamic 3D scenes that extends 3D Gaussian Splatting. It models each point as a 3D Gaussian whose attributes—position, rotation, scale, opacity, and spherical harmonic coefficients for color—are defined as continuous functions of time. Advantages include:

• Real-time rendering performance for dynamic content
• Explicit optimization of scene properties without neural network queries
• High visual fidelity suitable for interactive free-viewpoint video

A seminal method that jointly learns a time-varying neural radiance field and a 3D scene flow field from monocular video. It introduces the concept of ray bending, where a ray cast at time t is warped according to the estimated flow to query a canonical, static NeRF. This enables:

• Novel view synthesis for dynamic scenes from a single moving camera
• Unsupervised learning of 3D scene flow without ground-truth data
• Motion-compensated rendering that reduces blur and artifacts