Reprojection Error

Reprojection error is the Euclidean distance in the image plane between an observed 2D image point and the projection of its corresponding estimated 3D point. Formally, for a 3D point X and camera with projection function P, the error for its observed image point x is: || x - P(X) ||. This pixel-wise distance is the residual minimized in a non-linear least squares optimization framework.

In Bundle Adjustment, the total reprojection error across all cameras and all observed points is the objective function. The optimization jointly refines:

• 3D point coordinates (structure)
• Camera poses (motion)
• Camera intrinsic parameters (e.g., focal length)

Minimizing the sum of squared reprojection errors yields the maximum likelihood estimate under the assumption of Gaussian image noise. This is the backbone of Structure from Motion (SfM) and Visual SLAM.

Raw squared error (L2 norm) is highly sensitive to mismatched features (outliers). Robust cost functions are applied to limit their influence:

• Huber Loss: Quadratic for small errors, linear for large errors.
• Cauchy / Tukey Loss: Saturates for large residuals, effectively ignoring gross outliers.

These are used in conjunction with RANSAC during initial pose estimation (e.g., in Perspective-n-Point (PnP)) to pre-filter outlier correspondences before global bundle adjustment.

The mean and median reprojection error, measured in pixels, are standard metrics for evaluating reconstruction quality.

• < 0.5 pixels: Excellent, near-theoretical limit.
• 0.5 - 1.0 pixels: Good, typical for well-calibrated systems.
• > 2.0 pixels: Poor, indicates issues with calibration, feature matching, or model assumptions. High error can indicate poor camera calibration, incorrect lens distortion modeling, or excessive image noise.

Reprojection error directly validates the assumed pinhole camera model. It enforces consistency with epipolar geometry; for a stereo pair, a point's reprojection error is minimized when it lies on the corresponding epipolar line. The error's behavior during optimization helps diagnose problems: systematic errors may reveal incorrect focal length estimates, while spatially varying errors can indicate unmodeled radial distortion.

In Visual-Inertial Odometry (VIO) and Visual-Inertial SLAM, reprojection error forms the visual measurement term in a factor graph or filtering framework. It is combined with IMU pre-integration terms that model motion dynamics. The fusion provides scale observability (missing from monocular vision) and improves robustness during periods of poor texture or motion blur, where visual feature tracking and, consequently, reprojection error become unreliable.

Bundle adjustment is the non-linear optimization process that minimizes the total reprojection error across all cameras and all observed 3D points in a scene. It is the final, most accurate step in Structure from Motion (SfM) and SLAM pipelines.

• Function: Jointly refines 3D point positions, camera intrinsics, and camera extrinsics.
• Algorithm: Typically uses the Levenberg-Marquardt algorithm to solve this large, sparse least-squares problem.
• Output: Produces a globally consistent 3D reconstruction and camera trajectory.

Perspective-n-Point is the problem of estimating the camera pose (extrinsics) given a set of n known 3D points and their corresponding 2D image projections. The solution minimizes the reprojection error for those correspondences.

• Minimal Solutions: P3P (3 points) is a common minimal case, often solved within a RANSAC loop for robustness.
• Non-Minimal Solutions: For n > 3, methods like EPnP or iterative Levenberg-Marquardt optimization provide more stable estimates.
• Use Case: Essential for augmented reality (overlaying virtual objects) and robot localization.

The Direct Linear Transform is a linear algorithm used to compute a projective transformation (like a homography or a camera projection matrix) from point correspondences. It provides an initial solution that is often refined by minimizing reprojection error.

• Mechanism: Formulates the problem as a homogeneous linear system Ah = 0, solved via Singular Value Decomposition (SVD).
• Limitation: Minimizes an algebraic error, not the geometrically meaningful reprojection error. Sensitive to noise and outliers.
• Application: Foundational step in camera calibration and image stitching.

RANSAC (Random Sample Consensus) is a robust estimation algorithm used to fit a model (e.g., a fundamental matrix or homography) to data contaminated by outliers. It evaluates model hypotheses by counting inliers, often defined by a reprojection error threshold.

• Process: 1) Randomly sample minimal data points. 2) Compute model. 3) Score model by counting inliers (points with reprojection error < threshold). 4) Repeat and keep the best model.
• Purpose: Isolates a clean set of correspondences for reliable camera pose estimation and 3D reconstruction.
• Key Parameter: The inlier threshold, defined in pixels of reprojection error.

Epipolar geometry describes the projective relationship between two views. The fundamental matrix (F) encapsulates this geometry for uncalibrated cameras. The reprojection error in this context is often the Sampson distance or the distance from a point to its epipolar line.

• Constraint: For corresponding points x and x', the epipolar constraint is x'ᵀ F x = 0.
• Error Metric: The geometric error is the distance from x' to the epipolar line Fx, a form of reprojection error.
• Role: Used in two-view camera pose estimation and as a model for RANSAC in feature matching.

Triangulation is the process of computing the 3D world coordinates of a point from its 2D observations in two or more images with known camera poses. The accuracy of a triangulated point is directly measured by its reprojection error across all observing cameras.

• Optimal Method: Linear triangulation (via SVD) provides an initial estimate, often refined by non-linear optimization to minimize reprojection error.
• Cheirality Check: Ensures the triangulated point is in front of all cameras.
• Dependency: The precision of triangulation is highly sensitive to errors in the estimated camera poses, which are themselves optimized using reprojection error.