Fundamental Matrix

The fundamental matrix F is a 3x3 matrix with a rank of exactly 2. This singularity is a direct consequence of its definition: it maps a point x in one image to a line l' = Fx in the other. Since a line is a one-dimensional subspace, the matrix cannot have full rank. This implies det(F) = 0. In practice, algorithms that estimate F often enforce this constraint via Singular Value Decomposition (SVD), setting the smallest singular value to zero.

The primary geometric function of F is to transfer points to lines. For a point x in the first image, the corresponding epipolar line l' in the second image is given by l' = Fx. Conversely, for a point x' in the second image, the epipolar line in the first is l = Fᵀx'. Any correct correspondence x' for x must lie on the line l', satisfying the epipolar constraint: x'ᵀ F x = 0. This is the foundational equation for stereo correspondence search.

The fundamental matrix has 7 degrees of freedom (DoF), even though a general 3x3 homogeneous matrix has 8. This is derived from:

• 9 total matrix elements.
• Minus 1 for scale ambiguity (it's defined up to a non-zero scale factor).
• Minus 1 for the rank 2 constraint (det(F)=0). This explains why a minimum of 7 point correspondences are required for a linear solution (the 7-point algorithm). In comparison, the essential matrix has only 5 DoF, as it operates on calibrated images.

The fundamental matrix F is the generalization of the essential matrix E for uncalibrated cameras. They are related by the camera intrinsic matrices K and K': F = K'⁻ᵀ E K⁻¹. Where E encodes only the relative rotation and translation (E = [t]× R), F incorporates the internal camera parameters. This means F can be estimated from images without prior knowledge of focal length or principal point, making it crucial for Structure from Motion (SfM) pipelines that start from unknown cameras.

The epipoles are the points of intersection of the baseline (the line joining the two camera centers) with the respective image planes. Algebraically, the left epipole e' (in the second image) is the null vector of F (Fᵀ e' = 0), and the right epipole e (in the first image) is the null vector of Fᵀ (F e = 0). This property is used to directly extract the epipoles from a computed F and is critical for recovering camera motion.

Estimating F from noisy, real-world point correspondences is a central task. The 8-point algorithm provides a linear least-squares solution but is highly sensitive to outliers. Therefore, robust estimation frameworks like RANSAC are universally employed. Within each RANSAC iteration, a minimal sample (7 or 8 points) is used to hypothesize an F, and the epipolar constraint (x'ᵀ F x) is used to count inliers. The final matrix is often refined via non-linear optimization, minimizing the symmetric epipolar distance.

The fundamental matrix enables stereo rectification, a process that warps two images so their epipolar lines become horizontal and aligned. This simplifies the correspondence search for stereo matching and depth estimation to a one-dimensional scan along image rows. It is a critical pre-processing step for efficient dense stereo algorithms used in 3D reconstruction and robot navigation.

The epipolar constraint defined by the fundamental matrix acts as a powerful geometric filter for feature matching. After an initial set of putative matches is found (e.g., using SIFT or ORB descriptors), matches that violate the constraint—where the distance from a point to its corresponding epipolar line is too large—are rejected as outliers. This is often implemented within a RANSAC framework to compute the matrix itself from noisy data, dramatically improving match quality for subsequent Structure from Motion (SfM).

While the fundamental matrix does not provide metric scale, it enables projective reconstruction. Given the matrix and point correspondences, cameras can be retrieved up to a projective transformation of the 3D scene. This is the first step in hierarchical reconstruction pipelines. Further steps using techniques like self-calibration can upgrade this to a metric reconstruction, forming the basis of many photogrammetry software packages that work with unknown cameras.

In image-based rendering, the fundamental matrix is used to establish the geometric relationship between reference images. For a desired virtual camera view, corresponding pixels between source images can be transferred and blended accurately. This is foundational for creating seamless panoramas and for early methods in light field rendering, where knowing the epipolar geometry is essential for interpolating between sampled views.

For a robot or AR device with an uncalibrated camera, the fundamental matrix can estimate motion between frames (visual odometry). By decomposing the matrix (often into the essential matrix after approximate calibration), relative rotation and translation direction (up to scale) can be recovered. This provides a drift-prone but usable motion estimate and is a component in some Simultaneous Localization and Mapping (SLAM) systems, especially during initialization.

The fundamental matrix is a key input for self-calibration algorithms, such as the Kruppa equations. These methods exploit constraints from multiple fundamental matrices computed from images of a static scene taken by the same camera with varying poses. By analyzing these matrices, it's possible to solve for the camera's constant intrinsic parameters (like focal length), enabling a transition from a projective to a metric reconstruction without a physical calibration target.

Epipolar Geometry is the intrinsic projective geometry between two views. It is independent of scene structure, depending only on the cameras' internal parameters and relative pose. Key elements include:

• Epipoles: The points where the baseline (line connecting camera centers) intersects the image planes.
• Epipolar Lines: The projection of the ray from one camera center through a 3D point onto the other image plane. Corresponding points must lie on conjugate epipolar lines.
• Epipolar Constraint: The mathematical relationship x'^T F x = 0, where x and x' are corresponding points, and F is the Fundamental Matrix. This constraint reduces the search for correspondences from a 2D region to a 1D line.

Camera Calibration is the process of estimating a camera's intrinsic parameters (focal length, principal point, skew, distortion coefficients) and extrinsic parameters (pose relative to a world coordinate system). Accurate calibration is a prerequisite for moving from the Fundamental Matrix (uncalibrated) to the Essential Matrix (calibrated). Common methods involve capturing images of a known calibration pattern (like a checkerboard) and using algorithms like Zhang's method or the Direct Linear Transform (DLT) to solve for the parameters. The calibration matrix K is used to normalize image coordinates: x_norm = K^{-1} x_pixel.

Structure from Motion is the process of reconstructing 3D scene structure and camera poses from a collection of 2D images. The Fundamental Matrix is a foundational component in the two-view reconstruction pipeline:

1. Feature Matching & F Estimation: Find point correspondences and compute F between image pairs.
2. Camera Recovery: From F and calibration K, extract the Essential Matrix E and then the relative camera pose (R, t).
3. Triangulation: Estimate the 3D coordinates of matched points using the computed camera poses.
4. Bundle Adjustment: A non-linear optimization that jointly refines all 3D points and camera parameters to minimize total reprojection error. SfM scales this process to many images, often starting with a robust two-view reconstruction seeded by F.

Triangulation is the process of determining the 3D world coordinates of a point given its 2D projections in two (or more) images and the known poses (projection matrices P and P') of the cameras that captured them. It is the next step after computing the Fundamental Matrix and recovering camera motion. The 3D point X is found at the intersection of the two back-projected rays from the image points. In practice, due to noise, these rays do not intersect perfectly, so methods like Linear Least-Squares (solving AX = 0 via SVD) or Optimal Triangulation (iteratively correcting the image points to minimize reprojection error) are used. Accurate triangulation is essential for building dense 3D models in SfM and SLAM.