Essential Matrix

The essential matrix is a 3x3 matrix, denoted as E, that encapsulates the epipolar constraint between corresponding points in two calibrated camera views. It is defined as E = [t]× R, where R is the 3x3 rotation matrix, t is the 3x1 translation vector between cameras, and [t]× is the skew-symmetric matrix of t. For corresponding normalized image points x and x', the constraint is expressed as x'ᵀ E x = 0.

The essential matrix has a rank of 2. This is a direct consequence of its construction from a skew-symmetric matrix [t]×, which is always rank 2 for a non-zero translation. This property distinguishes it from a general 3x3 matrix and is enforced during estimation, often via Singular Value Decomposition (SVD). The two non-zero singular values are equal, a condition known as the singular value constraint (σ₁ = σ₂, σ₃ = 0).

The essential matrix encodes the relative pose between two cameras with five degrees of freedom (DoF).

• 3 DoF for rotation (yaw, pitch, roll).
• 2 DoF for the direction of translation (up to an unknown scale). The translation's magnitude cannot be recovered from image correspondences alone, leading to the well-known scale ambiguity in monocular vision. This 5-DoF structure is reflected in the matrix having only five independent parameters.

A valid essential matrix can be decomposed into four possible combinations of rotation R and translation t. Given E = U diag(1, 1, 0) Vᵀ from its SVD, the four solutions are:

• R = U W Vᵀ or R = U Wᵀ Vᵀ
• t = +u₃ or t = -u₃, where u₃ is the third column of U. Here, W is a fixed rotation matrix. The correct physical solution is identified by the cheirality constraint, requiring that triangulated 3D points lie in front of both cameras.

The essential matrix is the calibrated counterpart of the fundamental matrix (F). They are related by the camera intrinsic matrices K and K': E = K'ᵀ F K. While the fundamental matrix describes epipolar geometry for uncalibrated cameras (7 DoF), the essential matrix assumes known intrinsics, reducing the ambiguity and enabling direct metric pose recovery. This makes E crucial for Visual Odometry (VO) and Structure from Motion (SfM) where calibration is known.

The Eight-Point Algorithm is the classic linear method for estimating E from at least eight point correspondences, solving the system x'ᵀ E x = 0. To enforce the rank-2 constraint, the solution is projected onto the essential matrix space via SVD. In practice, RANSAC (Random Sample Consensus) is used robustly to handle outliers in feature matching. More efficient five-point algorithms exist, leveraging the matrix's five degrees of freedom for minimal solutions.

The essential matrix's most direct application is recovering the relative pose (rotation and translation up to scale) between two calibrated cameras. Given point correspondences, the matrix is decomposed via Singular Value Decomposition (SVD) to extract the rotation matrix R and a skew-symmetric matrix representing the translation direction t. This is a critical step in Visual Odometry (VO) and Structure from Motion (SfM) pipelines to initialize camera motion.

• Process: Compute E from correspondences, then enforce the internal constraint via SVD: E = U diag(1,1,0) V^T. The four possible solutions for (R, t) are tested with a single 3D point to select the physically valid configuration where the point is in front of both cameras.

The essential matrix encodes epipolar geometry, providing a powerful constraint for matching features between images. For a point x in the first image, its corresponding point x' in the second image must lie on the epipolar line defined by l' = E x. This reduces the search for matches from the entire second image to a single line, drastically improving efficiency and accuracy in feature matching.

• Example: In dense stereo matching, this constraint guides the search along scanlines after image rectification.
• Robust Estimation: Algorithms like RANSAC are used with the essential matrix to find a set of inlier correspondences while rejecting outliers from incorrect matches.

Once the relative camera pose is recovered from the essential matrix, it enables precise triangulation of 3D scene points. Given the known camera matrices P = K[I | 0] and P' = K'[R | t] (where K are the calibration matrices), the 3D location of a point seen in both images can be computed by solving a linear least-squares problem that finds the intersection of its back-projected rays.

• Foundation for SfM: This process is the backbone of Structure from Motion, building a sparse 3D point cloud from multiple views.
• Accuracy: The quality of the essential matrix directly impacts triangulation accuracy; errors propagate into 3D reconstruction noise.

In Visual Odometry (VO) and Visual-Inertial Odometry (VIO) systems, the essential matrix provides a metric (but scale-ambiguous) estimate of frame-to-frame motion for calibrated cameras. It serves as a robust initialization step before non-linear refinement via bundle adjustment.

• Scale Ambiguity: The translation t is recovered only up to an unknown scale factor. This scale is often resolved by integrating IMU data (in VIO) or by using known physical distances in the scene.
• SLAM: In Simultaneous Localization and Mapping, the essential matrix helps bootstrap the map by establishing the initial camera poses and 3D landmarks for keyframes.

The essential matrix is instrumental in self-calibration techniques, where camera intrinsic parameters are unknown. The relationship between the essential matrix E and the fundamental matrix F is E = K'^T F K, where K and K' are the calibration matrices. If the fundamental matrix is estimated from uncalibrated images, constraints on E (like its singular values being [σ, σ, 0]) can be used to solve for the intrinsic parameters.

• Kruppa Equations: This classical self-calibration method derives constraints from the essential matrix to solve for the camera's focal lengths and principal point.

The essential matrix acts as a robust geometric model for separating independently moving objects from a static background. In a multi-body scenario (e.g., a robot moving past a walking person), RANSAC can be used to fit multiple essential matrices, each corresponding to a different rigid motion.

• Process: Iteratively sample point correspondences, estimate an essential matrix, and count inliers. Points consistent with the dominant motion (static scene) form the largest inlier set. Remaining outliers may belong to other moving objects.
• Application: Critical for autonomous vehicles to distinguish between the ego-motion of the car and the motion of other traffic participants.

The fundamental matrix (F) is the uncalibrated counterpart to the essential matrix. It is a 3x3 matrix of rank 2 that describes the epipolar geometry between two cameras with unknown intrinsic parameters. While the essential matrix encodes metric motion (rotation and translation up to scale), the fundamental matrix encodes projective motion. Given camera calibration matrices K and K', the relationship is: E = K'ᵀ F K.

Epipolar geometry is the projective geometry between two views. It provides constraints without knowing the 3D scene structure. Key elements include:

• Epipoles: The projection of one camera center onto the other's image plane.
• Epipolar Lines: The line in one image where the corresponding point in the other image must lie.
• Epipolar Constraint: For corresponding points x and x', the constraint is x'ᵀ F x = 0 (or x'ᵀ E x = 0 for calibrated cameras). The essential matrix is the algebraic representation of this geometry for calibrated cameras.

Camera calibration is the prerequisite process for obtaining the essential matrix. It determines the intrinsic parameters of a camera (focal length, principal point, skew, distortion). These parameters form the calibration matrix K. The essential matrix can only be derived from image correspondences if the cameras are calibrated, as it is defined for normalized image coordinates: x̂ = K⁻¹ x, where x is a pixel coordinate. Without accurate K, only the fundamental matrix can be estimated.

Singular Value Decomposition is the critical linear algebra operation used to enforce the internal constraints of the essential matrix. A valid essential matrix must have two equal singular values and a third of zero (i.e., singular values [σ, σ, 0]). After estimating a matrix from point correspondences (e.g., using the Eight-Point Algorithm), the result is projected onto the essential matrix space via SVD: E = U diag(1, 1, 0) Vᵀ. This projection ensures the matrix satisfies the required rank and singular value constraints.

Pose decomposition is the process of extracting the relative rotation matrix R and translation vector t (up to scale) from the essential matrix E. Given E = [t]ₓ R, the decomposition is performed via SVD: E = U Σ Vᵀ. There are four mathematically possible solutions for (R, t). The correct pose is identified by the cheirality constraint: choosing the solution where the reconstructed 3D points lie in front of both cameras. This yields the 5-degree-of-freedom relative motion.

The Five-Point Algorithm is a minimal solution for estimating the essential matrix from five point correspondences, in contrast to the linear Eight-Point Algorithm. It is more efficient and numerically stable for calibrated cameras because it directly uses the minimal number of parameters (5 degrees of freedom for E). The algorithm solves a system of polynomial equations, often using the Gauss-Jordan elimination or a Groebner basis method. It is the foundation for robust estimation in modern Structure from Motion (SfM) and Visual Odometry (VO) pipelines.