Pinhole Camera Model

The intrinsic matrix encodes the internal geometry of the camera, transforming 3D coordinates in the camera's coordinate system into 2D pixel coordinates on the image plane. It is defined as:

K = [[f_x, s, c_x], [0, f_y, c_y], [0, 0, 1]]

• f_x, f_y: Focal lengths in pixels, determining the field of view.
• c_x, c_y: Coordinates of the principal point (image center), where the optical axis pierces the image plane.
• s: Skew coefficient, often zero for modern digital sensors. This matrix is central to the camera calibration process.

The extrinsic parameters define the camera's pose—its position and orientation—in the world coordinate system. They consist of:

• Rotation Matrix (R): A 3x3 orthogonal matrix (R^T R = I) representing the camera's orientation.
• Translation Vector (t): A 3x1 vector representing the camera's position. Together, they form a rigid transformation [R | t] that maps world points (X_w, Y_w, Z_w) to the camera's coordinate system: X_c = R * X_w + t. This is the target of camera pose estimation algorithms like PnP.

The core linear equation of the pinhole model combines intrinsic and extrinsic parameters to project a 3D point X_w to a 2D pixel x:

x = K [R | t] X_w

In homogeneous coordinates, this is written as: s * [u, v, 1]^T = K [R | t] [X, Y, Z, 1]^T

• s is an arbitrary scale factor (the projective depth).
• [u, v] are the pixel coordinates.
• The full 3x4 matrix P = K [R | t] is called the camera projection matrix. This equation is the foundation for triangulation and reprojection error calculation.

The image plane is a hypothetical plane positioned at a distance f (focal length) behind the pinhole aperture where the image is formed. Key elements include:

• Optical Axis: The line perpendicular to the image plane that passes through the pinhole (center of projection).
• Principal Point: The intersection of the optical axis with the image plane, its pixel coordinates are (c_x, c_y) in the intrinsic matrix.
• Field of View (FOV): Determined by the focal length and sensor size: FOV = 2 * arctan(sensor_width / (2 * f)). A shorter focal length yields a wider FOV.

The basic pinhole model is linear, but real lenses introduce non-linear radial and tangential distortion. These are modeled separately and corrected prior to using the linear model.

• Radial Distortion: Causes straight lines to appear curved. Modeled by coefficients k1, k2, k3, ....
• Tangential Distortion: Caused by lens misalignment. Modeled by coefficients p1, p2. Correction involves applying a distortion model to normalized image coordinates before multiplication by the intrinsic matrix K. This is a critical step in camera calibration pipelines.

Homogeneous coordinates are the mathematical framework that allows the pinhole projection to be expressed as a linear matrix multiplication. A 2D point (x, y) is represented as (x, y, w), where the actual Cartesian coordinates are (x/w, y/w).

• Enables representation of points at infinity (w=0).
• Makes the projection equation x = P X linear and elegant.
• Essential for solving problems like the Direct Linear Transform (DLT) for estimating the projection matrix P from point correspondences.

The model is defined by a projection matrix that maps 3D world points (X, Y, Z) to 2D image coordinates (u, v). This linear transformation combines intrinsic parameters (focal length, principal point) and extrinsic parameters (camera rotation and translation). The core equation is: [u, v, 1]^T ~ K [R | t] [X, Y, Z, 1]^T, where ~ denotes equality up to scale, K is the intrinsic matrix, and [R | t] is the extrinsic matrix defining the camera's pose.

These internal parameters define the camera's imaging geometry and are contained in the calibration matrix K. Key components include:

• Focal Length (f_x, f_y): The distance from the pinhole to the image plane, measured in pixels. It controls the field of view.
• Principal Point (c_x, c_y): The image coordinates where the optical axis intersects the image plane, often near the image center.
• Skew Coefficient (s): Accounts for non-rectangular pixels, though it is often zero for modern digital sensors. Calibration (e.g., using a checkerboard) is required to determine these values accurately.

Extrinsic parameters define the camera's pose—its position and orientation in the world. They are represented by a 3x3 rotation matrix R and a 3x1 translation vector t. Together, they form a rigid transformation [R | t] that converts world coordinates into camera coordinates. Estimating this pose (camera pose estimation) is a central problem in robotics (SLAM, VIO) and augmented reality, enabling a device to understand its own location in 3D space.

The pinhole model is essential for inverting the imaging process to recover 3D structure. Key techniques rely on it:

• Triangulation: Determines a 3D point's location from its 2D projections in two images with known camera poses.
• Structure from Motion (SfM): Recovers both the 3D scene geometry and all camera poses from a collection of 2D images.
• Stereo Vision: Uses two calibrated cameras (a stereo rig) to compute depth via disparity, directly applying the pinhole geometry.

The basic model is an idealization. Real-world lenses introduce distortions (radial and tangential) not captured by the linear model. Therefore, the standard pipeline involves:

1. Undistortion: Applying a non-linear correction to raw images using estimated distortion coefficients.
2. Projection: Using the corrected image with the ideal pinhole model. More advanced models (e.g., fisheye, omnidirectional) extend the pinhole concept to handle ultra-wide fields of view.

This model is not just for classical geometry; it's integral to deep learning-based vision:

• Differentiable Rendering: Neural networks (like those in NeRF) use a differentiable version of the pinhole projection to learn 3D scenes from 2D images via gradient descent.
• Camera Parameter Learning: In models for novel view synthesis, the network often learns to infer or refine pinhole camera parameters alongside the scene representation.
• Data Synthesis & Simulation: Generating realistic training data for autonomous systems requires accurate camera models to simulate perspective and occlusion correctly.

Camera intrinsics are the internal parameters of a camera that define its imaging geometry within the pinhole model. They are represented by a 3x3 matrix (K) and include:

• Focal length (fx, fy): The distance from the pinhole to the image plane, controlling the field of view.
• Principal point (cx, cy): The optical center's projection onto the image plane, often near the image center.
• Skew coefficient (s): Accounts for non-rectangular pixels (often zero for modern sensors).
• Lens distortion coefficients (k1, k2, p1, p2, ...): Model radial and tangential deviations from the ideal pinhole projection, corrected prior to using the linear model. Intrinsic calibration is a prerequisite for accurate metric 3D reconstruction and is performed using known patterns like checkerboards.

Camera extrinsics define the position and orientation (pose) of a camera in a world coordinate system, completing the pinhole projection equation. They consist of:

• A 3x3 rotation matrix (R) representing the camera's orientation.
• A 3x1 translation vector (t) representing the camera's position. Together, they form a 3x4 matrix [R | t] that transforms 3D world points into the camera's coordinate frame. Estimating extrinsics is the core problem of camera pose estimation, solved by algorithms like Perspective-n-Point (PnP) given known 3D-2D correspondences and calibrated intrinsics.

Perspective-n-Point (PnP) is the problem of estimating a calibrated camera's pose (extrinsics) given a set of n known 3D points in a world coordinate system and their corresponding 2D projections in the image. It directly applies the pinhole camera model.

• A minimum of 3 points (P3P) is required for a finite number of solutions, with 4 or more (e.g., EPnP, UPnP) providing a unique, stable solution via least-squares.
• PnP solvers are fundamental to augmented reality (overlaying virtual objects), robotics (localization), and are a key component in Structure from Motion (SfM) pipelines.
• Robust estimators like RANSAC are typically wrapped around PnP to handle outlier correspondences from erroneous feature matching.

A homography is a 3x3 projective transformation matrix that maps points from one plane to another. Under the pinhole model, if all observed 3D points lie on a common world plane, the relationship between their world coordinates (on that plane) and image coordinates simplifies to a homography.

• This is a special case of the general pinhole projection, reducing the problem from 3D-to-2D to 2D-to-2D.
• Homography estimation is used for:
  • Image stitching (panorama creation).
  • Augmented reality overlays on planar surfaces like floors or posters.
  • Camera pose estimation from a single plane, though this yields up to four ambiguous solutions.

Epipolar geometry describes the intrinsic projective geometry between two views of a scene, derived from the pinhole model. It constrains the search for corresponding points. Key entities include:

• Epipoles: The projection of one camera center onto the other's image plane.
• Epipolar lines: The line in one image where the corresponding point in the other image must lie, given a point in the first image.
• Fundamental Matrix (F): Encodes this geometry for uncalibrated cameras. For calibrated cameras, it becomes the Essential Matrix (E), from which relative camera rotation and translation (up to scale) can be extracted. This geometry is the foundation for stereo vision, Structure from Motion, and efficient feature matching.

Reprojection error is the quantitative measure of how well an estimated camera model (intrinsics and extrinsics) and 3D structure fit the observed image data. It is the Euclidean distance in the image plane between an observed 2D keypoint and the projection of its corresponding estimated 3D point via the pinhole camera model.

• This error is the core cost function minimized in bundle adjustment, a non-linear optimization that jointly refines all camera parameters and 3D point positions.
• Low average reprojection error (e.g., < 0.5 pixels) indicates a highly accurate and self-consistent reconstruction. It is the primary metric for evaluating the quality of Structure from Motion and camera calibration results.