Camera Intrinsics

The focal length is the distance between the camera's optical center and the image plane, measured in pixels. It defines the camera's field of view and magnification.

• fx, fy: Represent the focal length in the x and y image directions. They are often equal for square pixels.
• Real-world effect: A longer focal length creates a narrower field of view (zoom), while a shorter one creates a wider field of view (wide-angle).
• Example: In a calibration matrix K, fx and fy are the diagonal elements K[0,0] and K[1,1].

The principal point is the pixel coordinates where the optical axis intersects the image plane. It is the image center in an ideal, undistorted camera.

• cx, cy: Represent the translation of the image center from the top-left corner (0,0).
• Real-world deviation: Due to manufacturing imperfections, the principal point is often slightly offset from the true image center.
• Example: In the calibration matrix K, cx and cy are the elements K[0,2] and K[1,2].

Lens distortion parameters model deviations from the ideal pinhole model caused by lens imperfections. They are critical for accurate geometric measurements.

• Radial Distortion (k1, k2, k3): Causes straight lines to appear curved. Positive coefficients model barrel distortion; negative coefficients model pincushion distortion.
• Tangential Distortion (p1, p2): Arises from lens misalignment, causing the lens and image plane to be non-parallel.
• Correction: These coefficients are used in functions like OpenCV's cv2.undistort() to rectify images before processing.

The skew coefficient accounts for non-rectangular pixels, representing the angle between the x and y pixel axes. For most modern digital cameras with square pixels, this value is zero.

• Cause: Historically present in certain analog video systems or specific sensor designs.
• Mathematical role: In the calibration matrix K, it is the element K[0,1]. When zero, the matrix is upper-triangular.
• Practical note: In most computer vision pipelines, skew is assumed to be zero, simplifying the intrinsic matrix to five parameters: fx, fy, cx, cy, and a fixed scale factor.

The camera calibration matrix (or intrinsic matrix) K is a 3x3 matrix that consolidates the core intrinsic parameters for projective geometry.

Standard Form:

codeCopy
K = [ fx  s  cx ]
    [ 0  fy  cy ]
    [ 0   0   1 ]

• It transforms 3D points in the camera coordinate system to 2D coordinates on the image plane in homogeneous coordinates.
• The full projection from world to pixel coordinates is x_pixel = K * [R | t] * X_world, where [R | t] are the extrinsic parameters.

Accurate camera intrinsics are a prerequisite for robust camera pose estimation. They decouple the camera's internal properties from its position in the world.

• PnP Solvers: Algorithms like Perspective-n-Point (PnP) require known intrinsics to estimate the camera's extrinsic rotation and translation from 2D-3D correspondences.
• Visual Odometry/SLAM: Systems like Visual Inertial Odometry (VIO) and Simultaneous Localization and Mapping (SLAM) rely on stable intrinsics for feature tracking, triangulation, and bundle adjustment.
• Reprojection Error: Intrinsics directly define the projection model used to compute reprojection error, the core optimization metric in bundle adjustment.

Camera intrinsics are the essential first step in 3D reconstruction and Structure from Motion (SfM). They convert 2D pixel measurements into 3D rays, enabling the triangulation of points. Without accurate intrinsics, the estimated 3D structure will be distorted. This is critical for applications like:

• Photogrammetry for creating digital twins of buildings or terrain.
• Autonomous vehicle perception systems that build local 3D maps.
• Cultural heritage digitization, where precise geometry is paramount.

In robotics and Visual Odometry (VO), a robot uses its camera to estimate its own motion. Accurate intrinsics ensure that the apparent motion of pixels correctly translates to real-world rotational and translational movement. Errors in focal length or distortion coefficients directly cause drift in the robot's estimated trajectory. This is foundational for:

• Autonomous navigation of drones and mobile robots.
• Simultaneous Localization and Mapping (SLAM) systems.
• Augmented Reality (AR) devices that track their position in a room.

For Augmented Reality (AR) to convincingly overlay virtual objects, the virtual camera must perfectly match the real camera's projection model. This requires precise intrinsics (focal length, principal point, distortion) so that virtual geometry aligns with the real world's perspective and lines appear straight. This calibration enables:

• Realistic occlusion and lighting interactions.
• Stable object anchoring that doesn't 'swim' as the user moves.
• Mixed reality passthrough experiences with correct spatial scaling.

Camera calibration is the practical process of estimating intrinsic parameters. It typically involves capturing multiple images of a known calibration target (like a checkerboard) and solving an optimization problem. Key outputs include:

• Focal length (fx, fy): Measured in pixels, defining the field of view.
• Principal point (cx, cy): The image center, often not exactly at (width/2, height/2).
• Distortion coefficients (k1, k2, p1, p2, k3...): Model radial and tangential lens distortion. Tools like OpenCV's calibrateCamera function automate this process using Zhang's method.

In systems with multiple cameras (stereo rigs, surround-view) or fused sensors (Visual-Inertial Odometry), each camera must be individually calibrated. Consistent intrinsics are required to:

• Compute accurate disparity maps for stereo depth perception.
• Fuse visual data with Inertial Measurement Unit (IMU) readings, where scale is ambiguous from vision alone.
• Perform extrinsic calibration to find the precise rigid transformation between sensors.
• Enable 360-degree perception in automotive or robotics applications.

Modern neural rendering techniques like Neural Radiance Fields (NeRF) rely heavily on accurate camera intrinsics (and extrinsics). During training, a NeRF uses these parameters to cast rays from each training image into the scene. Incorrect intrinsics lead to blurry reconstructions and 'floaters' in the synthesized views. This makes calibration critical for:

• Generating high-fidelity 3D assets from photo collections.
• Creating digital twins for spatial computing.
• Novel view synthesis in film production and virtual production.

The foundational mathematical model for image formation, upon which camera intrinsics are defined. It describes the projection of 3D world points onto a 2D image plane through a single point (the pinhole or optical center).

• Key Equation: x = K[R|t]X, where K is the intrinsic matrix.
• Intrinsic Matrix (K): Encodes focal length (f_x, f_y), principal point (c_x, c_y), and skew. It transforms normalized camera coordinates into pixel coordinates.
• Simplification: Assumes an ideal, distortion-free lens, which real-world calibration must correct.

The parameters that define the position and orientation of a camera in a world coordinate system, as opposed to its internal properties (intrinsics).

• Representation: A 3x3 rotation matrix (R) and a 3x1 translation vector (t).
• Combined with Intrinsics: The full camera projection matrix is P = K[R|t]. Intrinsics (K) handle the camera's internal geometry, while extrinsics ([R|t]) handle its pose in the world.
• Critical for: 3D reconstruction, augmented reality overlays, and robot localization.

A set of parameters within the camera intrinsics that model deviations from the ideal pinhole model caused by physical lens imperfections.

• Radial Distortion: Causes straight lines to appear curved (barrel or pincushion distortion). Modeled with coefficients k1, k2, k3, etc.
• Tangential Distortion: Caused by lens misalignment relative to the image sensor. Modeled with coefficients p1, p2.
• Calibration Process: These coefficients are estimated by observing a known calibration pattern (like a checkerboard) and minimizing reprojection error. They are used to undistort raw images before further geometric processing.

The experimental process of estimating a camera's intrinsic parameters (focal length, principal point, distortion) and often its extrinsic parameters relative to a calibration target.

• Standard Method: Use a planar pattern (e.g., a checkerboard) with known dimensions. Multiple images are captured from different angles.
• Output: The precise intrinsic matrix (K) and distortion coefficients.
• Tools: Implemented in libraries like OpenCV (cv2.calibrateCamera) and MATLAB's Camera Calibrator. Accurate calibration is a prerequisite for metric 3D computer vision and augmented reality.

The primary metric for evaluating the accuracy of estimated camera parameters (both intrinsics and extrinsics) and 3D point locations.

• Definition: The Euclidean distance in pixels between an observed 2D image point and the 2D projection of its corresponding estimated 3D point.
• Mathematical Role: Serves as the cost function minimized during bundle adjustment and camera calibration.
• Interpretation: A low mean reprojection error (e.g., < 0.5 pixels) indicates a well-calibrated camera and a consistent 3D reconstruction. It is the gold standard for quantifying geometric accuracy.

A non-linear optimization technique that jointly refines 3D scene structure (point cloud), camera intrinsics, and camera poses (extrinsics) to minimize the total reprojection error across all images.

• Dependence on Intrinsics: While often holding intrinsics fixed, modern Structure from Motion (SfM) systems can refine them during bundle adjustment, especially when using multiple cameras or zoom lenses.
• Scale: It is the final, crucial step in pipelines like COLMAP and OpenMVG that produces globally consistent, high-accuracy reconstructions.
• Algorithm: Typically uses the Levenberg-Marquardt algorithm to solve this large-scale sparse least-squares problem.