Camera Calibration

Intrinsic parameters define the internal geometry and optical properties of the camera itself, independent of its position in the world. They form the camera matrix (K).

Key components include:

• Focal Length (fx, fy): The distance between the camera's sensor and its optical center, measured in pixels. It determines the field of view and magnification.
• Principal Point (cx, cy): The optical center of the image, where the optical axis intersects the image plane. It is typically near the image center.
• Skew Coefficient (s): Accounts for non-rectangular pixels, though it is often zero for modern digital sensors.

These parameters are used to project 3D points in the camera's coordinate system onto the 2D image plane.

Lens distortion coefficients model deviations from the ideal pinhole camera model caused by optical imperfections in the lens. They are critical for correcting image warping.

Two primary types are modeled:

• Radial Distortion: Causes straight lines to appear curved. It is modeled with coefficients (k1, k2, k3, ...). Barrel distortion (k1 < 0) pinches the image inward, while pincushion distortion (k1 > 0) bulges it outward.
• Tangential Distortion: Occurs when the lens is not perfectly parallel to the image sensor, modeled with coefficients (p1, p2).

These coefficients are used in an undistortion step to rectify images before further geometric processing.

Extrinsic parameters define the camera's position and orientation (its pose) in a fixed world coordinate system. They describe a rigid transformation from world to camera coordinates.

The parameters consist of:

• Rotation Matrix (R): A 3x3 orthogonal matrix representing the camera's orientation.
• Translation Vector (t): A 3x1 vector representing the camera's position.

Together, they form a 3D transformation: [X_camera] = R * [X_world] + t. These are estimated for each image in a sequence during processes like Structure from Motion (SfM) or Visual SLAM.

The camera matrix, or calibration matrix K, is a 3x3 matrix that consolidates the primary intrinsic parameters for the pinhole camera model.

Its standard form is:

codeCopy
K = [ fx   s  cx ]
    [  0  fy  cy ]
    [  0   0   1 ]

• fx, fy: Focal lengths in pixels.
• cx, cy: Coordinates of the principal point.
• s: Skew coefficient.

This matrix is used to project a 3D point [X, Y, Z] in camera coordinates to a 2D pixel [u, v] via the equation: [u, v, 1]^T = K * [X/Z, Y/Z, 1]^T. It is a foundational component in multi-view geometry and bundle adjustment.

Reprojection error is the primary metric used to evaluate and optimize calibration quality. It measures the discrepancy between an observed 2D image point and the re-projection of its estimated corresponding 3D world point.

• Calculation: For a known 3D point X and its detected 2D image point x, the reprojection error is the Euclidean distance (in pixels) between x and the projection proj(K, R, t, X).
• Role in Optimization: The sum of squared reprojection errors across all points and images is the objective function minimized during bundle adjustment. A low average reprojection error (e.g., < 0.5 pixels) indicates a well-calibrated camera.
• It directly quantifies the accuracy of the estimated intrinsic and extrinsic parameters.

The most widely used technique, popularized by Zhengyou Zhang. It uses a planar calibration target with a known checkerboard pattern.

• Process: Multiple images of the target at different orientations are captured. The algorithm detects inner corner points, estimates initial parameters via a closed-form solution, and refines them using non-linear optimization (e.g., Levenberg-Marquardt) to minimize reprojection error.
• Key Outputs: Intrinsic matrix, radial and tangential distortion coefficients (k1, k2, p1, p2), and extrinsic poses for each calibration image.
• Advantages: Simple, robust, and implemented in libraries like OpenCV (cv2.calibrateCamera).
• Limitations: Requires a physical target and careful image capture to cover the field of view.

A method that calibrates a camera from an unordered image set of a general static scene, without a known calibration target.

• Principle: Explores constraints from the absolute conic and its image, the dual image of the absolute conic (DIAC), using the consistency of projective transformations between views (the Kruppa equations). Modern implementations often use Structure from Motion (SfM) pipelines.
• Process: Features are matched across images, a projective reconstruction is computed, and metric upgrade is performed by enforcing constraints like known skew (often zero) or aspect ratio.
• Use Case: Essential for historical image analysis, consumer photogrammetry (e.g., from internet photos), and situations where a target is impractical.
• Challenge: Generally less accurate and stable than target-based methods, sensitive to degenerate motions.

Calibrates the intrinsic parameters of each camera and the extrinsic rigid transformations between them in a fixed multi-camera system (e.g., stereo pair, surround-view system).

• Stereo Calibration: A specific case for two cameras. It computes the rotation matrix (R) and translation vector (t) between the two camera coordinate systems, leading to the essential matrix (E) and fundamental matrix (F). Rectification parameters are also computed to align epipolar lines.
• Target-Based: Often uses a single large checkerboard visible to all cameras simultaneously, or a moving target.
• Targetless: Can use feature matching and bundle adjustment across overlapping views of a scene.
• Application: Critical for stereo depth estimation (stereo matching), 3D reconstruction, and visual odometry.

Techniques that continuously estimate and update camera parameters during operation, accounting for factors like thermal drift, mechanical vibration, or auto-focus changes.

• Visual-Inertial Calibration: Jointly calibrates camera intrinsics with an Inertial Measurement Unit (IMU). Uses Kalman Filters or factor graph optimization (e.g., in Visual-Inertial SLAM) to estimate time-varying parameters from motion.
• Vanishing Point Methods: Uses detected lines and vanishing points in man-made environments (with orthogonal directions) to constrain focal length and distortion.
• Photometric Calibration: Estimates camera response function and vignetting by analyzing pixel intensity variations across multiple exposures of a static scene.
• Importance: Enables long-term autonomy for robots and autonomous vehicles where factory calibration may degrade.

A computer vision technique that simultaneously estimates the 3D structure of a scene and the camera poses from a set of unordered 2D images. It is a core pipeline that depends on initial camera calibration or jointly refines it.

• Process: Detects keypoints, matches them across images, and uses triangulation to build a sparse 3D point cloud.
• Relation to Calibration: Often performs self-calibration, estimating intrinsic parameters (like focal length) alongside structure and pose, especially when no calibration target is available.
• Output: A sparse point cloud and the estimated camera trajectory.

A non-linear optimization process that jointly refines the 3D coordinates of scene points, camera poses, and intrinsic parameters to minimize the total reprojection error across all images. It is the final, crucial refinement step in SfM and Visual SLAM.

• Cost Function: Minimizes the difference between observed 2D image points and the re-projection of the estimated 3D points.
• Role of Calibration: Provides the initial, stable intrinsic parameter estimates (focal length, principal point, distortion) that the optimization fine-tunes, preventing the problem from becoming under-constrained.
• Scale: Essential for achieving metric accuracy in the final reconstruction.

A technique that generates dense 3D geometry (e.g., a dense point cloud or mesh) from multiple, calibrated images of a static scene. It operates after SfM has established camera poses.

• Prerequisite: Requires highly accurate camera intrinsics and extrinsics from calibration and SfM.
• Process: For each pixel in a reference image, it searches along epipolar lines in other images to find correspondences and compute depth via triangulation.
• Output: Creates orders of magnitude more points than SfM, enabling surface reconstruction. Poor calibration leads to misaligned depth maps and noisy geometry.

The computational problem where an agent (like a robot or AR device) simultaneously constructs a map of an unknown environment and localizes itself within it, using visual input as the primary sensor.

• Online vs. Offline: Unlike SfM, vSLAM runs in real-time as the camera moves.
• Calibration Dependency: The system's egomotion estimation (tracking) and map accuracy are directly tied to the precision of the pre-calibrated camera model. Lens distortion must be corrected to ensure feature tracking is geometrically valid.
• Representations: Often uses sparse features for tracking but can integrate dense RGB-D or TSDF volumes for mapping.

The geometric distance, measured in pixels, between a projected 3D point (using estimated camera parameters) and its corresponding measured 2D image feature. It is the fundamental metric for evaluating calibration and reconstruction quality.

• Calculation: (\text{error} = || \mathbf{x}_{\text{measured}} - \pi(\mathbf{X}, \mathbf{K}, \mathbf{R}, \mathbf{t}) ||^2), where (\pi) is the projection function.
• Use Case: Serves as the primary cost function in bundle adjustment. A low mean reprojection error indicates a well-calibrated system and a consistent 3D reconstruction.
• Diagnostic: High, systematic errors often indicate poor lens distortion modeling or incorrect intrinsic assumptions.

Defines the position and orientation (translation (\mathbf{t}) and rotation (\mathbf{R})) of a camera in a 3D world coordinate system. It represents the extrinsic parameters of the camera.

• Representation: Often a 4x4 transformation matrix or a rotation matrix and translation vector. (\mathbf{P} = [\mathbf{R} | \mathbf{t}]).
• Estimation: Found via processes like Perspective-n-Point (PnP) solving, which requires known 3D-2D correspondences and calibrated intrinsics.
• Critical For: Every task that relates image pixels to the world, including augmented reality (virtual object placement), robot navigation, and 3D reconstruction.