Camera Intrinsics

The focal length defines the distance between the camera's optical center and its image plane, measured in pixels. It determines the camera's field of view and magnification.

• fx and fy: Represent the focal length in the x and y directions. They are often equal for square pixels but can differ due to sensor scaling.
• Field of View: A longer focal length (higher fx/fy) results in a narrower field of view (zoom), while a shorter focal length provides a wider angle.
• Projection Role: In the pinhole camera model, focal length scales 3D coordinates (X, Y) into the image plane: u = fx * (X/Z) + cx.

The principal point is the pixel coordinates of the point where the optical axis intersects the image plane. It represents the image center in the camera's model.

• cx and cy: The column and row of the principal point, typically near the center of the image sensor (e.g., (width/2, height/2)).
• Optical Center: It accounts for any misalignment between the lens's optical axis and the sensor's geometric center.
• Projection Offset: In the projection equation, (cx, cy) acts as an offset: u = fx * (X/Z) + cx. It is crucial for accurate 3D-to-2D mapping.

Distortion coefficients model the deviation from the ideal pinhole projection caused by lens geometry, typically radial and tangential distortion.

• Radial Distortion (k1, k2, k3, ...): Causes straight lines to appear curved, either barrel (lines bow outwards) or pincushion (lines bow inwards). Corrected using a polynomial function of the radial distance from the principal point.
• Tangential Distortion (p1, p2): Arises from lens misalignment relative to the sensor, causing the image to appear skewed. Corrected using a model based on decentering.
• Undistortion: The process of applying the inverse of these distortion models to produce a rectified image suitable for geometric computer vision tasks.

The skew coefficient accounts for the non-orthogonality between the image sensor's x and y axes, representing a shear in the pixel grid.

• Cause: Rare in modern digital cameras with rectangular pixels but can occur if the sensor is mounted at a slight angle.
• Mathematical Role: In the camera matrix K, the skew s modifies the projection: u = fx * (X/Z) + s * (Y/Z) + cx. It is often assumed to be zero (s = 0).
• Impact: A non-zero skew factor must be calibrated for maximum geometric accuracy, especially in high-precision metrology or with certain industrial cameras.

The camera matrix or calibration matrix K is a 3x3 matrix that compactly represents the core intrinsic parameters for perspective projection.

• Structure: K = [[fx, s, cx], [0, fy, cy], [0, 0, 1]].
• Function: It transforms normalized 3D camera coordinates (in meters) into homogeneous pixel coordinates via the equation: [u, v, 1]^T = K * [Xc/Zc, Yc/Zc, 1]^T.
• Estimation: Obtained through a process called camera calibration, using images of a known calibration pattern (like a checkerboard) and solving using algorithms like Zhang's method or bundle adjustment.

While not part of the standard intrinsic parameter set, the physical sensor resolution and pixel size are foundational for converting between pixel units and real-world metric units.

• Pixel Size (dx, dy): The physical width and height of a single pixel on the sensor, typically in micrometers (µm). Focal length in pixels (fx) is related to focal length in millimeters (f_mm) by: fx = f_mm / dx.
• Resolution (width, height): The total number of pixels (e.g., 1920x1080). Defines the image bounds and validates that the principal point (cx, cy) lies within them.
• Aspect Ratio: Given by (width * dx) / (height * dy). Understanding these physical properties is critical for accurate simulation of camera sensors in synthetic data generation.

Camera extrinsics define the position and orientation (pose) of a camera relative to a world or robot coordinate system. Represented by a 3x3 rotation matrix (R) and a 3x1 translation vector (t), they form a 4x4 transformation matrix. Extrinsics answer "Where is the camera?" while intrinsics answer "How does the camera see?"

• Purpose: Projects 3D world points into the camera's 3D coordinate frame before intrinsic projection.
• Calibration: Determined via processes like hand-eye calibration or bundle adjustment using known 3D points.
• Relation to Intrinsics: The full camera projection combines both: Pixel Coordinates = Intrinsics × Extrinsics × World Coordinates.

Lens distortion models are mathematical functions that correct for optical aberrations not captured by the ideal pinhole model, a critical part of camera intrinsics. They rectify curved lines in raw images.

• Radial Distortion: Causes straight lines to appear curved, strongest at image edges. Modeled by coefficients k1, k2, k3, ....
• Tangential Distortion: Arises from lens misalignment with the sensor plane. Modeled by coefficients p1, p2.
• Brown-Conrady Model: A widely used model combining radial and tangential terms.
• Fisheye & Omnidirectional Models: Use different projection functions (e.g., Kannala-Brandt) for ultra-wide field-of-view lenses.

Sensor calibration is the experimental process of determining a sensor's accurate intrinsic and extrinsic parameters. For cameras, this involves capturing images of a known calibration target.

• Calibration Target: Typically a checkerboard or ChArUco board with precisely known dimensions.
• Process: Algorithms (e.g., Zhang's method) solve for parameters minimizing reprojection error between detected and predicted image points.
• Multi-Sensor Calibration: Extends to finding the spatial transform (extrinsic calibration) between different sensors (e.g., camera to LiDAR, camera to IMU).
• Simulation Use: Calibrated real-world parameters are injected into simulation to match virtual sensor behavior.

The pinhole camera model is the foundational, idealized geometric model for most camera intrinsic definitions. It describes how 3D points in front of the camera are projected onto a 2D image plane through a single point (the pinhole/optical center).

• Core Assumption: Rectilinear projection; straight lines in the world remain straight in the image.
• Projection Equation: [u, v, 1]^T = K * [X_c, Y_c, Z_c]^T, where K is the intrinsic matrix.
• Limitations: Does not model lens distortion, focus blur, or rolling shutter effects.
• Basis: All real camera intrinsics are defined as deviations or augmentations to this base model.

Homogeneous coordinates are a projective coordinate system used extensively in computer vision to represent points and transformations linearly. They are essential for expressing camera projection equations.

• Representation: A 2D point (x, y) becomes (x, y, 1); a 3D point (X, Y, Z) becomes (X, Y, Z, 1).
• Advantage: Allows translation, rotation, and projection to be represented as a single matrix multiplication.
• Camera Projection: The intrinsic matrix K operates on 3D points in camera coordinates in homogeneous form to produce 2D image coordinates in homogeneous form.
• Normalization: The final pixel coordinate is obtained by dividing by the third (homogeneous) component.

The render pipeline is the sequence of stages in a graphics engine that synthesizes a 2D image from 3D scene data. Camera intrinsics directly control the projection stage of this pipeline.

• Stages: Include vertex processing, projection, clipping, rasterization, and shading.
• Projection Matrix: In OpenGL/DirectX, a combined matrix (often using a different coordinate convention) encodes intrinsics (field of view, aspect ratio) to map 3D vertices to 2D normalized device coordinates.
• Simulation Fidelity: For photorealistic sensor simulation, the pipeline must also model lens effects (distortion, vignetting), sensor noise, and shutter effects (rolling/global).
• Output: The final rasterized image serves as synthetic camera data for training perception models.