Sensor Calibration

Intrinsic parameters define the internal geometry and optical characteristics of a sensor, independent of its placement in the world.

• For Cameras: These include the focal length, principal point (optical center), and lens distortion coefficients (radial and tangential). They are typically represented in a camera matrix (K) and distortion vector (D).
• For LiDAR/Depth Sensors: Intrinsics may include beam deflection angles, mirror rotation models, and timing offsets that define how raw time-of-flight measurements map to 3D points.
• For IMUs: This includes scale factors, biases, and non-orthogonality errors for each accelerometer and gyroscope axis.

Calibration is performed using known patterns (e.g., checkerboards) or controlled motion sequences.

Extrinsic parameters define the position and orientation (the 6-DOF pose) of one sensor relative to another or to a common reference frame, like the robot's base.

• Represented as a rigid transformation comprising a 3x3 rotation matrix (R) and a 3x1 translation vector (t).
• Critical for sensor fusion. For example, knowing the precise transform between a camera and an IMU enables tight coupling in Visual-Inertial Odometry (VIO).
• Calibration methods include hand-eye calibration (solving AX = XB) using motion data or observing a common target from multiple sensors simultaneously.
• Errors in extrinsic calibration directly cause misalignment in fused data, degrading state estimation.

Temporal calibration determines the time offset (or clock skew) between different sensor data streams, which is as critical as spatial alignment.

• Sensors sample data at different rates and with inherent processing delays. An uncorrected time lag between a camera image and an IMU reading can cause significant drift in VIO.
• The goal is to find the time offset Δt that, when applied, maximizes the consistency of the fused measurements. For a camera-IMU pair, this often involves maximizing the correlation between angular velocity from the gyroscope and the inter-frame rotation from visual features.
• Modern systems often perform spatio-temporal calibration jointly, optimizing for both the rigid transform and the time delay.

A distortion model mathematically characterizes the non-ideal projection of light through a lens (for cameras) or other systematic geometric errors in sensors.

• Radial Distortion: Causes straight lines to appear curved. Modeled with coefficients k₁, k₂, k₃, etc. (e.g., the Brown-Conrady model).
• Tangential Distortion: Arises from lens misalignment. Modeled with coefficients p₁, p₂.
• Equidistant / Fisheye Models: Use different projection functions (e.g., Kannala-Brandt) for wide-angle lenses.
• Calibration involves capturing images of a known grid pattern and solving for the coefficients that, when applied, 'undistort' the image to fit a perfect pinhole model. The choice of model impacts accuracy, especially at the image periphery.

Noise characterization involves modeling the stochastic properties of a sensor's measurements, which is essential for probabilistic estimation filters like the Kalman Filter or Particle Filter.

• This defines the measurement noise covariance matrix (R) used in state estimation. It quantifies the uncertainty and correlation of noise across sensor channels.
• For an IMU, this includes characterizing the Angular Random Walk (ARW) and Velocity Random Walk (VRW) of the gyroscope and accelerometer, respectively, as well as bias instability.
• For a camera, it can involve modeling the uncertainty of detected feature locations, often related to image resolution and blur.
• Accurate noise models allow estimation algorithms to correctly weight sensor data, trusting precise measurements more than noisy ones.

Calibration can be performed as a one-time setup (offline) or continuously during operation (online).

• Offline Calibration:
  • Performed in a controlled environment with calibration targets.
  • Yields highly accurate initial parameters.
  • Assumes parameters are static. Used for intrinsic and initial extrinsic calibration.
• Online Calibration (Self-Calibration):
  • Continuously estimates and refines parameters during normal operation.
  • Essential for correcting parameter drift due to temperature changes, mechanical shocks, or wear.
  • Often formulated as part of the SLAM back-end optimization, where calibration parameters are included as variables in the factor graph. This is key for long-term autonomy.

Calibration solves for two distinct sets of parameters. Intrinsic calibration determines the internal properties of a single sensor, such as a camera's focal length, principal point, and lens distortion coefficients. Extrinsic calibration determines the rigid transformation (rotation and translation) between the coordinate frames of two or more sensors, such as aligning a LiDAR to an IMU. Accurate mapping and localization require both.

This process estimates the parameters that map 3D world points to 2D image pixels. The core model is the pinhole camera model, defined by a calibration matrix containing focal length and principal point. Real lenses also introduce radial and tangential distortion, modeled with coefficients (e.g., k1, k2, p1, p2). Calibration is performed by capturing multiple images of a known calibration target (like a checkerboard) and solving for the parameters that minimize reprojection error using algorithms like Zhang's method.

An IMU contains accelerometers and gyroscopes that suffer from biases, scale factors, and non-orthogonality. Calibration involves:

• Deterministic errors: Misalignment between sensor axes.
• Stochastic errors: Time-varying bias and noise characteristics (modeled as Gaussian random walk). Static calibration determines scale and misalignment by collecting data at multiple known orientations. Allan variance analysis is used to characterize noise parameters for stochastic modeling in filters like the Kalman filter.

LiDAR calibration corrects systematic errors in the 3D point cloud. Intrinsic LiDAR calibration addresses:

• Beam model parameters: Vertical/horizontal angular offsets and range bias for each laser emitter.
• Time synchronization: Skew between internal clock and measurement timing. Extrinsic LiDAR calibration finds its pose relative to a robot's base frame or other sensors. This is often done by scanning a known target (e.g., a planar wall or a custom fixture with distinct corners) and aligning the observed point cloud to the expected model using algorithms like Iterative Closest Point (ICP).

A critical extrinsic calibration problem in robotic manipulation. It solves for the unknown transform between a robot's end-effector (hand) and a sensor mounted on it, typically a camera (eye). The fundamental equation is AX = XB, where A is the robot motion, B is the observed sensor motion, and X is the desired hand-eye transform. Solutions like Tsai-Lenz or Daniilidis algorithms use motions in at least two non-parallel axes to compute this transform, enabling the robot to accurately map sensor data into its own coordinate frame for grasping.

Calibration strategies differ based on when they are performed.

• Offline Calibration: Performed in a controlled setup before deployment. It uses dedicated targets and is highly accurate but assumes parameters remain static. This is standard for intrinsic and initial extrinsic parameters.
• Online (Self) Calibration: Runs concurrently with normal operation. It continuously refines parameters by observing the environment (e.g., using structure-from-motion principles or planar features). This is essential for correcting thermal drift in cameras or slow changes in LiDAR alignment, but is more complex and less precise than offline methods.

Extrinsic calibration estimates the rigid-body transformation (rotation and translation) between different sensor coordinate frames or between a sensor and the robot's base frame. This defines the pose of a LiDAR relative to a camera or an IMU relative to the robot's center. Accurate extrinsic parameters are critical for sensor fusion, ensuring measurements from different modalities are correctly aligned in a common reference frame. Techniques often involve solving a hand-eye calibration problem (AX = XB).

Hand-eye calibration solves the classic robotics problem of finding the transformation between a robot's end-effector (the 'hand') and a sensor mounted on it (the 'eye'), such as a camera or LiDAR. The core equation is AX = XB, where A is the robot's motion, B is the sensor's observed motion, and X is the unknown transformation. This is essential for tasks where the sensor moves with the robot arm, enabling accurate projection of sensor data into the world or robot base frame for manipulation.

Nonlinear optimization is the mathematical backbone of modern calibration. Calibration parameters are estimated by minimizing a cost function, typically the reprojection error (for cameras) or alignment error (for point clouds). This involves:

• Defining a parameter vector (intrinsics, extrinsics).
• Using iterative algorithms like Gauss-Newton or Levenberg-Marquardt.
• Handling outliers with robust kernels (e.g., Huber loss). This framework allows for the joint optimization of all parameters from a set of noisy observations.

Calibration methodologies are broadly categorized by their use of a known reference:

• Target-Based Calibration: Uses a physical artifact with a precisely known geometry (e.g., checkerboard, Charuco board, or AprilTag). It provides high accuracy and is standard for intrinsic and static extrinsic calibration.
• Targetless (or Self) Calibration: Estimates parameters from natural scene observations during operation, often by exploiting motion or environmental constraints. This is more flexible but can be less accurate and requires specific motion (e.g., non-degenerate trajectories) to be observable.