Calibration Error

Calibration error quantifies the gap that remains after a calibration process like Parameter Estimation or Bayesian Calibration. It is not the initial Reality Gap, but the irreducible error left over. This residual stems from:

• Unmodeled Dynamics not captured by the simulation's structure.
• Simulation Bias from numerical approximations and solver inaccuracies.
• Sensor Noise and measurement uncertainty in the real-world data.
• Fundamental limitations in the chosen model's complexity (e.g., assuming rigid bodies when components flex).

Calibration error is measured using specific Fidelity Metrics after performing Ground Truth Alignment. Common metrics include:

• Mean Squared Error (MSE): Average squared difference between simulated and real trajectories.
• Maximum Absolute Error: Captures worst-case deviations, critical for safety.
• Spectral Analysis: Compares frequency content of signals to identify mismatched dynamics.
• Task-Specific Error: For robotics, this could be final positioning error or force/torque discrepancies during contact. The choice of metric directly informs the Identification Protocol and the perceived success of calibration.

It is crucial to distinguish calibration error from Model Uncertainty. Calibration error is an observed, quantitative mismatch after a best-effort fit. Model Uncertainty is a probabilistic measure of what is not known about the system, often expressed as a confidence interval or distribution. In Bayesian Calibration, the posterior distribution of parameters encapsulates this uncertainty, while the error between the mean prediction and data is the calibration error. High uncertainty can lead to high potential error.

When calibration error is systematic and non-random, it indicates structured Unmodeled Dynamics. This often triggers Residual Modeling, a Grey-Box Identification technique. Here, a secondary, data-driven model (e.g., a neural network) is trained to predict the calibration error itself. This error model is then added to the physics simulator to compensate for its shortcomings, effectively reducing the observed Transfer Error for policies trained in the augmented simulation.

A high calibration error typically sends engineers back to earlier stages in the System ID Pipeline. It prompts re-evaluation of:

• Experiment Design: Were the Excitation Trajectories sufficiently rich for Persistent Excitation?
• Model Structure: Does the physics model need more degrees of freedom or different contact models?
• Data Quality: Is sensor data properly filtered and synchronized?
• Parameter Space: Are the right Physics Parameters being calibrated? Thus, calibration error is a key feedback signal for improving simulation fidelity.

Ultimately, the magnitude and character of calibration error are primary predictors of Sim-to-Real Transfer performance. A policy trained in a simulator with low calibration error across a wide range of states and actions will experience a smaller Reality Gap. This makes minimizing calibration error a central goal of Simulation Fidelity work. It is a more direct measure than Model Fidelity alone, as it evaluates the model after it has been tuned to the specific real hardware.

System identification is the overarching process of constructing mathematical models of dynamic systems from measured input-output data. It encompasses the entire workflow from experimental design to model validation.

• Core Goal: To characterize a system's behavior and estimate its unknown physics parameters.
• Key Methods: Includes grey-box identification (combining physics and data) and the use of excitation trajectories to ensure persistent excitation.
• Output: A calibrated model where calibration error is the residual mismatch after this process.

Parameter calibration is the specific optimization step within system identification where a simulation's numerical constants are adjusted.

• Targets: Physics parameters like mass, friction coefficients, motor constants, and damping.
• Process: Uses optimization algorithms to minimize a loss function (e.g., trajectory error) between simulation predictions and real-world sensor data.
• Relationship to Calibration Error: The outcome of calibration is a set of parameter values; the calibration error quantifies how much discrepancy remains even after this optimal adjustment.

Model fidelity is the holistic measure of how truthfully a simulation replicates a real system's behavior. Calibration error is a direct, quantitative contributor to low fidelity.

• Broader Scope: Encompasses visual, kinematic, dynamic, and semantic accuracy.
• Sources of Infidelity: Includes simulation bias from numerical approximations, unmodeled dynamics, and residual calibration error.
• Assessment: Measured via fidelity metrics and quantitative validation against ground truth alignment.

The reality gap is the performance drop when a simulation-trained policy fails on a real robot. A domain gap is the statistical mismatch between simulation and real data distributions.

• Primary Cause: Calibration error is a fundamental driver of both gaps, as an inaccurate dynamics model creates a divergent world for the policy.
• Manifestation: Leads to transfer error in task performance.
• Mitigation: Reducing calibration error through precise system identification is a core strategy for narrowing these gaps.

Residual modeling is a technique used when calibration error from a first-principles model is structured but difficult to eliminate via parameter tuning alone.

• Process: A secondary, data-driven model (e.g., a neural network) is trained to predict the discrepancy (calibration error) between the physics simulator and real-world observations.
• Application: This residual model is then added to the base simulator's outputs to correct its predictions, effectively reducing the apparent calibration error for downstream tasks like policy training.

Quantitative validation is the rigorous process of assessing simulation accuracy, for which calibration error is a key metric.

• Procedure: Involves ground truth alignment to synchronize simulation and real data streams for direct comparison.
• Metrics: Uses fidelity metrics like Mean Squared Error (MSE) on states, forces, or power consumption to quantify calibration error.
• Purpose: Provides an objective measure to gauge the success of parameter calibration and the overall system identification pipeline.