Model Fidelity

The core measure of fidelity, expressed through statistical metrics that compare simulation outputs to real-world ground truth data. This involves ground truth alignment to synchronize data streams before calculating errors.

• Common Metrics: Mean Absolute Error (MAE), Root Mean Square Error (RMSE) for continuous signals (e.g., joint angles, forces).
• Task-Specific Metrics: Success rate, completion time, or energy consumption for evaluating a policy's transferred performance, which relates directly to transfer error.
• Calibration Error: The residual discrepancy after parameter calibration, quantifying how well the tuned model fits the identification data.

Defines the operational envelope within which the simulation produces valid outputs. A high-fidelity model must accurately capture non-linear behaviors at operational extremes, not just near a nominal operating point.

• Saturation Effects: Correctly modeling actuator torque limits, velocity caps, and sensor saturation.
• Failure Modes: Reliably simulating system behavior under stress or outside normal parameters, which is critical for safety and failure mode simulation.
• Stability Boundaries: Accurately predicting the conditions under which the real system becomes unstable, which depends on correct physics parameters like inertia and friction.

The accuracy of the simulation's behavior over time, ensuring that sequences of events, transient responses, and long-term integrations are physically plausible. This is fundamental for training temporal policies via reinforcement learning for robotics.

• Latency and Phase: Matching the real system's response delays and phase shifts to inputs.
• Numerical Stability: Using integration methods (e.g., variable-step solvers) that prevent energy drift or unnatural damping in contact and rigid body dynamics.
• Real-Time Factor: For hardware-in-the-loop testing, the simulation must often run at a deterministic, wall-clock speed.

The accuracy of the simulation's representation of physical geometry, scale, and spatial relationships. This is paramount for tasks involving manipulation, navigation, and vision.

• Collision Geometry: Precision of mesh representations and collision detection algorithms, directly impacting contact and rigid body dynamics.
• Sensor Simulation: Realism of sensor and actuator simulation, including camera intrinsics/extrinsics, LiDAR raycasting, and depth sensor noise models.
• Kinematic Chain: Exact replication of link lengths, joint axes, and degrees of freedom from the CAD model of the real robot.

The degree to which the simulation's physics parameters can be uniquely and reliably determined from observable data. A model with poor identifiability is difficult to calibrate, leading to high model uncertainty.

• Persistent Excitation: Requires designing excitation trajectories that stimulate all dynamic modes.
• Dynamic Regressor: The mathematical formulation must be well-conditioned to allow for accurate parameter estimation.
• Observability: The system's internal states (and thus the parameters influencing them) must be inferable from outputs, a key concern in state estimation and system identification.

The ability of a calibrated model to make accurate predictions for inputs and conditions not present in the calibration dataset. This tests for overfitting and the presence of unmodeled dynamics.

• Cross-Validation: Evaluating the model on a separate test trajectory not used during data-driven calibration.
• Extrapolation Risk: Assessing performance degradation when the robot operates outside the range of motions used for system identification.
• Robustness to Disturbances: Accurate prediction of system response to unforeseen external forces or perturbations, indicating a deep understanding of the underlying forward dynamics and inverse dynamics.

System identification is the process of constructing mathematical models of dynamic systems from measured input-output data to characterize their behavior and unknown parameters. It is the foundational engineering discipline for calibrating a simulation.

• Core Methods: Includes frequency-domain analysis, time-series fitting, and optimization-based parameter estimation.
• Application: Used to determine unknown physics parameters like inertia, friction coefficients, and motor constants from real robot data.

The reality gap is the performance discrepancy between a policy or model trained in simulation and its performance when deployed on the corresponding real-world physical system. It is the primary problem that high model fidelity seeks to minimize.

• Causes: Results from simulation bias, unmodeled dynamics, and domain gaps in visuals and physics.
• Mitigation: Addressed through techniques like domain randomization, system identification, and residual modeling.

A domain gap is the statistical difference between the data distribution of a source environment (e.g., simulation) and a target environment (e.g., reality), which can degrade model performance during transfer. It is a specific manifestation of the reality gap.

• Types: Includes visual domain gaps (textures, lighting) and dynamics domain gaps (friction, latency).
• Bridging: Closed via domain adaptation algorithms and by increasing simulation fidelity through parameter calibration.

Parameter calibration is the process of adjusting the numerical values of a simulation model's parameters to minimize the discrepancy between its predictions and observed real-world data. It is a direct method for improving model fidelity.

• Process: Involves collecting real-world excitation trajectories, then optimizing parameters to match sensor readings.
• Approaches: Ranges from manual tuning to automated data-driven calibration and Bayesian calibration.

Simulation bias is a systematic error introduced by the approximations, assumptions, and numerical methods inherent in a simulator, causing its predictions to consistently deviate from real-world behavior. It is a key component of model uncertainty.

• Sources: Includes simplified contact models, fixed integration time-steps, and ignored actuator dynamics.
• Impact: Directly contributes to transfer error and must be quantified during quantitative validation.

Quantitative validation is the process of assessing simulation fidelity by comparing numerical outputs from the simulator against corresponding high-fidelity real-world data using statistical metrics. It provides the fidelity metrics needed to gauge calibration error.

• Procedure: Requires careful ground truth alignment of simulation and real data streams.
• Metrics: Common metrics include Mean Squared Error (MSE) for trajectories, force/torque error, and spectral analysis.