Fidelity Metric

A core characteristic of a fidelity metric is its quantitative nature, providing a numerical score rather than a qualitative assessment. This objectivity allows for:

• Direct comparison between different simulation models or parameter sets.
• Tracking improvements over time during the system identification and parameter calibration process.
• Setting clear, measurable thresholds for model acceptance (e.g., "mean squared error must be < 0.01"). Examples include Mean Squared Error (MSE), Root Mean Square Error (RMSE), and Pearson correlation coefficient.

Fidelity metrics are categorized by their scope. General-purpose metrics evaluate low-level physical agreement, such as comparing joint angle trajectories or contact forces. These are essential for physics parameter calibration.

Task-specific metrics evaluate high-level functional performance, such as:

• Success rate for a manipulation task.
• Energy efficiency of a locomotion policy.
• Final positioning error for a reaching motion. These metrics are critical for assessing transfer error and the ultimate utility of a sim-to-real transfer.

A valid fidelity metric requires ground truth alignment between simulation and reality. This involves:

• Temporal Synchronization: Ensuring data streams from the simulator and real sensors (e.g., motion capture) are aligned in time.
• Spatial Registration: Aligning coordinate frames between the virtual and physical worlds.
• Signal Correspondence: Comparing directly analogous signals (e.g., simulated vs. real joint torque). Without precise alignment, even a perfect model will show high calibration error, misleading the identification process.

An effective fidelity metric must be sensitive to the types of model uncertainty and unmodeled dynamics most relevant to the target application. For example:

• A metric for a walking robot should be highly sensitive to ground friction parameters.
• A metric for a drone should be sensitive to aerodynamic drag coefficients. Poorly chosen metrics may fail to penalize significant simulation bias, leading to policies that fail due to the reality gap. Sensitivity analysis is often required during metric selection.

Fidelity metrics are not just for final evaluation; they are the optimization target within the system identification pipeline. The process is:

1. Execute an excitation trajectory on the real system and in simulation.
2. Collect synchronized data streams.
3. Compute the chosen fidelity metric (e.g., MSE of end-effector position).
4. Adjust physics parameters in the simulator via Bayesian calibration or data-driven calibration to minimize the metric.
5. Validate with a separate test trajectory. This iterative minimization directly reduces the domain gap.

While a low error on a fidelity metric indicates good model fidelity, it does not guarantee successful policy transfer. However, it is a strong leading indicator. High simulation bias revealed by poor metric scores often correlates with high transfer error. Therefore, fidelity metrics are a cornerstone of sim-to-real benchmarking. They provide an efficient, safe, and repeatable way to vet simulation environments and trained policies before costly and risky physical deployment.

These metrics compare the time-series evolution of system states. They are fundamental for evaluating kinematic and dynamic accuracy.

• Mean Squared Error (MSE): The average squared difference between simulated and real trajectories for a given state variable (e.g., joint angle, end-effector position).
• Dynamic Time Warping (DTW): Measures similarity between two temporal sequences that may vary in speed, useful for comparing motion profiles where timing may differ slightly.
• Frechet Distance: A measure of similarity between curves that takes into account the location and ordering of points along the curves, often used for path comparison.

Critical for tasks involving manipulation or locomotion, these metrics assess the accuracy of simulated physical interactions.

• Ground Reaction Force Error: The difference between simulated and measured forces at contact points (e.g., foot-ground contact for a legged robot).
• Contact Timing Discrepancy: Measures errors in the timing of contact events (make/break), which can destabilize controllers.
• Impulse/Momentum Transfer: Compares the change in momentum of bodies during collisions between simulation and reality.

High-level, binary, or scalar metrics that evaluate the ultimate outcome of a policy or controller. These are often the most direct measure of transfer utility.

• Success Rate: The percentage of trials in which a simulated policy achieves the defined real-world task goal (e.g., object placed in bin, door opened).
• Reward Correlation: The correlation coefficient between the reward signal achieved in simulation and a proxy reward measurable in the real world.
• Time to Task Completion: Comparing the duration required to finish a task, where significant deviation indicates dynamic inaccuracies.

Used when the simulation includes rendering for camera-based sensors. These assess the visual realism and its impact on perception algorithms.

• Structural Similarity Index (SSIM): Measures the perceived visual quality and structural information difference between simulated and real images.
• Domain Shift Score: Quantifies the distributional difference between simulated and real image datasets using metrics like the Frechet Inception Distance (FID).
• Perceptual Loss: The error between high-level feature representations of images extracted by a pre-trained neural network (e.g., VGG).

Evaluate the efficiency and realism of actuation. Discrepancies here often point to inaccurate actuator or friction models.

• Torque/Current RMSE: Root Mean Square Error between simulated and measured motor torques or currents for a given trajectory.
• Total Energy Consumption: Comparing the integrated power usage over a task cycle.
• Control Bandwidth Fidelity: Assessing how well the simulated system replicates the real system's frequency response to control inputs.

These metrics compare not just single trajectories, but the distribution of outcomes or states, which is crucial for robust policy evaluation.

• Kullback-Leibler (KL) Divergence: Measures how one probability distribution (e.g., of end-effector positions after 100 trials) diverges from a second, reference distribution (from real-world trials).
• Wasserstein Distance: Another metric for comparing probability distributions, often more stable than KL divergence.
• Confidence Interval Overlap: Assessing whether the distribution of a simulated metric (e.g., final position error) falls within the confidence intervals observed in reality.

Model fidelity is the overarching degree to which a simulation model accurately represents the real-world system's behavior, dynamics, and outputs. It is the qualitative goal that fidelity metrics quantify. High model fidelity is achieved through accurate physics parameters, comprehensive system identification, and compensation for unmodeled dynamics.

The reality gap is the observed performance discrepancy when a policy trained in simulation fails or degrades upon transfer to a real robot. It is the practical consequence of insufficient simulation fidelity. Transfer error is the quantitative metric for this gap. Closing it requires techniques like domain randomization and residual modeling to make policies robust to inaccuracies.

System identification is the foundational process of building a mathematical model of a physical system from measured data. It directly informs and improves fidelity metrics by providing accurate parameters for the simulator. Key steps include:

• Designing excitation trajectories for persistent excitation.
• Performing parameter estimation (e.g., mass, friction).
• Validating the model via quantitative validation against ground truth.

Parameter calibration is the optimization process of tuning a simulation's physics parameters (e.g., coefficients of friction, motor constants) to minimize the calibration error between simulated and real sensor data. This is a primary method for improving fidelity metrics. Approaches include:

• Data-driven calibration using optimization.
• Bayesian calibration for probabilistic parameter distributions.
• Grey-box identification combining physics and data.

Simulation bias is a systematic error inherent to the simulator, causing its predictions to consistently deviate from reality. It arises from numerical approximations, simplified contact models, or ignored unmodeled dynamics. Fidelity metrics measure the effects of this bias. Mitigation strategies involve residual modeling to learn the bias or domain randomization to train robust policies that are insensitive to it.

Quantitative validation is the rigorous process of assessing simulation accuracy by comparing its outputs to high-fidelity real-world data. It is the application framework for fidelity metrics. The process requires:

• Ground truth alignment to synchronize simulation and real data streams.
• Calculating statistical fidelity metrics (e.g., MSE, MAE, DTW).
• Following a standardized identification protocol to ensure reproducible, comparable results.