Simulation Fidelity

These metrics assess the perceptual realism of rendered scenes and objects. High visual fidelity is critical for training vision-based policies.

• Fréchet Inception Distance (FID): Calculates the statistical distance between feature distributions of real and simulated images. A lower FID score indicates higher visual similarity.
• Learned Perceptual Image Patch Similarity (LPIPS): Uses a deep neural network to measure perceptual similarity, correlating better with human judgment than pixel-wise metrics like PSNR.
• Structural Similarity Index (SSIM): Measures the perceived change in structural information, luminance, and contrast between image pairs.
• Domain Classifier Accuracy: A low accuracy score from a classifier trained to distinguish real from synthetic images indicates the simulation has successfully blurred the domain boundary.

This measures how accurately the simulation models physical laws, contact forces, and actuator dynamics—the core of robotic training.

• System Identification Error: The mean squared error between predicted and observed state transitions (e.g., position, velocity) when the same control inputs are applied to the simulation and the real robot.
• Contact Model Accuracy: Evaluates the correctness of collision detection, friction coefficients, and restitution forces by comparing object interaction trajectories.
• Actuator Delay & Saturation Modeling: Quantifies how well the simulation replicates real motor response times, torque limits, and gearbox backlash.
• Real-Time Factor (RTF) Deviation: While a high RTF is desirable for training, a significant deviation from 1.0 during hardware-in-the-loop testing can indicate simplified physics that fail to capture real-time computational constraints.

The ultimate test: does a policy trained in simulation achieve similar performance on the real task? This is measured through controlled real-world episodes.

• Success Rate Delta: The difference in success rate between simulation evaluation and physical deployment. A small delta indicates high behavioral fidelity.
• Cumulative Reward Correlation: The correlation coefficient between the cumulative reward earned per episode in sim versus real life.
• Trajectory Similarity: Metrics like Dynamic Time Warping (DTW) or Frechet distance applied to state-action trajectories to measure execution consistency.
• Out-of-Distribution (OOD) Robustness: The policy's maintained performance under conditions not explicitly modeled in simulation (e.g., novel object textures, lighting changes) indicates the simulation taught robust, generalizable skills.

Assesses the accuracy of simulated sensor readings, which are the policy's sole input. This includes proprioceptive and exteroceptive sensors.

• Noise Model Accuracy: Comparison of the statistical distribution (mean, variance, spectral density) of simulated sensor noise (e.g., Gaussian, quantization) to noise profiles captured from real hardware.
• Latency Modeling: Measurement of the simulated sensor update rate and processing delay against real sensor data streams.
• Depth & LiDAR Error: For depth sensors and LiDAR, metrics like Mean Absolute Error (MAE) in measured distances to surfaces across a calibrated scene.
• Camera Intrinsics/Extrinsics Error: The error in reproducing the real camera's focal length, distortion parameters, and mounting position/orientation.

Ablation studies are a critical methodology for isolating the impact of specific fidelity improvements on final transfer performance.

• Component Isolation: Systematically enable/disable high-fidelity modules (e.g., accurate motor model, complex contact solver, visual post-processing) and measure the change in zero-shot transfer success rate.
• Parameter Sensitivity Sweep: Vary key simulation parameters (e.g., gravity, friction, mass) around their identified real values and observe the resulting policy performance distribution. A robust policy shows low sensitivity, indicating the simulation taught general physics understanding.
• Compute vs. Fidelity Trade-off: Plot the improvement in a transfer metric (e.g., Normalized Score) against the increased computational cost of a higher-fidelity simulation component. This defines the Pareto frontier for simulation engineering.

In practice, perfect fidelity is neither achievable nor always optimal. The goal is to find the optimal point on the trade-off curve between simulation accuracy and training efficiency.

• Diminishing Returns: Increasing physics solver iterations or rendering quality yields smaller gains in transfer performance after a certain point, while Real-Time Factor (RTF) drops significantly.
• Domain Randomization as a Compensator: High domain randomization can often compensate for moderate fidelity shortcomings by forcing the policy to be robust to a wider distribution of environments, including the real world.
• Task-Dependent Requirements: A manipulation task requiring precise peg-in-hole insertion demands higher contact dynamics fidelity than a navigation task where coarse visual cues may suffice.
• Measurement-Driven Iteration: A rigorous sim-to-real pipeline uses these metrics in a cycle: measure fidelity gaps, improve the simulation model, retrain the policy, and re-evaluate real-world performance.

System identification is the process of building or calibrating a mathematical model of a physical system (like a robot's dynamics) from observed input-output data. It is the foundational engineering step for achieving high simulation fidelity.

• Purpose: To minimize the reality gap by making the simulation's physics engine match the real robot's behavior.
• Process: Involves collecting data from real hardware (e.g., motor torques, joint positions) and using optimization to tune simulation parameters like mass, friction, and motor constants.
• Outcome: A calibrated simulator that serves as a high-fidelity digital twin for training and testing.

Domain randomization is a sim-to-real transfer technique that intentionally varies simulation parameters during training to force a policy to learn robust, domain-invariant strategies.

• Core Idea: By training across a wide distribution of simulated conditions (e.g., object masses, surface friction, lighting, textures), the policy becomes less sensitive to the specific inaccuracies of any single simulation setup.
• Contrast with Fidelity: It is often used as a complement or alternative to pursuing perfect fidelity. Instead of making the simulation perfectly match one reality, it exposes the policy to many possible 'realities'.
• Key Parameters: Includes visual properties (domain randomization) and physical dynamics (dynamics randomization).

The sim-to-real gap, or reality gap, is the performance degradation observed when a policy trained in simulation is deployed on physical hardware. It is the direct consequence of insufficient simulation fidelity.

• Primary Causes: Modeling errors in physics (e.g., contact forces, actuator dynamics), sensor discrepancies (noise, latency), and visual differences (lighting, textures).
• Quantification: Measured as the drop in key performance metrics like success rate or cumulative reward between simulation evaluation and real-world testing.
• Bridging the Gap: Addressed through system identification, domain randomization, and domain adaptation techniques.

The Real-Time Factor (RTF) is a critical performance metric for simulation infrastructure, defined as the ratio of simulated time to wall-clock time.

• RTF > 1.0: The simulation runs faster than real-time. This is essential for accelerated training, where years of simulated experience can be generated in hours.
• RTF < 1.0: The simulation runs slower than real-time, which is acceptable for precise analysis but prohibitive for large-scale reinforcement learning.
• Trade-off with Fidelity: Higher physical and visual fidelity often reduces RTF. Engineering efforts focus on optimizing simulation engines to maintain high RTF without sacrificing necessary accuracy.

A digital twin is a high-fidelity, continuously updated virtual model of a specific physical asset, process, or system. In robotics, it represents the pinnacle of simulation fidelity for a single instance.

• Beyond Generic Simulation: While a standard simulator models a class of robots, a digital twin is meticulously calibrated to mirror a specific robot, incorporating its unique wear, calibration errors, and environmental context.
• Applications: Used for predictive maintenance, hardware-in-the-loop testing, what-if scenario analysis, and as a ultra-high-fidelity training environment for that specific unit.
• Data Flow: Continuously synchronized with real-world sensor data, allowing the twin to evolve alongside the physical system.

Out-of-distribution (OOD) generalization is the ability of a learned policy to perform accurately under conditions that differ significantly from its training data distribution. It is the ultimate test of robustness enabled by high-fidelity or strategically randomized simulation.

• Relation to Fidelity: A perfectly faithful simulator that only replicates one specific real-world condition may produce policies that fail under OOD shifts (e.g., a different floor texture).
• Evaluation: Testing a sim-trained policy in the real world is, by definition, an OOD generalization test due to the inherent distribution shift.
• Promoting OOD Robustness: Achieved through techniques like domain randomization, invariant risk minimization (IRM), and distributionally robust optimization (DRO) during simulation-based training.