Performance Metric

These metrics evaluate the fundamental success of a policy in achieving its goal. They are the most direct indicators of functional performance.

• Success Rate: The percentage of trials where the agent successfully completes the defined task (e.g., placing a peg in a hole, navigating to a goal).
• Completion Time: The time or number of steps taken to finish a task, measuring efficiency.
• Success Weighted by Path Length (SPL): A composite metric for navigation that combines success with path optimality, penalizing meandering successful paths. Calculated as: (Success) * (Optimal Path Length / Actual Path Length).

These metrics originate from the reinforcement learning training process and measure the quality of behavior according to the defined reward function.

• Cumulative Reward (Return): The total sum of rewards accrued over an episode. It is the primary objective maximized during RL training.
• Normalized Score: The cumulative reward scaled relative to a baseline (e.g., a random policy or an expert demonstrator). This allows for comparison across tasks with different reward scales. A score of 1.0 often represents expert performance, 0.0 represents random.
• Regret: The difference between the cumulative reward achieved by the optimal policy and the reward achieved by the learned policy, quantifying the cost of sub-optimality.

These metrics assess how performance degrades under variation, which is central to evaluating sim-to-real transfer. They measure a system's resilience to the reality gap.

• Performance under Distribution Shift: The change in a core metric (like success rate) when evaluated on a target domain (real world) versus the source domain (simulation). A small drop indicates good transfer.
• Out-of-Distribution (OOD) Generalization Score: Performance on test conditions deliberately excluded from training (e.g., novel object textures, lighting angles, or friction coefficients).
• Variance across Trials: The standard deviation of a performance metric (e.g., success rate) over multiple random seeds or environmental initializations. Low variance indicates consistent, reliable behavior.

These metrics evaluate the practical resource consumption of training and deployment, crucial for real-world feasibility and economic viability.

• Sample Efficiency: The number of environmental interactions (simulated or real) required to achieve a target performance level. High sample efficiency reduces training time and cost.
• Real-World Episode Cost: The physical wear, time, and human supervision required to run evaluation trials on hardware. This directly impacts the speed of iterative development.
• Inference Latency: The time taken for the policy to process sensor observations and output an action. Low latency is critical for dynamic real-time control.

These metrics evaluate the tools and processes used for training, which indirectly influence final real-world performance.

• Real-Time Factor (RTF): The ratio of simulated time to wall-clock computation time. An RTF >> 1.0 enables massively accelerated training. For example, an RTF of 1000 means 1000 simulated seconds pass per 1 real-world second.
• Simulation Fidelity Error: Quantitative measures of discrepancy between simulation and reality, such as the Fréchet Inception Distance (FID) for visual domains or dynamics error for physical parameters.
• System Identification Accuracy: The error between the calibrated simulation model's predictions and recorded real-world data (e.g., mean squared error on joint torque predictions).

These are specialized metrics that combine several factors or are tailored to specific robotic sub-fields.

• Mean Average Precision (mAP): Standard in vision-based tasks (like object detection for manipulation), measuring detection accuracy across confidence thresholds.
• Smoothness / Jerk Metrics: Measures of control policy quality, quantifying sudden changes in acceleration that can stress hardware.
• Safety Violation Rate: The frequency of actions that breach predefined safety constraints (e.g., exceeding torque limits, entering a forbidden zone), critical for physical deployment.
• Human Evaluation Scores: Qualitative ratings from expert operators on aspects like task execution naturalness or perceived robustness, used when quantitative metrics are insufficient.

The Sim-to-Real Gap, also known as the reality gap, is the performance degradation observed when a machine learning policy trained in a simulated environment is deployed on a physical system. This gap arises from discrepancies between the simulation and reality, including imperfect physics modeling, sensor noise, and visual artifacts. It is the fundamental problem that sim-to-real transfer learning aims to overcome, and its magnitude is quantified by the delta in performance metrics (e.g., success rate) between simulation evaluation and real-world testing.

Domain Adaptation is a machine learning technique that improves a model's performance on a target domain (e.g., the real world) by leveraging knowledge from a related source domain (e.g., simulation). In sim-to-real, this involves adapting a policy or perception model to bridge the distribution shift. Techniques include:

• Fine-tuning with limited real-world data.
• Using domain-adversarial training to learn domain-invariant features.
• Domain randomization, which is a proactive form of adaptation performed during simulation training.

An Evaluation Protocol is a predefined, rigorous procedure for testing and scoring an algorithm's performance to ensure fair and reproducible comparisons. For sim-to-real, a robust protocol is critical and typically specifies:

• The exact initial conditions and random seeds for each trial.
• The number of real-world episodes to run for statistical significance.
• The hardware configuration and environmental conditions.
• The precise calculation of the primary and secondary performance metrics. Adherence to a shared protocol allows meaningful benchmarking across research papers and systems.

A Benchmark Suite is a standardized collection of tasks, environments, and evaluation protocols designed to systematically compare algorithms. In robotics and sim-to-real, suites like MetaWorld, RoboSuite, or Habitat provide:

• A set of canonical tasks (e.g., door opening, block stacking).
• Simulation environments with standardized APIs.
• Pre-defined performance metrics and scoring scripts.
• Leaderboards to track progress. Using a common suite accelerates research by providing a consistent framework for measuring progress against the sim-to-real gap.

Policy Robustness refers to the ability of a learned control policy to maintain stable and successful performance despite variations not encountered during training. In sim-to-real, these variations include:

• Changes in lighting or visual appearance.
• Sensor noise and calibration errors.
• Actuator dynamics like motor wear or friction changes.
• Physical perturbations (e.g., a pushed object). Robustness is measured by testing a policy under a wide range of distorted conditions and observing the variance or worst-case performance of the primary success metric.

Distribution Shift is a change in the statistical distribution of input data or environmental conditions between the training phase (simulation) and deployment phase (reality). It is the underlying cause of the sim-to-real gap. Key types include:

• Covariate Shift: Change in the distribution of input features (e.g., image textures).
• Dynamics Shift: Change in the system's transition dynamics (e.g., friction coefficients).
• Label Shift: Change in the conditional distribution of outputs. Performance metrics are the primary tool for quantifying the impact of a distribution shift on a deployed model.