Success Rate

Success rate is fundamentally a binary metric. A trial is scored as a success (1) or failure (0) based on a clear, predefined terminal condition. This condition must be objective and measurable, such as:

• An object being placed within a tolerance zone.
• A robot reaching a target pose.
• A task being completed within a time limit. The strictness of this definition directly impacts the reported metric, making precise task specification critical for meaningful comparison.

As a proportion, success rate's confidence depends heavily on the number of independent trials (N). A reported rate of 80% from 5 trials carries high uncertainty, whereas the same rate from 500 trials is statistically robust. Standard practice involves:

• Running a large number of episodes (often 100-1000) to compute a reliable mean and standard error.
• Ensuring trials are statistically independent (different random seeds, environment resets).
• Reporting confidence intervals (e.g., 95% CI) to communicate uncertainty, which is essential when comparing policies or reporting sim-to-real transfer performance.

Success rate measures completion of a single, specific task and does not capture the quality or efficiency of the execution. Two policies with identical success rates can differ drastically in:

• Path efficiency (excess movement, energy use).
• Execution speed (time to completion).
• Motion smoothness or safety.
• Generalization to minor task variations. Therefore, success rate is almost always reported alongside secondary metrics like cumulative reward, path length, or completion time to provide a complete performance profile.

In sim-to-real research, the relative drop in success rate from simulation to physical hardware is a direct, quantitative measure of the reality gap. It answers the core question: "How well does simulation training transfer?"

• A high sim success rate with a low real-world rate indicates a large domain shift or poor simulation fidelity.
• A minimal drop indicates effective domain randomization or adaptation techniques. This makes it the headline metric for most sim-to-real benchmark papers, as it directly correlates to engineering viability.

While intuitive, success rate has key limitations that necessitate complementary metrics:

• No partial credit: It ignores near-successes or graceful degradation.
• Sensitivity to thresholding: Slight changes in success criteria can cause large metric swings.
• No insight into failure modes: It doesn't explain why failures occur. Hence, it is used in conjunction with metrics like:
• Success weighted by Path Length (SPL): For navigation, penalizes inefficient successful paths.
• Mean Average Precision (mAP): For detection/segmentation tasks within a pipeline.
• Robustness scores: Measuring performance under distribution shift or adversarial conditions.

For success rate to be a valid tool for comparison across research, it must be computed under a standardized evaluation protocol. This includes:

• A fixed, public benchmark suite of tasks (e.g., MetaWorld, RLBench).
• A defined number of random seeds and environment initializations.
• Identical success condition definitions and evaluation code.
• Reporting of aggregate statistics (mean, std, confidence intervals) across all tasks. Without this rigor, reported success rates are not reproducible or comparable, undermining scientific progress in sim-to-real transfer learning.

A performance metric scaled relative to a baseline policy (e.g., random or expert) to enable fair comparison across tasks with different reward scales. In sim-to-real, it's used to contextualize raw success rates.

• Calculation: (Agent Score - Random Score) / (Expert Score - Random Score).
• Purpose: Mitigates the issue where a 90% success rate on a simple task may be less impressive than a 60% rate on a complex one.

A composite navigation metric that penalizes success based on the excess path length taken by an agent compared to an optimal shortest path. It balances success rate with efficiency.

• Formula: SPL = (Success) * (Optimal Path Length / Actual Path Length).
• Application: Critical for evaluating mobile robots and embodied AI, where a meandering successful path is less desirable than a direct one.

The sum of all reward signals received by an agent over an episode in reinforcement learning. While success rate is binary, cumulative reward provides a nuanced, continuous measure of performance quality.

• Insight: A policy that completes a task clumsily (low cumulative reward) may have the same success rate as one that does it elegantly (high cumulative reward), highlighting different aspects of robustness.

Measures the number of environmental interactions (samples) an algorithm requires to achieve a target performance level. It's a key economic metric for sim-to-real transfer.

• Real-World Impact: A high success rate achieved after billions of simulation samples may be impractical if the policy cannot adapt to reality with limited, costly real-world episodes.

The ability of a learned control policy to maintain stable performance despite variations in environmental conditions, sensor noise, or actuator dynamics not seen during training. It is the qualitative property that a high success rate should imply.

• Evaluation: Tested by measuring success rate across a benchmark suite of randomized conditions or through ablation studies that introduce perturbations.

The ability of a model to perform accurately on input data from a statistical distribution significantly different from its training data. A high in-simulation success rate with poor OOD generalization indicates overfitting and predicts sim-to-real transfer failure.

• Related Techniques: Domain randomization and invariant risk minimization (IRM) are explicitly designed to improve OOD generalization from simulation to reality.