Evaluation Protocol

The protocol must begin with an unambiguous definition of the task and the operational environment. This includes:

• Task Success Criteria: Binary or continuous conditions for task completion (e.g., object placed within 2cm of target).
• Initial State Distribution: The range of permissible starting configurations for the robot and environment.
• Environmental Parameters: Specification of lighting, surface properties, and object variations allowed during testing.

Example: For a block-insertion task, the protocol would define the pegboard geometry, block dimensions, and the allowable range of starting positions for the block in the gripper.

Objective, quantitative metrics are essential for comparison. A robust protocol employs multiple metrics to capture different aspects of performance:

• Primary Metric: Often Success Rate (percentage of successful trials over N episodes).
• Quality Metrics: Cumulative reward, task completion time, or path efficiency (e.g., SPL for navigation).
• Robustness Metrics: Performance variance across randomized conditions or the worst-case performance within a test set.

Metrics should be aggregated statistically (e.g., mean ± standard deviation over multiple random seeds and physical trials) rather than reported from single runs.

This component dictates the exact sequence and conditions for evaluation to ensure reproducibility:

• Number of Trials (N): A statistically significant number of real-world episodes (e.g., 100+ trials per policy).
• Randomization Seed Control: Seeds for environment randomization are fixed and reported to allow exact replication of test conditions.
• Evaluation Phases: May include zero-shot transfer tests, followed by adaptation phases if the protocol allows limited real-world fine-tuning.
• Hardware Reset Protocol: A defined method for resetting the robot and environment to the initial state between trials.

Performance is meaningless without comparison. A protocol mandates the evaluation of standard baseline policies:

• Random Policy: Provides a lower-bound performance floor.
• Scripted/Expert Policy: A hand-coded or demonstration-based solution that provides an upper-bound or traditional benchmark.
• Prior State-of-the-Art: The protocol should enable direct comparison with results from previously published work on the same or a similar benchmark suite.

Results are often reported as a normalized score relative to these baselines.

To evaluate policy robustness and OOD generalization, the protocol includes deliberate distribution shift tests:

• Systematic Perturbations: Testing under unseen lighting conditions, payload masses, or surface friction.
• Adversarial Scenarios: Introducing targeted noise to sensors or actuators.
• Domain Shift Quantification: Using metrics like Fréchet Distance between simulation and real-world observation distributions to correlate with performance drop.

This often involves an ablation study to see which randomization techniques during training contributed most to real-world stability.

The final component ensures the protocol's output is actionable and verifiable. It requires detailed reporting of:

• Hyperparameters & Architecture: Full model and training configuration.
• Simulation Details: Physics engine, simulation fidelity settings, and domain randomization ranges used during training.
• Compute Resources: Training time and hardware used.
• Failure Modes: Qualitative analysis of common failure cases observed during real-world testing.
• Public Artifacts: Where possible, release of policy checkpoints, environment code, and raw result logs to fulfill the principle of reproducibility.

A benchmark suite is a standardized collection of tasks, environments, and associated evaluation protocols designed for systematic, apples-to-apples comparison of algorithms. For sim-to-real research, suites like MetaWorld (for manipulation) or Habitat (for navigation) provide:

• Standardized tasks with clear success criteria.
• Pre-defined training and test splits.
• Built-in domain randomization parameters.
• Leaderboards to track state-of-the-art performance. Their primary function is to move the field forward by providing a common ground for measuring progress.

A performance metric is the quantitative measure used to score an algorithm's output within an evaluation protocol. In robotics and sim-to-real transfer, metrics must be objective, reproducible, and task-relevant. Common categories include:

• Task Success: Binary success rate over N trials.
• Efficiency Metrics: Like Success weighted by Path Length (SPL) for navigation, which penalizes circuitous routes.
• Quality Metrics: Such as final positioning error for a manipulation task.
• Robustness Metrics: Performance under increasing levels of domain shift or perturbation. The choice of metric directly dictates what properties the learning algorithm will optimize.

An ablation study is a critical experimental component of a robust evaluation protocol. It systematically removes or modifies individual components of a proposed system (e.g., a specific domain randomization parameter, a network layer, a reward shaping term) to isolate and quantify their contribution to overall performance. A well-designed ablation:

• Establishes causality, proving which components are necessary for good performance.
• Provides interpretability, showing why a method works.
• Guides future research by highlighting the most impactful innovations. Without ablation studies, it is difficult to attribute performance gains to specific algorithmic advances versus other factors.

Reproducibility is the cornerstone principle that an evaluation protocol must ensure. It means that independent researchers, using the same algorithm description, code, hyperparameters, and experimental conditions, can obtain statistically equivalent results. Key enablers include:

• Publicly releasing code and model checkpoints.
• Detailed specification of simulation parameters (e.g., physics engine version, friction coefficients).
• Seeding all random number generators.
• Documenting hardware specifications (both for simulation and real-world testing). Protocols that lack reproducibility undermine scientific progress and make reported results non-verifiable.

A real-world episode is the fundamental unit of evaluation in the final stage of a sim-to-real protocol. It refers to a single, contiguous trial—from an initial state to a terminal state (success, failure, or timeout)—executed by the policy on physical hardware. Protocol design must specify:

• The number of episodes (e.g., 50) to achieve statistical significance.
• The initialization procedure for each episode to ensure fair testing.
• How human intervention (if any) is recorded and penalized.
• The data logged (e.g., sensor streams, actions, outcomes) for post-hoc analysis. The aggregate performance across all real-world episodes is the ultimate measure of sim-to-real transfer success.

Out-of-Distribution (OOD) Generalization is the overarching capability that evaluation protocols for sim-to-real seek to measure. It is a model's ability to maintain performance when deployed in conditions that differ significantly from its training data distribution. A rigorous protocol tests OOD generalization by:

• Holding out certain physical parameters (e.g., object mass, surface friction) during training and testing on them.
• Introducing novel visual distractors or lighting conditions at test time.
• Testing on physically different robot hardware (e.g., a different gripper). Metrics should capture not just average performance, but the variance and worst-case performance across these OOD tests, which is critical for real-world robustness.