Reproducibility

The cornerstone of reproducibility is determinism—the property that the same inputs always produce the same outputs. In simulation, this requires controlling all sources of randomness.

• Seeding: Explicitly setting seeds for random number generators in the simulation engine (e.g., PyBullet, MuJoCo), neural network initialization, and environment stochasticity.
• Fixed Physics: Ensuring rigid body dynamics, contact models, and numerical integrators produce identical results across runs.
• Hardware Abstraction: For real-world testing, using Real-Time Factor (RTF) control and synchronized sensor polling to minimize non-deterministic timing effects.

Every component used in an experiment must be immutably versioned and linked. This creates a complete chain of custody.

• Code: Version control (Git) with explicit commit hashes, not just branch names. Containerization (Docker) with pinned base images.
• Data: Versioned simulation parameters, System Identification datasets, and any real-world episode logs. Tools like DVC (Data Version Control) or MLflow are standard.
• Models: Storing not just final weights but the exact optimizer state, hyperparameters, and the code version used for training.
• Environment: Documenting OS, library versions (via pip freeze or conda env export), and driver versions for specialized hardware like GPUs or robot controllers.

A predefined, unambiguous testing procedure eliminates subjective interpretation of results. This is the operational definition of an Evaluation Protocol.

• Fixed Initial Conditions: Testing from a standardized set of start states, not just a single 'favorable' initialization.
• Statistical Significance: Reporting metrics like Success Rate or Normalized Score over a large number of trials (N > 10, often 100+) with confidence intervals.
• Benchmark Suite Adherence: Using community-standard tasks and metrics (e.g., Success Weighted by Path Length (SPL) for navigation, Mean Average Precision (mAP) for detection) as defined in a Benchmark Suite.
• Independent Verification: The protocol should be detailed enough for a third party to execute it without consulting the original authors.

Reproducibility fails in the gaps of unwritten knowledge. Documentation must capture the 'why' and 'how' beyond the raw code and data.

• Hyperparameter Justification: Explaining the search space and rationale for final chosen values, not just listing them.
• Simulation Configuration: Documenting all Domain Randomization ranges, Simulation Fidelity settings, and any known simplifications versus reality.
• Failure Modes: Recording common Out-of-Distribution (OOD) scenarios where the policy fails, which is critical for understanding robustness limits.
• Hardware Specifications: Detailed specs of the real robot (motor models, sensor calibrations, firmware versions) and the test environment (lighting, surface friction).

Manual processes introduce error. Reproducibility is best enforced through automation that rebuilds the experiment from versioned artifacts.

• End-to-End Scripts: Single-command scripts to recreate the virtual environment, install dependencies, train the policy, and run evaluation.
• Continuous Integration (CI): Automatically running tests on new code commits to ensure deterministic execution and that performance regressions are caught.
• Artifact Publishing: Automatically packaging and publishing trained models, results logs, and analysis notebooks with each experiment run.
• Pipeline Orchestration: Using tools like Kubeflow Pipelines or Metaflow to define, execute, and track the multi-stage workflow of data prep, training, and evaluation.

True understanding of what drives performance comes from systematic variation, which itself must be reproducible.

• Ablation Studies: Automated scripts to run controlled experiments removing key components (e.g., disabling Domain Randomization, using a perfect state estimator) to quantify their contribution.
• Cross-Validation: In simulation, using different seeds and environment splits to ensure results are not dependent on a single random partition.
• Sensitivity Analysis: Measuring how performance on a Benchmark Suite degrades with intentional perturbations to simulation parameters or policy inputs, quantifying Policy Robustness.
• Baseline Comparison: Always comparing against standard baselines (e.g., random policy, classical controller, prior SOTA) using the same exact Evaluation Protocol.

Maintain a single source of truth for all experiment artifacts using Git or similar systems. This includes:

• Model architecture definitions and hyperparameters.
• Training scripts with exact library versions (use requirements.txt or container images).
• Simulation configuration files (e.g., URDF/SDF for robots, physics engine parameters).
• Evaluation scripts and metric calculations. Tag each experiment with a unique commit hash. For complex dependencies, use containerization (Docker) to capture the complete software environment.

Automatically log every detail of an experimental run to a structured format (e.g., JSON, YAML). Critical logs include:

• Random seeds for all stochastic processes (Python, NumPy, PyTorch/TensorFlow, simulation engine).
• All hyperparameters, including those with default values.
• Training curves (loss, reward) and evaluation metrics at each checkpoint.
• Simulation parameters that were randomized via domain randomization.
• Hardware calibration data used for system identification. Store logs alongside model checkpoints in a queryable experiment tracking system (e.g., Weights & Biases, MLflow).

Eliminate non-determinism to guarantee identical simulation rollouts. Key steps:

• Set explicit random seeds at all levels: Python, deep learning framework, and physics engine (e.g., PyBullet, MuJoCo, Isaac Sim).
• Control parallel execution: Ensure the order of operations in parallelized simulation is deterministic.
• Fix simulation initial states: Standardize object spawn locations, robot poses, and environmental conditions.
• Use fixed-step simulation: Avoid variable time-step solvers that can introduce divergence. Document the exact version of the simulation engine, as updates can change physics outcomes.

Preserve immutable copies of all data and models used to produce published results.

• Training Data: Store the exact dataset (or generative procedure for synthetic data) and its hash.
• Model Checkpoints: Save the final weights and intermediate checkpoints.
• Simulation Assets: Archive 3D models, textures, and terrain meshes.
• Real-World Datasets: For adaptation, store the target domain data used for fine-tuning or system ID. Use a Digital Object Identifier (DOI) for public datasets. Clearly document the lineage from raw data to final results.

Define and document a rigid evaluation protocol to ensure consistent performance measurement.

• Number of trials: Specify the exact number of real-world episodes or simulation rollouts.
• Initial condition distribution: Define the set or range of start states for testing policy robustness.
• Metric calculation: Provide the exact code for computing success rate, normalized score, SPL, or cumulative reward.
• Hardware state: Document the physical setup, including camera positions, lighting conditions, and any wear on robot components. Publish this protocol alongside results to enable direct comparison.

A publication or report must contain sufficient detail for independent replication. Essential elements include:

• Complete system specifications: CPU/GPU, OS, software library versions.
• Physics engine details: Solver type, friction model, simulation frequency.
• Domain randomization ranges: The minimum and maximum values for every randomized parameter (dynamics, visuals).
• Results of ablation studies: Showing the impact of key components.
• Description of the real-world setup: With photos or diagrams.
• Clear statement of limitations: Addressing known sources of distribution shift or non-determinism. Link to a permanent repository containing code, data, and logs.

An evaluation protocol is a predefined, rigorous procedure for testing and scoring an algorithm's performance to ensure fair and reproducible comparisons. In sim-to-real research, this includes standardized:

• Initialization states for each trial
• Termination conditions (success, failure, timeout)
• Fixed random seeds for environment and policy stochasticity
• Number of evaluation episodes (e.g., 100 real-world trials)
• Specified hardware and sensor configurations Without a strict protocol, reported performance metrics like success rate are meaningless for comparative analysis.

A benchmark suite is a standardized collection of tasks, environments, and evaluation protocols designed to systematically compare different algorithms. For sim-to-real, suites like MetaWorld (robotic manipulation) or Habitat (embodied AI) provide:

• Diverse task definitions with clear success criteria
• Open-source simulation environments for training
• Standardized physical counterparts or specifications for real-world testing
• Leaderboards for tracking state-of-the-art performance These suites are critical for measuring progress in policy robustness and zero-shot transfer capabilities across the research community.

An ablation study is an experimental design that systematically removes or modifies components of a system to isolate and quantify their individual contributions to overall performance. In sim-to-real research, this is essential for understanding what enables transfer. Common ablations test:

• The impact of specific domain randomization parameters (e.g., friction ranges)
• The necessity of particular observation spaces or reward function terms
• The contribution of system identification or fine-tuning steps By reporting ablation results, researchers provide evidence for which design choices are necessary for reproducibility and robust out-of-distribution generalization.

Cross-validation is a statistical technique for assessing how the results of a model will generalize to an independent dataset by partitioning data into complementary subsets for training and validation. In simulation-heavy workflows, it's adapted to:

• Environment splits: Training on a set of randomized simulation environments and validating on a held-out set to estimate policy robustness.
• Hyperparameter tuning: Selecting optimal parameters (e.g., learning rates, randomization bounds) based on performance across multiple simulation seeds.
• Meta-validation: In meta-learning approaches like MAML, using a set of validation tasks to avoid overfitting to the meta-training distribution. This practice guards against over-optimistic performance reports and improves the likelihood of successful real-world transfer.

System identification is the process of building or calibrating a mathematical model (e.g., a simulation's dynamics engine) from observed input-output data collected from a real physical system. It is a key methodology for closing the sim-to-real gap and achieving reproducible physics. The process involves:

• Collecting state-action trajectories from the real robot
• Optimizing simulation parameters (e.g., motor gains, link masses, friction coefficients) to minimize the difference between simulated and real trajectories
• Using the calibrated high-fidelity simulation for policy training or verification Accurate system ID reduces distribution shift and is a prerequisite for reproducible training outcomes that transfer consistently.

A performance metric is a quantitative or qualitative measure used to assess the effectiveness, efficiency, or robustness of a system. Reproducible sim-to-real research requires clear, unambiguous metrics. Common examples include:

• Success Rate: The percentage of trials where a task is completed.
• Cumulative Reward: The total reward accrued in an episode, standard in reinforcement learning.
• Success weighted by Path Length (SPL): For navigation, penalizes success by excess path length.
• Normalized Score: Performance scaled between a random and expert baseline.
• Mean Average Precision (mAP): For perception tasks like object detection. Defining and consistently reporting these metrics is fundamental to reproducibility and meaningful comparison across studies.