Sim-to-Real Gap

Simulations rely on approximate physics models (e.g., rigid body dynamics with simplified contact models) that cannot perfectly capture the non-linear, stochastic behavior of real-world materials and interactions. Key mismatches include:

• Contact mechanics: Friction, deformation, and restitution are often modeled with coarse approximations.
• Actuator dynamics: Real motors have saturation, backlash, and non-linear torque-speed curves.
• Fluid and soft-body dynamics: Extremely computationally expensive to simulate with high fidelity. These inaccuracies mean a policy optimized for simulated physics may fail when encountering the true physical response.

The visual and sensory input a robot receives in simulation differs statistically from real-world sensor data. This is a classic case of distribution shift. Causes include:

• Renderer limitations: Synthetic images lack the noise, motion blur, and complex lighting (global illumination, caustics) of real cameras.
• Sensor simulation: Modeling proprioceptive sensors (joint encoders, force-torque sensors) and exteroceptive sensors (LIDAR, depth cameras) involves simplifying noise models and failure modes.
• Texture and appearance: Simulated textures and object models are often less varied and detailed than their real counterparts. A policy trained on pristine synthetic pixels may be confused by real-world visual noise.

Simulations often provide agents with perfect, full-state information (e.g., exact object positions and velocities), while real robots must estimate state from noisy, delayed, and incomplete sensor streams. This introduces several challenges:

• State estimation error: Real odometry drifts and filter latency create a difference between the true state and the believed state.
• System latency: From sensor readout to actuator command, real systems have non-deterministic delays not perfectly modeled in lock-step simulations.
• Hidden variables: Simulation may omit variables like internal battery voltage, motor temperature, or uneven floor surfaces that affect real performance. Policies relying on perfect state access can fail when deployed on systems with inherent uncertainty.

The reward function or success criteria in simulation are often clean mathematical abstractions that do not capture the full complexity of the real-world task. Discrepancies arise from:

• Reward hacking: Policies exploit simulation quirks to maximize reward without solving the intended physical task.
• Unmodeled constraints: Real tasks have safety constraints, human interaction protocols, and implicit objectives (e.g., 'move gracefully') that are absent in simulation.
• Sparse reward signals: Dense reward shaping in simulation can create a deceptive gradient that disappears in reality, where feedback is sparser. A policy may achieve 100% simulated success but fail on the real task because the simulation's goal was an incomplete proxy.

Simulations are discrete-time, deterministic approximations of a continuous-time, stochastic world. This fundamental abstraction layer creates gaps:

• Fixed time-stepping: Physics engines update at discrete intervals, aliasing continuous events. Real physics is continuous.
• Determinism: Simulations are often perfectly repeatable. Reality has inherent stochasticity from sensor noise, environmental perturbations, and quantum effects.
• Floating-point precision: Numerical errors accumulate differently in simulation versus analog physical systems. Policies may overfit to the deterministic, discretized simulation loop and lack the robustness needed for the noisy continuum of reality.

Bridging the sim-to-real gap requires calibrating the simulation model to match a specific physical robot—a process known as system identification (SysID). Inevitable errors in this process directly cause the gap:

• Incomplete parameter sets: It's impractical to identify every mass, friction, and inertia parameter perfectly.
• Time-varying parameters: Real system parameters (e.g., belt tension, battery level) change over time, while simulation parameters are static.
• Unidentifiable dynamics: Some non-linear behaviors cannot be fully captured by the chosen simulation model structure. Even a well-calibrated simulation is an approximation of the robot at a single point in time under specific conditions.

A machine learning subfield focused on improving model performance on a target domain (e.g., the real world) using knowledge from a related source domain (e.g., simulation). It is the overarching framework for addressing the sim-to-real gap. Techniques include:

• Feature alignment to make simulation and real data distributions similar.
• Adversarial training to learn domain-invariant representations.
• Fine-tuning on limited real-world data.

The fundamental cause of the sim-to-real gap. It occurs when the statistical distribution of inputs (e.g., sensor readings, visual textures, physical dynamics) differs between the training environment (simulation) and the deployment environment (reality). Key types include:

• Covariate Shift: Change in input data distribution.
• Dynamics Shift: Change in how actions affect the environment.
• Label Shift: Change in output distribution. Measuring and mitigating this shift is central to successful transfer.

A primary technique for bridging the sim-to-real gap by training policies in simulations with randomized parameters. The goal is to force the policy to learn robust, invariant features rather than overfitting to simulation artifacts. Randomized elements include:

• Visual properties: Textures, lighting, colors.
• Physical dynamics: Mass, friction, motor gains.
• Sensor noise: Proprioceptive and camera sensor models. This creates a broad support set of experiences, improving zero-shot transfer performance.

The process of building or calibrating a simulation's mathematical models using observed data from the real physical system. It directly addresses simulation inaccuracy, a key component of the dynamics gap. Methods involve:

• Collecting input-output data from real robot actuators and sensors.
• Fitting parameters (e.g., inertia, damping) to the simulation's rigid-body dynamics equations.
• Using Bayesian optimization or neural network models to capture complex, non-linear behaviors. Higher-fidelity system models reduce the sim-to-real gap.

The ability of a learned control policy to maintain successful performance despite perturbations and variations not explicitly encountered during training. A robust policy is less susceptible to the sim-to-real gap. It is characterized by:

• Stability under sensor noise and actuator latency.
• Resilience to changes in object mass, friction, or appearance.
• Consistent success across a range of environmental conditions (e.g., lighting, clutter). Robustness is often engineered via domain randomization or adversarial training.

The degree to which a virtual environment replicates the visual, physical, and behavioral characteristics of the target real-world system. Fidelity exists on a spectrum:

• Low-Fidelity: Fast, abstract simulations for rapid prototyping and broad randomization.
• High-Fidelity: Physically accurate, photo-realistic simulations that closely match reality but are computationally expensive. The choice involves a trade-off between real-time factor (RTF) and the magnitude of the sim-to-real gap. High fidelity reduces the reality gap but increases computational cost.