Reality Gap

Simulation engines use simplified mathematical models to approximate real-world physics, leading to inherent inaccuracies. Key mismatches include:

• Contact dynamics: Simplified collision detection and friction models fail to capture the complexity of real-world interactions, such as soft-body deformation or granular materials.
• Actuator dynamics: Idealized motor models ignore real-world effects like backlash, saturation, non-linear torque-speed curves, and communication latency.
• System parameters: Inaccurate calibration of real robot properties (mass, inertia, center of mass) in the simulation model.
• Numerical integration: Discretization errors and solver approximations accumulate over time, causing state drift. These inaccuracies mean a policy optimized for simulated physics may apply forces or torques that are ineffective or unstable on real hardware.

The difference between simulated sensor data and real-world sensor readings creates a significant distribution shift for perception models. This includes:

• Rendering artifacts: Computer-generated imagery lacks the noise, motion blur, lens distortions, and complex lighting conditions (e.g., glare, shadows) of real cameras.
• Texture and material realism: Procedural textures and simple shaders do not replicate the visual complexity of real surfaces.
• Sensor noise and latency: Simulated sensors often provide perfect, instantaneous readings, ignoring real-world sensor noise, dropouts, and processing delays.
• Proprioceptive differences: Simulated joint encoders and force-torque sensors lack the quantization error, bias, and drift present in physical sensors. A policy trained on pristine simulated observations can fail when presented with the noisy, incomplete data stream from real sensors.

Simulations are necessarily closed-world systems, while the real world is open-ended and infinitely variable. Critical unmodeled factors include:

• Stochastic disturbances: Air currents, vibrations, uneven flooring, and incidental human contact are rarely modeled.
• Object property variance: Real objects have variable weight, dimensions, friction, and compliance that are hard to parameterize exhaustively in simulation.
• Wear and tear: Simulation assumes consistent hardware performance, ignoring mechanical wear, battery drain, or thermal effects that alter system behavior over time.
• Unstructured elements: Simulation environments are typically clean and orderly, lacking the clutter, debris, and unpredictable obstacles of real workspaces. This lack of variability training leads to policies that are brittle and fail when encountering novel real-world conditions.

This is the statistical manifestation of the above causes: the joint probability distribution of observations and dynamics in simulation $P_{sim}(s', r | s, a)$ differs from the real-world distribution $P_{real}(s', r | s, a)$. This shift occurs in:

• State transition dynamics: The function that predicts the next state given the current state and action is different.
• Observation function: The mapping from the underlying state to the perceived observation (e.g., image) is different.
• Reward function: Even if the reward is algorithmic, its calculation based on inaccurate states or observations will be flawed. Reinforcement learning policies, which are essentially functions optimized for $P_{sim}$, suffer degraded performance when the underlying distribution changes to $P_{real}$. This is the core statistical challenge of sim-to-real transfer.

The timing of control loops in simulation is idealized and deterministic, which rarely matches real-time execution on physical hardware. Key discrepancies include:

• Control frequency: Simulation often runs at a fixed, ideal timestep (e.g., 1 kHz), while real control loops are subject to jitter and may run at lower, variable frequencies due to computational load.
• Action buffering and delay: In real systems, commanded actions experience communication delays, processing queues, and actuator response times that are typically absent or minimized in simulation.
• Synchronization issues: Simulated sensors are perfectly synchronized with the physics engine. Real sensor fusion must handle asynchronous data streams from multiple hardware components.
• Real-time constraint violation: Policies that require intensive computation may violate real-time deadlines when deployed, leading to dropped commands or unstable control. These temporal misalignments can destabilize feedback controllers that were stable in the temporally perfect simulation environment.

Simulations often provide direct, privileged access to the full ground-truth state of the world (e.g., exact object poses, velocities), while real systems must infer this state from noisy, partial sensor data.

• Privileged information: Policies trained with access to perfect state vectors (like [x, y, z, qx, qy, qz, qw, vx, vy, vz, ...]) develop a dependency on this information, which is unavailable in reality.
• State estimation error: Real-world state estimators (e.g., Kalman filters, visual odometry) introduce lag and error. A policy expecting perfect state will not be robust to this estimation noise.
• Occurrence of unobservable states: Certain real-world states (e.g., internal friction temperature, battery internal resistance) are not just noisy but completely unmeasured, yet they affect system dynamics. Bridging this gap requires either training policies directly on raw sensor data (observation-space) or developing sim-to-real transfer methods for the state estimator itself.

A core technique for bridging the reality gap by training a policy in a simulation where visual and physical parameters (e.g., textures, lighting, friction, mass) are randomly varied across episodes. This forces the model to learn robust, domain-invariant features rather than overfitting to specific simulation artifacts. It is a form of data augmentation at the environment level.

• Key Mechanism: Introduces variability to prevent the policy from exploiting simulation-specific quirks.
• Example: Training a robot arm to grasp objects with randomized object colors, table textures, and lighting directions.

The process of building or calibrating a mathematical model of a physical system (like a robot's actuator dynamics or joint friction) from measured input-output data. Accurate system identification is critical for improving simulation fidelity, thereby reducing the reality gap before policy training begins.

• Purpose: To create a simulation that more closely matches the real-world dynamics of the target hardware.
• Methods: Often involves collecting telemetry from the real robot and using optimization to tune simulation parameters like motor gains, inertia, and damping coefficients.

A broader machine learning field focused on adapting a model trained on a source domain (simulation) to perform well on a different but related target domain (reality). Techniques aim to minimize the distribution shift between domains. Key approaches include:

• Feature Alignment: Minimizing statistical distance (e.g., using Maximum Mean Discrepancy) between simulation and real data features.
• Adversarial Training: Using a domain classifier and gradient reversal layer to learn domain-invariant representations.
• Self-Training: Using the model's own predictions on unlabeled real data to generate pseudo-labels for further training.

The real-time adjustment of a policy's parameters or internal state during deployment on the physical system. This allows the agent to compensate for unexpected dynamics, wear and tear, or environmental changes not captured in simulation.

• Contrast with Zero-Shot Transfer: Does not assume a perfect, one-time transfer; instead, it continuously learns from real-world interaction.
• Methods: Can involve fine-tuning neural network weights, adapting a dynamics model, or adjusting a Bayesian filter to account for observed state estimation errors.

The degree of accuracy with which a simulation replicates the dynamics, visuals, and sensor modalities of the target real-world system. High-fidelity simulations aim for photorealism and physically accurate contact models but are computationally expensive. The reality gap is inherently tied to the trade-off between fidelity and computational cost.

• High-Fidelity: Uses accurate physics engines (e.g., MuJoCo, Bullet) and detailed rendering (e.g., ray tracing). May still have gaps due to unmodeled effects.
• Low-Fidelity: Faster, enabling massive parallelism for RL training, but relies on techniques like domain randomization to compensate for inaccuracies.

The ideal outcome where a policy trained exclusively in simulation is deployed directly on a physical robot without any fine-tuning or additional training on real-world data. Success depends entirely on the robustness of the simulation training methodology (e.g., extensive domain randomization) to bridge the reality gap.

• Benchmark: Considered the gold standard for evaluating sim-to-real methods, as it eliminates the need for costly and time-consuming real-world data collection.
• Challenge: Requires the simulation training distribution to encompass the real-world test distribution, which is difficult to guarantee.