Sim-to-Real Gap

Simulators use simplified physics engines (e.g., MuJoCo, PyBullet, Isaac Sim) that approximate real-world dynamics. Key inaccuracies include:

• Contact dynamics: Simulated friction, restitution, and soft-body deformations rarely match the complex, multi-point contacts of a robotic hand manipulating an object.
• Actuator dynamics: Idealized motor models ignore real-world effects like backlash, stiction, torque ripple, and bandwidth limits.
• Object properties: Mass, inertia, and center of mass are often estimated or standardized, unlike the variability of real objects. These approximations lead to policies that exploit 'sim physics' and fail when physical laws differ.

The synthetic perception pipeline in simulation differs drastically from real sensor streams.

• Rendering artifacts: Perfect lighting, textureless objects, and noiseless depth maps in simulation contrast with the variable lighting, motion blur, and sensor noise (e.g., from RGB-D cameras) in the real world.
• Sensor simulation: Simulated tactile sensors (e.g., GelSight) or force-torque sensors provide clean, high-fidelity signals, while real sensors have latency, calibration drift, and electrical noise.
• Domain shift: A policy trained on perfectly segmented, simulated object meshes cannot generalize to the visual clutter and occlusions of a real workbench.

Critical low-level hardware behaviors are absent or oversimplified in simulation.

• Control latency: Simulation often assumes instantaneous command execution and state feedback. Real systems have non-negligible loop delays from sensor processing, network communication, and actuator response.
• Joint flexibility & vibration: Many simulators model robotic arms as rigid bodies, ignoring harmonic vibrations and flex in gears and links that occur during high-speed or high-force manipulation.
• Electrical dynamics: Power supply limitations, current saturation, and thermal effects on motor performance are typically not simulated. A policy commanding maximum torque may cause a real motor to overheat and shut down.

Simulators provide perfect, low-dimensional ground truth state (e.g., exact object pose, joint angles), while real robots must infer state from noisy, partial observations.

• State estimation error: In reality, object 6D pose estimation from vision is noisy and can fail under occlusion. Joint encoders have quantization error.
• Hidden variables: Simulation state vectors exclude real-world variables like dust on a surface, electrostatic forces, or slight adhesive properties of materials, all of which affect manipulation.
• Partial observability: A policy trained on full state may not learn to be robust to the uncertainty inherent in perceptual systems, causing failures when state estimates are incorrect.

Learning algorithms can exploit deterministic quirks and shortcuts present only in the simulated environment.

• Deterministic randomness: If a simulator's random number generator has a limited seed space, the policy may memorize successful actions for specific seeds rather than learning robust strategies.
• Discretization artifacts: Simulated time is discretized into fixed steps. Policies can learn to 'tap' objects at precise simulation steps to achieve motion, a strategy that fails with continuous real-world time.
• Simulation-specific features: The policy may latch onto non-physical rendering artifacts or simulator-specific state variables as predictive features, creating a dependency that doesn't transfer.

Simulations typically model a pristine, unchanging robot and environment, unlike the real world.

• Mechanical wear: Joint backlash increases, grease degrades, and tactile sensor surfaces get scratched over time, altering system dynamics.
• Object variability: Real objects have manufacturing tolerances, surface imperfections, and deformability (e.g., a slightly bent spoon) not captured by a canonical CAD model.
• Environmental drift: Ambient temperature and humidity affect material properties and sensor readings. A policy trained in a climate-controlled sim is not exposed to this drift. This lack of long-term variability training leads to brittle performance.

A robot trained in simulation to pick diverse items from bins often fails in the real warehouse due to visual domain shift (different lighting, worn item textures) and physical modeling errors (unmodeled object deformability, static friction coefficients).

• Real Gap: A policy achieving 99% success in simulation drops to ~70% on physical KUKA or UR arms.
• Mitigation: Domain randomization of lighting, colors, and textures during training, combined with system identification to calibrate the simulation's physics engine (e.g., MuJoCo, PyBullet) to the real robot's dynamics.

Drones trained to fly through obstacle courses in simulation crash when deployed due to unmodeled aerodynamic effects (wind gusts, ground effect) and sensor latency/noise not present in the perfect simulated sensors.

• Real Gap: Perfect flight in AirSim or FlightGym leads to instability and collisions outdoors.
• Mitigation: Injecting synthetic noise and latency into simulated sensor streams (IMU, camera) and training with robust control paradigms like Model Predictive Control (MPC) that can handle dynamic disturbances.

Training a da Vinci surgical robot in simulation to suture soft tissue fails because simulated tissue mechanics are overly simplified, lacking complex non-linear deformation, tearing, and tool-tissue interaction forces.

• Real Gap: A smooth suturing policy in simulation causes tissue damage or suture breakage on physical organ phantoms.
• Mitigation: Using finite element method (FEM) simulations for higher-fidelity soft-body dynamics and adaptive control strategies like admittance control that react to real-time force feedback, bypassing inaccuracies in the forward dynamics model.

A quadruped robot (e.g., Unitree A1, Boston Dynamics Spot) mastering complex terrain in simulation collapses on real grass or gravel due to terrain parameter mismatch (simulated friction, compliance) and actuator dynamics (motor backlash, bandwidth) not captured in the model.

• Real Gap: Robust walking in RaiSim or Isaac Sim translates to stumbling and falls.
• Mitigation: Domain randomization over terrain friction, restitution, and motor strength. Simulation calibration using real robot data collection. Deployment often uses a hierarchical controller where a learned policy outputs desired joint torques, but a fast, model-based low-level controller ensures stability.

Manipulating cables, cloth, or dough in simulation is notoriously difficult to transfer. The sim-to-real gap is extreme due to the high dimensionality of deformation states and the inaccurate material models (hyperelasticity, plasticity) in even the best simulators like NVIDIA FleX or SoftGym.

• Real Gap: A policy that folds a simulated towel cannot manipulate a real, crumpled towel.
• Mitigation: System identification of material parameters from real-world videos. Latent space action spaces that abstract low-level control. Heavy reliance on real-world fine-tuning or demonstration learning from a few physical examples.

Engineers deploy a suite of algorithms to minimize the sim-to-real gap, treating simulation not as a perfect replica but as a structured data generator.

• Domain Randomization (DR): Randomizing visual (textures, lighting) and physical (masses, friction) parameters during training to force the policy to learn invariant features.
• System Identification: Automatically tuning simulation parameters (e.g., using Bayesian optimization) to match real-world robot data traces.
• Domain Adaptation: Using techniques like adversarial training to align the feature spaces of simulated and real sensor data.
• Reinforcement Learning with Real-World Priors: Starting with a policy pre-trained in simulation and then performing fine-tuning or online adaptation on the physical system with safe exploration constraints.

A synonym for the sim-to-real gap, emphasizing the fundamental discrepancy between any computational model and the physical world. This gap manifests in two primary forms:

• Visual gap: Differences in rendering, lighting, and texture between synthetic images and real camera feeds.
• Dynamics gap: Inaccuracies in the simulated physics engine regarding contact forces, friction, material deformation, and actuator models. The reality gap is the primary source of performance degradation when transferring a policy, necessitating the techniques listed in other cards.

The ideal outcome where a policy trained exclusively in simulation operates successfully on the first attempt in the real world, without any fine-tuning or additional real-world training. Achieving this requires highly robust training techniques, such as:

• Extensive domain randomization across all relevant parameters.
• Learning from demonstrations (IL) that are scripted in simulation.
• Robust policy architectures that filter out domain-specific noise. Success is measured by the policy's performance on the real-world task metric immediately upon deployment.