Performance Drop

This is the divergence between the physics models used in simulation and the true physical laws governing the real robot and its environment. Even high-fidelity simulators make approximations.

Key discrepancies include:

• Contact and friction modeling: Simulated collisions and surface interactions are often simplified, leading to inaccurate force feedback during manipulation or locomotion.
• Actuator dynamics: Motors, gears, and hydraulics have non-linear properties like saturation, backlash, and response latency that are difficult to model perfectly.
• Mass and inertia properties: Inaccurate CAD models or unmodeled payloads change the robot's dynamic response.
• Fluid and aerodynamics: Air resistance or water currents are frequently omitted in rigid-body simulators.

Example: A robot arm trained in simulation to insert a peg might fail because simulated contact forces don't match the real vibrations and stick-slip behavior.

The difference between rendered synthetic images and real sensor readings creates a major bottleneck for vision-based policies. This gap affects both the content and the noise characteristics of the data.

Critical factors are:

• Rendering artifacts: Perfect textures, global illumination, and lack of lens effects (e.g., chromatic aberration, vignetting) make simulation visually 'clean'.
• Sensor noise and distortion: Real cameras exhibit Gaussian noise, motion blur, rolling shutter effects, and lens distortion absent in perfect renders.
• Lighting and material properties: Simulated lighting (e.g., perfect point lights) and material reflectivity rarely match the complex, dynamic illumination of real scenes.
• Domain shift in features: A policy may learn to rely on simulation-specific visual cues that don't exist in reality, a form of overfitting to simulation.

This is why Reinforcement Learning from Pixels is particularly susceptible to performance drop.

Real robots have imperfect low-level control and state feedback, while simulations often assume perfect command execution and full observability.

This encompasses:

• Control latency: The delay from command computation to physical movement is non-zero and variable, disrupting timing-critical tasks.
• Tracking error: A real actuator cannot instantaneously achieve a commanded torque or position, leading to steady-state error.
• Proprioceptive sensor noise: Encoders, IMUs, and torque sensors provide noisy, biased, and delayed readings compared to perfect ground truth in sim.
• Calibration drift: Over time, kinematic and dynamic calibration parameters change due to wear and temperature, making the initial simulation model obsolete.

Example: A walking robot trained in simulation with perfect joint angle feedback may fall over in reality because its state estimator fuses noisy IMU data, providing a delayed and inaccurate estimate of torso orientation.

Simulations are closed worlds with bounded parameters, while the real world presents open-world challenges and long-tail distributions of events.

Sources of variability include:

• Object and material diversity: A policy trained to grasp a few simulated object meshes may fail on the vast diversity of shapes, textures, and compliance found in real objects.
• Unstructured obstacles: Simulated environments often have clean, known geometry. Real spaces contain wires, loose debris, and moving people.
• Stochastic disturbances: Random air currents, vibrations from other machinery, or uneven flooring are typically not modeled.
• Task specification ambiguity: Real-world task goals (e.g., 'tidy the room') are more ambiguous than the precisely defined reward functions of simulation.

Techniques like Domain Randomization explicitly target this cause by exposing the policy to a vast range of randomized simulation parameters during training.

Practical constraints force simulations to operate at limited temporal resolution, spatial scale, or with simplified world semantics, creating systematic gaps.

Common bottlenecks are:

• Fixed simulation timestep: A coarse timestep (e.g., 1ms vs. real-world continuity) aliases high-frequency dynamics, missing critical transient states.
• Collision mesh simplification: Complex geometries are approximated with primitive shapes (spheres, capsules) for computational speed, altering contact points.
• Lack of soft-body and deformation physics: Most robotics simulators are rigid-body based, unable to simulate flexing cables, deforming bags, or compliant grips accurately.
• Discrete vs. continuous action spaces: Policies are often trained with discretized actions, but real actuators operate in a continuous domain.

These abstractions are necessary for training speed but create a fundamentally different interaction space.

In long-horizon tasks, small inaccuracies in dynamics, perception, or control do not cancel out; they accumulate over time, leading to catastrophic failure. Simulation-trained policies lack the experience to recover from these drifted states.

This manifests as:

• State distribution shift: The policy encounters out-of-distribution (OOD) states in the real world that were never visited during simulation training, as early errors push it off its expected trajectory.
• Lack of robustness to perturbations: A simulated policy may succeed only on near-perfect executions of its planned path and has no learned strategy for correction.
• Error propagation in closed-loop control: An initial misperception of an object's location leads to a poor grasp, which leads to a failed placement—a chain of failures not seen in sim.

This is why Residual Policy Learning, where a learned network corrects a stable base controller, is often more effective than training an end-to-end policy from scratch in sim.

The Reality Gap is the fundamental discrepancy between the simulated training environment and the physical world, which causes Performance Drop. This gap manifests in three primary areas:

• Dynamics Gap: Differences in physics, friction, actuator response, and contact modeling.
• Visual Gap: Differences in lighting, textures, sensor noise, and rendering artifacts.
• State Estimation Gap: Differences between perfect simulated state access and noisy, delayed real-world sensor readings. The goal of sim-to-real transfer is to develop techniques that bridge this gap.

Domain Randomization is a core technique for combating Performance Drop by training a policy to be robust to a wide spectrum of randomized simulation conditions. Instead of chasing perfect fidelity, it exposes the agent to variability in:

• Visual parameters: Textures, lighting, colors, and camera angles.
• Physical parameters: Mass, friction coefficients, motor gains, and latency.
• Environmental parameters: Object sizes, initial positions, and obstacle layouts. The policy learns invariant features, improving its chances of generalizing to the unseen real world, thereby reducing the expected Performance Drop.

System Identification is the process of building or refining a mathematical model of a physical robot's dynamics by observing its input-output behavior. It directly addresses the dynamics component of the Reality Gap. Common approaches include:

• Black-box modeling: Using neural networks to map control inputs to observed states.
• Grey-box modeling: Fitting parameters (e.g., inertia, friction) to a known physics model structure. A more accurate identified model can be used to create a higher-fidelity simulation, reducing the Performance Drop for policies trained within it.

Domain Adaptation is a machine learning subfield focused on transferring knowledge from a labeled source domain (simulation) to an unlabeled or sparsely labeled target domain (reality). Techniques aim to learn domain-invariant features so a model performs well in both. Key methods relevant to sim-to-real include:

• Domain-Adversarial Training: A discriminator network tries to identify the domain of features, while the feature extractor learns to fool it.
• CycleGAN: Translates unpaired images from simulation to a photorealistic style, helping perception models. Successful domain adaptation minimizes the Performance Drop for perception and control models.

These are two primary paradigms for deploying a simulation-trained policy, each with different implications for Performance Drop management:

• Zero-Shot Transfer: The policy is deployed on the physical system without any real-world training data. Performance Drop is inevitable and must be mitigated entirely through robust training techniques like Domain Randomization. It's ideal for tasks where real-world interaction is expensive or dangerous.
• Fine-Tuning Transfer: The policy is first pre-trained in simulation, then adapted using a limited amount of real-world interaction data. This approach explicitly trades off some real-world data collection for a significant reduction in Performance Drop. Techniques like Residual Policy Learning are often used here.

Simulation Fidelity is the degree to which a virtual environment replicates real-world characteristics. Simulation Validation is the process of quantifying this fidelity. They are critical for diagnosing and predicting Performance Drop.

• High-Fidelity Sims: Use accurate physics engines (e.g., MuJoCo, Isaac Sim) and photorealistic renderers. Reduce the dynamics gap but are computationally expensive.
• Validation Metrics: Include comparing trajectory rollouts, contact forces, or power consumption between sim and real for identical control inputs. A validated, high-fidelity simulation provides a reliable baseline; the Performance Drop observed when moving to a lower-fidelity or unvalidated sim can inform the severity of the real-world gap.