Simulation Bias

The core engine that simulates physical laws is inherently simplified. Key approximations include:

• Contact dynamics: Simplified collision detection and response models that lack microscopic surface properties.
• Friction models: Use of basic Coulomb friction, ignoring temperature, wear, and velocity-dependent effects.
• Actuator dynamics: Idealized motor models that omit backlash, saturation, and non-linear torque-speed curves.
• Material properties: Homogeneous, linear models for rigidity, damping, and deformation. These approximations mean a policy never experiences the true complexity of physical interaction, leading to failure when real-world nuances emerge.

Simulated sensors provide a clean, often perfect, abstraction of real hardware. This creates an observation space mismatch. Common biases include:

• Rendering artifacts: Computer graphics pipelines produce noise-free, perfectly lit images without real lens distortions, motion blur, or sensor noise.
• Proprioceptive latency: Simulated joint encoders report ground-truth state instantly, ignoring the real signal processing and communication delays.
• Depth sensor inaccuracies: Simulated lidar/radar returns perfect geometry, missing multi-path reflections, material absorption, and beam divergence.
• Tactile simulation: Crude models for force/torque sensors that lack the hysteresis and calibration drift of physical strain gauges. A policy trained on this pristine data is brittle when faced with the noisy, delayed, and incomplete perceptions of real sensors.

The simulation defines a limited, often non-representative, distribution of environmental conditions. This is a form of covariate shift. Examples are:

• Lighting and texture: A fixed set of HDRI skyboxes and object textures, lacking the infinite variability of natural illumination and surface wear.
• Object geometry and mass: Using canonical CAD models with perfect dimensions and uniform density, unlike manufactured parts with tolerances and internal imperfections.
• Terrain generation: Procedurally generated floors with mathematically smooth noise, failing to capture the stochastic roughness, debris, and compliance of real ground.
• Distractor omission: The absence of unmodeled objects, air currents, vibrations, or electromagnetic interference that exist in operational environments. The policy's world model is built on this curated subset of reality, limiting its generalization.

Simulation runs in discrete time steps, creating artifacts not present in continuous reality. Key issues are:

• Integration error: Numerical solvers (e.g., Euler, Runge-Kutta) approximate continuous dynamics, accumulating error over long rollouts, especially in stiff systems.
• Fixed time-stepping: Using a constant Δt (e.g., 1ms) simplifies computation but can alias high-frequency dynamics and cause artificial energy gain or loss.
• Action hold times: Policies typically output actions at a lower control frequency than the physics engine updates, creating zero-order-hold behavior that doesn't match real controller output.
• Deterministic execution: Perfect synchronization between perception, computation, and actuation cycles, unlike the jitter and variable latency of real-time operating systems. These discretization effects train policies for a temporally idealized world that does not exist.

The reward signal, which shapes policy behavior, is often easier to compute in simulation than to measure in reality, creating a reward bias. Discrepancies include:

• Ground-truth state rewards: Providing reward based on the simulator's internal, perfect state (e.g., exact object position), which is unavailable or noisy in the real world.
• Dense reward shaping: Engineers add many intermediate reward signals to guide learning, which may not correspond to any physically measurable quantity.
• Simplified cost terms: Penalizing squared joint velocities or torques, while real hardware has complex, non-quadratic wear-and-tear costs.
• Omission of unmodeled costs: Failing to penalize high-frequency vibrations or specific actuator modes that cause overheating in physical motors. The policy optimizes for a proxy objective that may not align with true operational success and hardware longevity.

The process of tuning simulation parameters to match a real system (system identification) is imperfect, leaving residual dynamics mismatch. Limitations are:

• Under-parameterized models: The chosen simulation model structure (e.g., a rigid body with 6 DoF) may be insufficient to capture complex flexures, hydraulic compliance, or cable dynamics.
• Identification data scope: Calibration is performed in a limited regime (e.g., slow movements), leaving dynamics in high-acceleration or contact-rich regimes poorly modeled.
• Non-stationary parameters: Real system parameters (like friction) change over time due to wear, lubrication, and temperature—changes not reflected in the static simulation.
• Proprietary black boxes: For commercial robots, low-level controller dynamics and firmware are often opaque, making their accurate simulation impossible. This gap means the simulation is, at best, a calibrated approximation of the robot at a single point in its lifecycle.

The reality gap, also known as the sim2real gap, is the performance discrepancy observed when a policy trained in simulation fails to perform as expected on physical hardware. It is the direct, measurable consequence of simulation bias. This gap manifests as:

• Task failure or degraded performance metrics.
• Unstable behavior or crashes not seen in simulation.
• The need for extensive real-world fine-tuning to recover performance. The primary goal of sim-to-real research is to develop methods that minimize this gap, enabling direct, reliable transfer.

Domain randomization is a primary technique for combating simulation bias. Instead of trying to create a perfect simulation, it intentionally varies simulation parameters (e.g., lighting, textures, friction coefficients, object masses) across a wide range during training. This forces the policy to learn robust, invariant features rather than overfitting to the specific biases of a single simulated world. Key randomized elements include:

• Visual properties: Colors, textures, camera noise.
• Dynamics parameters: Mass, friction, motor strength, latency.
• Environmental conditions: Object positions, lighting angles, background clutter. The policy learns to ignore irrelevant simulation artifacts and focus on the core task.

System identification (SysID) is the process of building or calibrating a mathematical model of a physical system using empirical data. In sim-to-real, it is used to reduce dynamics mismatch, a major source of simulation bias. The process involves:

• Collecting data from the real robot (e.g., joint positions, velocities, torques) under known control inputs.
• Fitting parameters of the simulation's physics engine (e.g., inertia tensors, damping coefficients, contact model parameters) to match the real data.
• Iteratively refining the simulation model to minimize the prediction error. While it can significantly narrow the reality gap, it is often impossible to identify all parameters perfectly, leading to residual bias.

Domain adaptation is a broader machine learning subfield focused on transferring knowledge from a labeled source domain (simulation) to an unlabeled target domain (reality). It provides the algorithmic framework for addressing simulation bias. Techniques include:

• Feature-level adaptation: Learning domain-invariant representations (e.g., via adversarial training).
• Instance-level adaptation: Reweighting simulated data to better match the real distribution.
• Model-level adaptation: Directly fine-tuning the policy on limited real-world data. These methods explicitly model and compensate for the distribution shift caused by bias, rather than relying solely on randomization.

Dynamics mismatch is a specific, critical type of simulation bias where the physics model in the simulation inaccurately represents the true physical dynamics. This is often the most challenging bias to overcome. Causes include:

• Simplified contact models that don't capture complex friction, deformation, or multi-point contact.
• Inaccurate actuator models ignoring motor saturation, backlash, or communication latency.
• Incorrect inertial parameters for robot links and payloads.
• Omitted effects like cable dynamics, flex in components, or thermal drift. Policies trained under biased dynamics learn an incorrect internal model of the world, leading to poor force control and instability on real hardware.

Observation space mismatch occurs when the sensory data provided to the policy in simulation differs fundamentally from real-world sensor streams. This is a perceptual form of simulation bias. Common mismatches include:

• Perfect state vs. noisy estimation: Simulation often provides perfect joint angles and object positions, while reality requires noisy state estimation from sensors.
• Rendered images vs. real camera feed: Synthetic images lack real sensor noise, lens distortion, motion blur, and complex lighting artifacts.
• Latency and synchronization: Simulated sensors are often perfectly synchronized and instantaneous, while real sensors have variable latency and jitter. Policies must be trained with simulated sensor models that inject appropriate noise and latency to be robust to this bias.