Observation Space Mismatch

In simulation, a policy often receives a perfect state vector—a direct, noiseless feed of internal variables like joint angles, positions, and velocities. In reality, this state must be estimated from imperfect sensors.

• Simulation: Direct access to [x, y, z, θ, vx, vy, vz].
• Reality: Must fuse data from an Inertial Measurement Unit (IMU), encoders, and cameras, each with inherent noise, bias, and latency.
• Consequence: The policy receives fundamentally different input distributions, a classic covariate shift problem.

Visual observations from simulation engines contain systematic rendering artifacts not present in real camera feeds.

• Simulation Artifacts: Perfectly sharp edges, uniform lighting, simplified textures, and a lack of motion blur or lens distortion.
• Real-World Complexity: Dynamic lighting (shadows, glare), complex textures, motion blur, sensor noise (e.g., Gaussian noise, hot pixels), and optical imperfections.
• Impact: A convolutional neural network (CNN) feature extractor trained on synthetic visuals may latch onto these artifacts as shortcuts, failing to generalize to natural images.

The timing of observations is critically different. Simulations typically provide synchronous, zero-latency state updates.

• Simulation: Policy receives observation O_t exactly at time t.
• Reality: Sensor latency, communication delays, and processing time mean the observation O_t actually corresponds to the system's state at time t - Δt. This temporal misalignment can destabilize control loops, especially for high-speed tasks.
• Example: A drone controller expecting instant lidar feedback will fail if the real sensor loop has a 50ms delay.

Simulations often use simplified models for proprioceptive sensors that measure internal state, leading to subtle but critical mismatches.

• Force/Torque Sensors: Simulated contact forces may use basic spring-damper models, ignoring hysteresis, non-linear calibration curves, and cross-axis coupling present in real strain gauges.
• Motor Encoders: Simulated encoders provide perfect resolution, ignoring quantization error, electrical noise, and backlash in gearboxes.
• Result: A policy relying on precise force feedback for delicate manipulation (e.g., insertions) will fail due to these unmodeled dynamics.

Real robots often have sensors not modeled in simulation, or simulation provides data in a form that aliases real-world complexity.

• Missing Modalities: Real systems may use microphones, thermal cameras, or tactile sensors absent from the sim, leaving the policy without crucial data streams.
• Aliasing: Simulation may provide a single-beam lidar point, while the real sensor is a spinning 32-beam lidar producing a point cloud. The policy is trained on a grossly simplified representation of the same perceptual modality.
• Challenge: This requires either sensor dropout during training or modality translation networks at deployment.

Every real sensor has systematic biases that are static but unmodeled in simulation, where sensors are assumed perfectly calibrated.

• Camera Intrinsics/Extrinsics: Real cameras have focal length error, principal point offset, and lens distortion parameters (k1, k2, p1, p2) that are only approximately known.
• IMU Bias: Gyroscope and accelerometer biases drift over time and with temperature.
• Effect: These fixed errors cause a persistent shift in the observation space. A policy trained in a perfectly calibrated sim will interpret the world incorrectly, as if seeing through a consistently misaligned lens.

In simulation, a policy often receives a perfect state vector (e.g., exact joint angles, object positions). In reality, it must rely on noisy encoders, IMUs, and delayed sensor feedback. A policy trained on perfect state may fail because it never learned to filter noise or handle latency.

• Example: A robotic arm trained in simulation to insert a peg receives exact 6D pose of the peg and hole. On real hardware, it receives joint encoder readings with ±0.5° error and a camera feed with 33ms latency, causing overshoot and failed insertions.

Simulated synthetic RGB images lack the artifacts of physical cameras. A vision-based policy trained on clean, perfectly lit synthetic data will fail when presented with lens distortion, motion blur, auto-exposure changes, and compression artifacts from a real camera.

• Example: A warehouse robot trained to navigate using synthetic visuals fails in a real aisle because real camera feed exhibits motion blur from fast movement, lens flare from overhead lights, and varying color temperatures, making visual features unrecognizable.

Simulated ideal depth sensors provide perfect, noise-free distance measurements with infinite range and no missing data. Real LiDAR and structured-light sensors (e.g., Intel RealSense) produce sparse point clouds, specular reflection errors, and missing data on dark or transparent surfaces.

• Example: A drone trained in simulation to avoid obstacles using perfect depth maps crashes when its real depth sensor fails to return data from a shiny, black floor or a glass window, creating 'holes' in its perception of the environment.

High-fidelity control often requires force/torque sensing. In simulation, these values are computed directly from the physics engine. On real robots, strain gauges and torque sensors introduce calibration drift, saturation limits, and high-frequency electrical noise.

• Example: A policy for a legged robot trained to walk on uneven terrain using perfect ground reaction forces from simulation becomes unstable on real hardware because the raw force-torque sensor data is contaminated with vibrational noise from the motors, leading to incorrect footing adjustments.

Simulations often operate in lockstep, with observations delivered instantly after an action. Real systems have pipeline latency: sensor readout, serial communication, network delay, and computation time create a temporal mismatch between action and observation.

• Example: A self-driving car policy trained in a synchronous simulator makes steering decisions based on the immediate result of its last action. Deployed on a real car, the policy receives camera and LiDAR data that is 100ms old due to processing latency, causing it to over-correct and oscillate.

A policy may be trained in simulation with a simplified sensor suite (e.g., only vision). The real robot may have additional critical sensors (e.g., tactile sensors, microphones, thermal cameras) that the policy cannot interpret, or lack sensors that the policy implicitly relied on.

• Example: A manipulation policy trained in simulation with a wrist-mounted force sensor for delicate assembly fails on a real robot that lacks such a sensor. Conversely, a policy trained without a tactile sensor cannot leverage one if it becomes available, missing a key source of information for contact-rich tasks.

The reality gap (or sim2real gap) is the overarching performance discrepancy between a policy's behavior in simulation and its behavior on physical hardware. Observation space mismatch is a primary contributor, alongside dynamics mismatch and actuation differences.

• Causes: Inaccurate sensor models, simplified rendering, perfect state access in sim.
• Impact: Policies that excel in simulation fail or behave unpredictably in the real world.
• Mitigation: Techniques like domain randomization, system identification, and robust policy training aim to minimize this gap.

Simulation bias refers to the systematic errors or simplifications built into a simulation engine that cause the training environment to diverge from reality. In the context of observations, this includes:

• Perfect State Vectors: Simulation often provides direct access to ground-truth object positions and velocities.
• Noise-Free Sensors: Simulated cameras or lidar lack the grain, motion blur, and calibration errors of real hardware.
• Latency Ignorance: Simulation typically processes observations instantly, unlike real sensor pipelines which introduce delay. This bias creates a policy that is overfit to an idealized, non-existent perceptual world.

Domain randomization is a key technique for combating observation space mismatch. It involves training a policy in a simulation where visual and perceptual parameters are randomly varied across episodes.

• Visual Randomization: Textures, lighting, colors, and object appearances are altered.
• Sensor Noise: Artificial noise, dropout, and distortion are injected into simulated sensor readings.
• Goal: By never seeing the same observation twice, the policy is forced to learn robust features that are invariant to these superficial changes, improving generalization to the unseen real-world observation distribution.

Sensor fusion is the algorithmic process of combining data from multiple, heterogeneous real-world sensors (e.g., camera, IMU, lidar) to create a more accurate and reliable state estimate. It is a critical real-world counterpart to simulated perfect state.

• Compensates for Mismatch: Fuses noisy, partial observations from individual sensors.
• Examples: Kalman filters, particle filters, and deep learning-based fusion networks.
• Role in Transfer: A policy trained on fused simulated sensor data must be robust to the failures and latencies inherent in the real fusion pipeline.

State estimation is the process of inferring the hidden internal state of a system (e.g., a robot's pose, velocity) from a sequence of noisy sensor measurements. In simulation, this state is often provided directly, creating a major observation space mismatch.

• Real-World Necessity: Cameras don't output "robot x,y coordinates"; an estimator must derive this from pixels.
• Algorithms: Includes Extended Kalman Filters (EKF), optimization-based visual-inertial odometry (VIO), and SLAM.
• Transfer Challenge: Policies trained on perfect state must adapt to the errors and drift inherent in any real-world state estimator.

Online adaptation enables a deployed policy to adjust its parameters or behavior in real-time based on streaming observations from the real world. This is a direct method for correcting for observation space mismatch after deployment.

• Mechanism: Uses recent experience to fine-tune a perception network or adjust policy weights.
• Use Case: Adapting to new lighting conditions, sensor degradation, or unforeseen visual clutter.
• Requirement: Must be computationally efficient and stable to avoid catastrophic forgetting or unsafe oscillations during operation.