Sim-to-Real Gap

A primary technique for bridging the Sim-to-Real Gap by randomizing the visual and physical parameters of the simulator during training. Instead of training on a single, highly accurate environment, the agent is exposed to a vast distribution of dynamics, including varying friction, mass, lighting, and camera positions. This forces the policy to learn invariant features that are essential to the task, treating the real world as just another sample from the training distribution. OpenAI famously used this to train a robotic hand to solve a Rubik's Cube entirely in simulation.

A primary technique for closing the sim-to-real gap by massively varying simulation parameters during training.

• Randomizes visual properties: lighting, textures, camera position
• Randomizes physical dynamics: friction, mass, joint stiffness
• Forces the policy to learn invariant features rather than memorizing simulator artifacts
• Policy trained this way treats reality as just another randomization instance

Example: OpenAI's robotic hand manipulation trained entirely in simulation with randomized gravity, joint friction, and object dimensions, then deployed zero-shot to a physical robot.

The process of calibrating a simulator's physical parameters to match a specific real-world system, reducing the reality gap at the source.

• Measures real-world properties: motor response curves, link masses, damping coefficients
• Minimizes the discrepancy between simulated and observed trajectories
• Often combined with domain randomization for residual unmodeled effects
• Critical for high-precision tasks like robotic assembly

Trade-off: Overly precise system identification can lead to brittle policies that fail under minor wear-and-tear or environmental changes.

A transfer learning approach that aligns feature representations between the source domain (simulation) and target domain (reality) without requiring labeled real-world data.

• Uses adversarial training to learn domain-invariant features
• Minimizes Maximum Mean Discrepancy (MMD) between simulation and real feature distributions
• Can operate at the pixel level (making simulated images look real) or feature level
• Often paired with a small amount of unlabeled real-world data

Application: Enables a vision-based navigation policy trained on synthetic images to operate on real camera feeds by learning to ignore textural differences.

A neural architecture designed to transfer knowledge from simulation to reality without catastrophic forgetting.

• Instantiates a new column (network) for each new task or domain
• Lateral connections from previously frozen columns enable feature reuse
• The real-world column learns to leverage simulated features while adapting to reality
• Eliminates the need to fine-tune and potentially destroy simulation-learned behaviors

Advantage: Maintains a library of reusable skills across the sim-to-real continuum, crucial for lifelong learning in physical robots.

The overarching methodology of deploying a policy trained entirely or primarily in simulation onto a physical system, encompassing the entire pipeline from training design to reality gap mitigation.

• Zero-shot transfer: Policy deploys directly to reality with no fine-tuning
• Few-shot adaptation: Policy updates with a small amount of real-world interaction data
• Iterative transfer: Alternating between simulation training and real-world deployment, using real data to improve the simulator

Key Metric: The sim-to-real transfer ratio—the percentage of simulation performance retained in reality. State-of-the-art approaches achieve over 90% for locomotion tasks.

A specific form of domain randomization focused exclusively on physical parameters to bridge the sim-to-real gap in contact-rich manipulation and locomotion.

• Randomizes: mass, inertia, friction coefficients, joint damping, actuator gains
• Randomizes: contact parameters like stiffness and penetration depth
• Policy learns robust control strategies that work across a distribution of dynamics
• Particularly effective for legged locomotion where ground contact modeling is imprecise

Result: The ANYmal quadruped robot learned to trot, pace, and recover from falls entirely in simulation using dynamics randomization, transferring zero-shot to physical hardware.