Residual Policy Learning

The core mechanism is additive composition. A base controller (e.g., a PID, MPC, or scripted policy) provides a nominal action a_base. A learned residual policy π_θ observes the state and outputs a corrective delta Δa. The final executed action is a = a_base + Δa.

• Key Benefit: The base controller provides stability and safety guarantees, handling fundamental dynamics. The residual network learns only the complex, unmodeled nuances.
• Example: A drone's base controller maintains hover. The residual policy learns to compensate for unmodeled aerodynamic effects like rotor wash or payload asymmetry.

This architecture directly addresses the sim-to-real transfer problem. The base controller is designed for the simplified simulation model. The reality gap—differences in friction, actuator latency, sensor noise—is treated as a disturbance to be corrected.

The residual policy is trained to output the specific corrections needed to achieve the simulated outcome in the real world. This decomposes the learning problem: instead of learning flight from scratch, the network learns only the delta between sim and real.

Residual policies are typically trained using Reinforcement Learning or Imitation Learning.

• RL in Simulation: The agent learns Δa to maximize reward, with a_base fixed. This is often more sample-efficient than learning a from scratch.
• Imitation Learning from Real Data: An expert (human or optimized controller) provides demonstrations of a_expert. The residual policy learns Δa = a_expert - a_base. This is common for fine-tuning transfer.
• Domain Randomization: The base controller's parameters or the simulation's physics are randomized during training to force the residual policy to learn a robust correction strategy.

Residual Policy Learning has formal parallels to disturbance observer and adaptive control theory from classical robotics.

• The residual policy acts as a non-linear disturbance estimator, learning to predict and cancel out unmodeled dynamics in real-time.
• Unlike fixed-gain observers, the neural network can model highly non-linear and state-dependent disturbances.
• This provides a learning-based upgrade path for legacy control systems, enhancing performance without a full rewrite.

A major engineering advantage is inherent safety. Since a_base is generated by a verifiable controller, the system can fall back to it if the learned component fails.

• Output Clamping: The residual Δa can be strictly bounded (|Δa| < ε) to prevent the network from issuing dangerous, large corrections.
• Graceful Degradation: If the perception system feeding the residual policy fails, the base controller maintains baseline operation, preventing catastrophic failure.
• This makes the architecture suitable for high-stakes physical systems like autonomous vehicles and industrial robots.

Manipulation: A factory robot uses a classical motion planner for a pick-and-place trajectory. A residual policy, trained with real vision data, learns fine-grained corrections to compensate for gripper slip and object deformation.

Legged Locomotion: A bipedal robot uses a model-based controller for balance. A residual policy, trained via RL in simulation with randomized ground friction, allows the robot to walk on unseen real-world surfaces like gravel or grass.

Autonomous Racing: A formula-style car uses a traditional trajectory-tracking controller. A residual policy learns the complex tire dynamics and aerodynamics not captured by the simple bicycle model, enabling faster lap times.