Online Adaptation

This approach uses a learned or identified approximate dynamics model of the real system to perform online planning or model predictive control (MPC). The model is updated continuously with real-world data, allowing the policy to replan optimal actions in real-time to account for observed discrepancies.

• Core Mechanism: A predictive model (e.g., neural network, Gaussian process) is queried at each timestep to simulate potential future states.
• Real-time Adjustment: The controller solves for actions that optimize a cost function over a short horizon, effectively adapting to the current conditions.
• Example: A drone uses an onboard learned wind model to adjust its motor commands milliseconds before a gust hits, maintaining stable flight.

This method leverages meta-learning frameworks like Model-Agnostic Meta-Learning (MAML) to pre-train a policy with a specific adaptation mechanism. During deployment, the policy performs rapid few-shot adaptation using a short buffer of recent experience.

• Pre-training Phase: The policy is meta-trained in simulation to quickly adapt to a wide distribution of randomized dynamics.
• Online Phase: On the real system, the policy takes a few gradient steps (or uses a closed-form update) based on a support set of recent state-action tuples, adjusting its parameters in seconds.
• Use Case: A robotic hand meta-trained on various object weights and surface friction can adjust its grip force within a handful of grasps when encountering a novel, slippery object.

This technique treats unknown real-world parameters (e.g., friction coefficient, payload mass) as latent variables to be estimated online. Algorithms like the Kalman Filter or Particle Filter recursively update a belief distribution over these parameters using incoming sensor data.

• Process: The filter maintains a probability distribution over system parameters. As new observations arrive, it uses Bayes' rule to update this belief.
• Policy Adjustment: The control policy is conditioned on the current parameter estimate, effectively becoming adaptive.
• Practical Application: An autonomous warehouse robot estimates the changing coefficient of friction of the floor as it moves from concrete to a polished surface, modulating its acceleration and braking accordingly.

Inspired by robust control and adversarial training, this method actively injects learned disturbance signals into the policy's observation or action space during deployment. A separate adversarial network is trained online to produce disturbances that challenge the policy, forcing it to learn compensating behaviors.

• Dual-Network Architecture: A primary policy network executes the task. A smaller, rapidly adapting disturbance network learns to generate perturbations.
• Online Training Loop: The disturbance network is updated to maximize a divergence (e.g., from a nominal trajectory), while the policy is updated to minimize task error despite the disturbances.
• Result: The policy becomes robust to a wide range of unmodeled dynamics, as it continuously trains against a simulated 'worst-case' scenario.

This approach maintains an ensemble of multiple models or policies. Disagreement within the ensemble is used as a proxy for epistemic uncertainty in unfamiliar real-world states. The system then adapts by weighting the actions of more confident ensemble members or seeking safer, exploratory actions.

• Mechanism: An ensemble of neural networks with different initializations is trained on simulation data. Their predictions will diverge in states not well-covered by training.
• Online Decision Rule: In deployment, high predictive variance triggers a fallback strategy, such as switching to a conservative controller, reducing action magnitude, or initiating a targeted data collection phase for adaptation.
• Benefit: Provides a built-in safety monitor and a clear signal for when online adaptation is most critically needed.

This is a direct parameter optimization approach. A subset of simulation parameters (e.g., motor torque constants, link masses) is continuously optimized to minimize the error between predicted and observed robot state transitions.

• Process: A differentiable simulator or an analytical dynamics model runs in parallel with the real robot. An optimizer (e.g., gradient descent) adjusts the model's physical parameters so its forward prediction matches the last N steps of real sensor data.
• Closed-Loop Adaptation: The updated, higher-fidelity model is then used by the policy or planner for the next control cycle.
• Example: A legged robot performs online system ID to estimate the weight of an unknown payload it has picked up, allowing its gait controller to instantly compensate for the new inertia.

Quadruped and bipedal robots use online adaptation to maintain stable walking on unseen, uneven terrain. The policy continuously adjusts joint torques and gait parameters based on real-time proprioceptive feedback (e.g., joint angles, IMU data) to compensate for slippery surfaces, slopes, or unexpected obstacles. This is essential for deployment in unstructured environments like construction sites or disaster zones.

When a robot arm trained in simulation attempts to grasp a real object, small variations in object weight, surface friction, or gripper alignment can cause failure. Online adaptation allows the controller to:

• Modify grip force in real-time based on tactile sensor slip detection.
• Adjust trajectory mid-execution if visual feedback shows the object has moved.
• Compensate for payload dynamics when lifting an object of unknown mass.

Self-driving systems face rapidly changing environmental conditions. Online adaptation is used to:

• Calibrate sensor models in real-time during sudden fog, rain, or harsh sunlight.
• Adjust object detection confidence thresholds based on observed noise levels in LiDAR or camera data.
• Fine-tune a perception model's batch normalization statistics to match the current visual domain, preventing degradation from domain shift.

Drones must adapt to wind gusts, aerodynamic disturbances from nearby structures, or changes in their own dynamics (e.g., after a minor collision). Online adaptation techniques:

• Update the parameters of a learned dynamics model in real-time using streaming sensor data.
• Employ adaptive control laws (e.g., Model Reference Adaptive Control) to ensure stable hover and trajectory tracking despite unknown disturbances.
• This allows for reliable operation in GPS-denied or windy environments.

In manufacturing, robot arms experience wear and tear, thermal expansion, and payload changes that degrade the accuracy of their pre-programmed kinematic model. Online adaptation systems:

• Continuously estimate joint stiffness and backlash parameters.
• Dynamically correct the inverse kinematics solution to maintain precise end-effector positioning over thousands of cycles.
• This reduces downtime for manual recalibration and maintains product quality.

Advanced systems use meta-learning frameworks like MAML to pre-train a policy with the explicit ability to adapt quickly. The policy is meta-trained in simulation across a distribution of tasks/dynamics. During real-world deployment:

• The policy performs a few steps of gradient-based adaptation using a short buffer of real experience.
• This rapidly tailors the policy to the specific dynamics of the physical hardware, achieving effective few-shot online adaptation.

A core pre-deployment training technique that exposes a policy to a vast range of randomized simulation parameters (e.g., textures, lighting, physics properties like mass and friction). This forces the policy to learn domain-invariant features, building robustness before real-world deployment. It is often a prerequisite for effective online adaptation.

• Dynamics Randomization: A specific subtype focusing on varying physical parameters like actuator latency, joint damping, and contact friction.
• Automatic Domain Randomization (ADR): An advanced version that automatically expands the randomization range to create a curriculum of increasingly difficult environments.

The process of building or calibrating a mathematical model of a physical system's dynamics from real-world input-output data. This is often performed online to improve the fidelity of the simulator used for adaptation.

• Provides precise parameters (e.g., inertial tensors, motor constants) to reduce the reality gap.
• Enables model-based adaptation, where an identified model is used for real-time planning or to generate synthetic training data for the policy.
• Contrasts with domain randomization's 'robustness through diversity' approach by seeking an 'accurate model' approach.

The process of taking a policy pre-trained in simulation and continuing its training with a limited amount of data collected from the real-world system. This is a foundational form of adaptation.

• Key Difference from Online Adaptation: Fine-tuning often implies a distinct, batch-based training phase with collected data, whereas online adaptation typically implies continuous, real-time adjustment during task execution.
• Risk: Naive fine-tuning can lead to catastrophic forgetting of robust behaviors learned in simulation if the real-world data is limited or noisy.

A broad machine learning subfield focused on adapting models from a source domain (simulation) to a different but related target domain (reality). Online adaptation is a specific, real-time instance of this. Key algorithmic families include:

• Feature Alignment: Minimizing distributional distance (e.g., using Maximum Mean Discrepancy - MMD or CORrelation ALignment - CORAL) between simulation and real data features.
• Adversarial Domain Adaptation: Using a domain classifier and a gradient reversal layer to learn domain-invariant features that confuse the classifier.
• Domain-Adversarial Neural Networks (DANN): A canonical architecture implementing adversarial domain adaptation.

A meta-learning algorithm that trains a model's initial parameters so it can adapt rapidly to new tasks with only a few gradient steps and a small amount of new data. This is highly relevant for few-shot online adaptation.

• The policy is meta-trained in simulation across a distribution of randomized environments.
• During real-world deployment, the policy uses online data to perform a few steps of gradient-based adaptation, effectively fine-tuning itself in real-time to the specific physical instance it encounters.
• Enables fast adaptation without forgetting core skills.

The fundamental challenge that makes adaptation necessary: the performance discrepancy caused by mismatches between simulation and reality. Sources include:

• Visual Discrepancy: Differences in lighting, textures, and sensor noise (e.g., camera vs. render).
• Dynamics Discrepancy: Inaccurate modeling of friction, actuator latency, motor backlash, and material deformation.
• Simulation Fidelity: The degree of accuracy of the simulator directly impacts the size of the initial reality gap.
• Bridging the Gap: The combined goal of domain randomization, system identification, and online adaptation is to minimize or bridge this gap for successful Sim-to-Real Transfer.