Online Fine-Tuning

Unlike one-time deployment, online fine-tuning involves continuous, incremental learning from a live stream of data. The policy updates its parameters in real-time or near-real-time based on new experiences, allowing it to adapt to unmodeled dynamics, wear and tear, or changing environmental conditions. This creates a closed-loop learning system where performance improves over the operational lifetime of the robot.

Data is collected on-policy from the target environment (the real robot). This introduces major safety constraints:

• Safe Exploration: The policy must explore cautiously to gather informative data without causing damage or unsafe states. Techniques like action clipping, reward shaping for safety, and probabilistic ensembles to estimate uncertainty are critical.
• Human-in-the-Loop (HIL): Often requires human supervision to reset the robot from failures or provide corrective demonstrations, making it a data-expensive and labor-intensive process.

Online fine-tuning demands algorithms with specific properties:

• Sample Efficiency: Must learn effectively from limited real-world data due to the high cost of physical interaction.
• Stability: Updates must avoid catastrophic forgetting of useful behaviors learned in simulation. Algorithms like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are favored for their stable update rules.
• Off-Policy Capability: The ability to learn from a replay buffer of past experiences is crucial for reusing scarce real-world data.

This process directly addresses the sim-to-real gap—the discrepancy between the simulated training environment and the real world. Discrepancies occur in:

• Dynamics: Friction, motor backlash, and cable tension not perfectly modeled.
• Perception: Lighting variations, sensor noise, and camera distortions.
• Actuation: Delays and non-linearities in motor controllers. Online fine-tuning bridges this gap by allowing the policy to correct for these systematic errors through direct experience.

Requires a robust MLOps-style pipeline on the robot or a connected edge server:

• Inference Engine: Runs the policy network with low latency.
• Data Logger: Records state-action-reward-next_state tuples.
• Training Loop: Periodically retrains the policy using aggregated data, often during robot downtime.
• Model Versioning & Rollback: Essential for reverting to a stable policy if fine-tuning degrades performance. This contrasts with offline fine-tuning, which uses a pre-collected static dataset.

Online fine-tuning is a form of target-domain adaptation. Related techniques used to reduce the required online data include:

• Domain Randomization: Training in simulation with randomized parameters (e.g., textures, masses) to create a robust policy that requires less fine-tuning.
• System Identification: Briefly characterizing the real robot's dynamics to adjust the simulation model before policy transfer, narrowing the initial gap.
• Meta-Learning: Training a policy in simulation that is explicitly initialized for fast adaptation, requiring fewer gradient steps online.

A broader machine learning field where a model trained on a source domain (e.g., simulation) is adapted to perform well on a different but related target domain (e.g., reality). Online fine-tuning is a specific, interactive form of domain adaptation for RL policies. Key methods include:

• Feature alignment: Minimizing the discrepancy between simulated and real feature distributions.
• Adversarial training: Using a domain classifier to learn domain-invariant representations.
• Self-supervised learning: Leveraging unlabeled real-world data for pre-training.

The process of building or calibrating a mathematical model of a physical system's dynamics using observed input-output data. For sim-to-real, system ID is often performed offline before fine-tuning to reduce the reality gap. It involves:

• Estimating physical parameters (e.g., friction coefficients, motor constants, link masses).
• Fitting the transition function of the simulation to match real-world rollouts.
• Providing a more accurate digital twin as a starting point for online policy adaptation.

A fundamental algorithmic distinction in RL that dictates how online fine-tuning is performed.

• On-Policy Learning (e.g., PPO): The agent learns from data collected by its current policy. Online fine-tuning is inherently on-policy for the real-world data stream.
• Off-Policy Learning (e.g., SAC, DDPG): The agent can learn from data collected by past policies stored in a replay buffer. This allows more efficient reuse of expensive real-world data during fine-tuning. Hybrid approaches use a replay buffer of real-world data for stable off-policy updates while collecting new on-policy data.

The tendency of a neural network to abruptly and drastically forget previously learned information when trained on new data. This is a major risk in online fine-tuning, where the policy must adapt to the real world without losing the robust skills acquired in simulation. Mitigation strategies include:

• Elastic Weight Consolidation (EWC): Penalizes changes to network weights deemed important for prior tasks.
• Experience Replay: Interleaving batches of old simulation data with new real-world data during training.
• Progressive Networks: Using a frozen simulation-trained network as a feature extractor for a new, adaptable network.

The set of constraints and algorithms designed to prevent a policy from taking dangerous actions during online fine-tuning on physical hardware. Since real robots can break themselves or their surroundings, exploration must be bounded. Techniques include:

• Constrained RL: Formulating the problem with explicit safety constraints on states or actions.
• Action Scaling / Damping: Limiting the maximum torque or velocity of actuator commands.
• Recovery Policies: Deploying a backup, conservative policy if the fine-tuning agent enters an unsafe state region.
• Simulated Rollout Verification: Checking proposed actions in the digital twin before execution.

The extreme end of online fine-tuning where policy updates occur at control-loop frequencies (e.g., 100Hz+). This is distinct from episodic fine-tuning where data is collected, then a batch of training is performed. Real-time learning aims for lifelong adaptation to changing conditions. It presents unique challenges:

• Compute Constraints: Policy updates must be extremely lightweight to run on embedded hardware.
• Stability: Rapid weight changes can lead to unstable, oscillatory control.
• Data Efficiency: Must learn from very few samples per update cycle. Often relies on meta-learned or adaptive controllers that can adjust via small parameter shifts.