On-Policy Learning

In on-policy learning, the agent's behavior policy (used to collect data) and its target policy (being evaluated and improved) are identical. This creates a tight coupling where:

• Every policy update is immediately reflected in subsequent data collection.
• The agent learns the value of the policy it is actually following, not a hypothetical optimal one.
• This is in direct contrast to off-policy methods like Q-learning, which can learn about a greedy target policy while following an exploratory behavior policy.

The fundamental constraint of on-policy algorithms is their reliance on data generated by the current policy. This means:

• Experience replay buffers from past policies cannot be reused, as they would introduce bias.
• After each policy update, old data becomes obsolete and must be discarded.
• Training typically proceeds by collecting a batch of new trajectories, performing an update, and then discarding that data to collect more. This requirement often leads to lower sample efficiency compared to off-policy methods but provides more stable and predictable learning dynamics.

Since exploration must come from the policy itself, on-policy methods typically use stochastic policies. The policy outputs a probability distribution over actions, ensuring inherent exploration.

• Common implementations use a Gaussian distribution for continuous actions or a softmax distribution for discrete actions.
• The degree of exploration is controlled by parameters like the variance of the Gaussian or the temperature of the softmax.
• Algorithms like Proximal Policy Optimization (PPO) and REINFORCE explicitly manage this exploration entropy to balance learning progress with sufficient state-space coverage.

On-policy methods are renowned for their training stability but often at the cost of requiring more environmental interactions.

• Stability: By updating based on the current policy's performance, updates are less likely to diverge catastrophically. Techniques like PPO's clipping further enforce stable, conservative updates.
• Sample Intensity: The inability to reuse old data makes these methods less data-efficient. For real-world robotics, this often makes them more suitable for simulation-based training where millions of samples can be generated cheaply and in parallel, rather than for direct online learning on physical hardware.

Proximal Policy Optimization (PPO) is the dominant modern on-policy algorithm, particularly in robotics and simulation. Its key mechanism is a clipped surrogate objective that prevents destructively large policy updates.

• It collects a batch of data under the current policy.
• It computes the probability ratio between the new and old policy for each action.
• The objective is clipped to constrain this ratio, ensuring the new policy does not stray too far from the old one.
• This simple yet effective mechanism has made PPO a default choice for complex sim-to-real transfer pipelines, where stable and reliable training in simulation is paramount.

On-policy learning is exceptionally well-suited for simulation-based reinforcement learning, which is the foundation of modern robotic training.

• Massive Parallelization: Thousands of simulated environments can run in parallel, generating the vast amounts of fresh on-policy data required, overcoming the sample efficiency limitation.
• Deterministic Rollouts: In simulation, policies can be evaluated precisely, and data collection is fast and cost-free, perfectly matching the on-policy paradigm's needs.
• Stable Policy Development: The reliable convergence of algorithms like PPO allows researchers to develop complex policies for locomotion or manipulation in simulation before attempting zero-shot transfer or online fine-tuning on real hardware.

REINFORCE (Monte Carlo Policy Gradient) is the foundational on-policy algorithm. It uses complete episode trajectories to compute the gradient of expected reward.

• Mechanism: Updates policy parameters by ascending the gradient of the log-probability of taken actions, weighted by the total return from that state.
• Characteristics: High variance due to Monte Carlo returns, often requiring many episodes for stable learning. It is a pure policy gradient method without a value function critic.
• Use Case: Serves as a conceptual building block; modern variants like PPO build upon its core idea with stabilization techniques.

Synchronous Advantage Actor-Critic (A2C) is the synchronous, often more efficient, counterpart to A3C.

• Mechanism: Multiple workers collect trajectories in parallel, but all workers synchronize their gradients before a single, averaged update is applied to the global network.
• Advantage over A3C: Eliminates the potential for stale policy parameters inherent in fully asynchronous updates, often leading to more stable and reproducible training.
• Practical Use: Frequently preferred in modern implementations where parallel simulation is available (e.g., in a robotics lab with multiple simulation instances).

The choice between on-policy and off-policy algorithms (like DDPG, SAC, TD3) involves fundamental engineering trade-offs critical for robotics.

• On-Policy (PPO, TRPO):
  • Pro: Typically more stable and predictable during training. Directly optimizes the policy being executed.
  • Con: Lower sample efficiency; all training data must be discarded after each policy update.
• Off-Policy (SAC, DDPG):
  • Pro: Higher sample efficiency; can reuse data from a replay buffer collected by older policies.
  • Con: Can be less stable, with risks of divergence. The decoupling of behavior and target policies introduces complexity.
• Sim-to-Real Implication: On-policy methods are often favored in simulation where data is cheap to generate in parallel, prioritizing training stability for reliable transfer.

Policy Gradient Methods are a class of algorithms that directly optimize a parameterized policy function by ascending the gradient of expected reward. They are the foundation of most modern on-policy algorithms.

• Direct Optimization: They adjust policy parameters (θ) to increase the probability of high-reward actions.
• On-Policy Nature: Classic REINFORCE and vanilla policy gradients are inherently on-policy; they estimate the gradient using trajectories collected under the current policy.
• High-Dimensional/Continuous Actions: Excel in spaces where valuing every action (as in Q-learning) is infeasible. Advanced methods like PPO and TRPO build on this core to enable stable, large-scale deep RL.

Sample Efficiency measures the number of environment interactions an agent requires to achieve a given performance level. It is a critical metric, especially in robotics where real-world data is costly.

• On-Policy Trade-off: Pure on-policy methods are often less sample-efficient than off-policy methods because they discard data after a single update.
• Simulation Leverage: This trade-off makes on-policy algorithms like PPO ideal for sim-to-real transfer, where massive, parallelized simulation can generate the required volume of on-policy data cheaply.
• Algorithm Choice: The decision between on-policy and off-policy often hinges on the cost of data collection versus the need for data reuse.