On-Policy Learning

The fundamental mechanism of on-policy learning is that the agent evaluates and improves the same policy it uses to act. The data collected from the environment—the behavior policy—is used directly to update the target policy. This creates a closed loop where the policy's performance is assessed using trajectories it generated itself. For example, algorithms like SARSA use the action actually taken in the next state to update the Q-value for the current state-action pair.

On-policy methods often directly optimize a parameterized policy function π(a|s; θ). They use gradient ascent on the expected cumulative reward to adjust the parameters θ. The gradient is estimated from Monte Carlo returns or actor-critic estimates derived from on-policy trajectories. This approach is central to algorithms like REINFORCE, A2C/A3C, and Proximal Policy Optimization (PPO), which are staples in robotics and control for learning continuous action policies.

The strategy for exploration is intrinsically linked to the learning policy. The agent must explore using the same policy it is trying to evaluate, which means exploration mechanisms like ε-greedy or adding entropy regularization are baked directly into the policy. This differs from off-policy methods, where a separate, more exploratory behavior policy can be used. This coupling can make balancing exploration and exploitation more challenging, especially in sparse-reward environments common in robotics.

A key limitation is sample inefficiency. Since data can only be used to improve the policy that generated it, old data becomes obsolete as the policy updates. This often necessitates discarding data after a single update or a small number of updates, leading to high sample complexity. Furthermore, gradient estimates from on-policy trajectories can have high variance, which techniques like advantage normalization and Generalized Advantage Estimation (GAE) are designed to mitigate.

On-policy methods, particularly policy gradient algorithms, often provide stronger theoretical convergence guarantees to at least a local optimum when compared to some off-policy value-based methods. This is because the policy is updated in the direction of the true gradient of the performance objective, estimated from its own actions. Modern algorithms like PPO and TRPO introduce constraints (e.g., a trust region) to ensure updates are stable and monotonic, which is critical for safety-sensitive control applications.

The core distinction lies in the separation of policies. In off-policy learning (e.g., Q-Learning, DDPG, SAC), the agent learns about a target policy (often the optimal policy) using data generated by a different behavior policy. This allows for:

• Experience replay: Reusing old data from past policies.
• Greater sample efficiency.
• Learning from demonstrations or exploratory policies. On-policy learning sacrifices this flexibility for simpler, often more stable, direct policy optimization.

REINFORCE (Monte Carlo Policy Gradient) is a foundational policy gradient algorithm. It uses complete episode returns to estimate the policy gradient, making it a Monte Carlo method. Its key mechanism is:

• Collect a full trajectory under the current policy.
• Compute the return from each timestep to the end of the episode.
• Update the policy parameters via gradient ascent, scaling the gradient by the return.

Pros: Simple, guarantees policy improvement. Cons: High variance in gradient estimates, slow and sample-inefficient as it requires complete episodes before an update.

The Advantage Actor-Critic (A2C) algorithm reduces the variance of policy gradient updates by introducing a value function critic. The critic estimates the state-value function, which is used to compute the advantage function (A = Q - V). This advantage tells the actor how much better an action is than average.

Core Components:

• Actor: Policy network updated by the policy gradient, weighted by the advantage.
• Critic: Value network trained to minimize the TD error.

It is synchronous, typically collecting batches of experience from multiple parallel environments before performing a single, coordinated update.

Proximal Policy Optimization (PPO) is a dominant on-policy algorithm designed for stability and ease of use. Its primary innovation is a clipped surrogate objective that prevents destructively large policy updates.

The algorithm works by:

1. Collecting data with the current policy.
2. Computing the probability ratio between the new and old policy for each action.
3. Maximizing a clipped objective: min(ratio * advantage, clip(ratio, 1-ε, 1+ε) * advantage).

This clipping ensures updates are proximal, maintaining trust in the new policy. PPO often uses multiple epochs of minibatch updates on the same dataset, improving sample efficiency.

Trust Region Policy Optimization (TRPO) is the theoretical predecessor to PPO. It rigorously constrains policy updates using a trust region, enforced via a KL divergence constraint between the old and new policy distributions. This guarantees monotonic improvement.

Key Mechanism: It solves a constrained optimization problem at each step:

• Maximize the expected advantage (the surrogate objective).
• Subject to: Average KL divergence between policies ≤ δ (a small threshold).

TRPO provides stronger theoretical guarantees than PPO but is computationally more complex, requiring conjugate gradient methods to approximate the natural policy gradient.

Asynchronous Advantage Actor-Critic (A3C) is a parallelized variant of A2C designed for CPU-based training. Instead of synchronous updates, multiple worker agents operate in parallel, each with its own instance of the environment and a local copy of the global network.

Process:

1. Each worker interacts with its environment asynchronously.
2. After a set number of steps, it computes gradients based on its local experience.
3. It asynchronously pushes these gradients to update a global shared network.
4. It pulls the latest parameters from the global network.

This architecture eliminates the need for a replay buffer and can lead to more diverse exploration.

SARSA (State-Action-Reward-State-Action) is a classic tabular and temporal-difference (TD) on-policy control algorithm. It learns the action-value function Q(s,a) for the policy it is following.

The update rule is: Q(S_t, A_t) ← Q(S_t, A_t) + α [R_{t+1} + γ * Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t)]

Key Characteristics:

• On-Policy: It uses the action (A_{t+1}) actually selected by its current policy (e.g., ε-greedy) for the bootstrap target.
• Contrast with Q-Learning: Q-Learning uses max_a Q(S_{t+1}, a) (off-policy), while SARSA uses the next action from its policy, making it more conservative in risky environments (e.g., near cliffs in grid worlds).

Off-Policy Learning is a reinforcement learning paradigm where an agent learns the value of a target policy (often the optimal one) while following a different behavior policy to collect experience. This decoupling allows for greater data efficiency and the reuse of past experiences.

• Key Mechanism: Uses importance sampling or other correction techniques to learn from data generated by a different policy.
• Primary Benefit: Enables the use of large experience replay buffers filled with historical data, which is critical for sample-efficient learning in robotics.
• Common Algorithms: Q-Learning, Deep Q-Networks (DQN), and Soft Actor-Critic (SAC) are classic off-policy methods.

Policy Gradient Methods are a class of algorithms that directly optimize a parameterized policy function using gradient ascent on the expected cumulative reward. They form the foundation for most modern on-policy algorithms.

• Direct Optimization: They adjust the policy parameters θ to increase the probability of actions that lead to higher reward.
• On-Policy Nature: Standard policy gradient updates (e.g., REINFORCE) require fresh samples from the current policy, making them inherently on-policy.
• Stability vs. Efficiency: Pure policy gradients can have high variance; advanced methods like PPO and TRPO introduce constraints to stabilize these on-policy updates.

Proximal Policy Optimization (PPO) is a dominant on-policy algorithm that uses a clipped surrogate objective to constrain policy updates, preventing destructively large changes and enabling stable, reliable training.

• Clipped Objective: The core innovation is a objective function that penalizes policy changes that would move too far from the previous policy, ensuring monotonic improvement.
• Workhorse Algorithm: Due to its simplicity and robustness, PPO is a standard choice for training control policies in simulated environments like MuJoCo and for real-world robotic tasks.
• Implementation: Commonly available in libraries like Stable-Baselines3 and Ray RLlib.

Trust Region Policy Optimization (TRPO) is an on-policy precursor to PPO that more rigorously enforces a trust region constraint on policy updates using conjugate gradient and Fisher information matrix calculations.

• Theoretical Guarantee: It optimizes a surrogate objective subject to a hard constraint on the KL-divergence between the new and old policy, theoretically ensuring monotonic improvement.
• Computational Cost: The constraint enforcement is more computationally intensive than PPO's clipping heuristic.
• Niche Use: While largely superseded by PPO in practice, TRPO's theoretical foundations are critical for understanding constrained on-policy optimization.

The Actor-Critic Architecture is a framework that combines a policy network (the actor) with a value network (the critic). It is the standard structure for modern on-policy and off-policy algorithms.

• Actor: The parameterized policy that selects actions. Updated via policy gradient.
• Critic: Estimates the value function (e.g., state-value V(s) or action-value Q(s,a)), providing a lower-variance baseline for the actor's updates.
• On-Policy Critic: In on-policy methods like PPO and A3C, the critic is trained exclusively on data from the current actor policy to accurately evaluate its performance.

The Exploration-Exploitation Tradeoff is the fundamental dilemma where an agent must balance trying new actions to gather information (exploration) with choosing known rewarding actions (exploitation). On-policy methods handle this tradeoff intrinsically.

• On-Policy Exploration: The agent's exploration strategy is defined by its current policy (e.g., via entropy regularization in the objective). To explore differently, the policy itself must change.
• Contrast with Off-Policy: Off-policy agents can maintain an exploratory behavior policy (e.g., epsilon-greedy) while learning a deterministic target policy.
• Critical for Control: In robotics, safe and structured exploration is paramount, often requiring carefully tuned on-policy entropy bonuses or noise injection.