On-Policy vs. Off-Policy Learning

On-policy learning evaluates and improves the exact same policy (the decision-making function) that is being used to interact with the environment and generate data. The agent learns from its own current behavior.

Off-policy learning evaluates and improves a target policy (the one we want to optimize) using data generated by a different behavior policy. This allows learning from exploratory, historical, or expert-generated data.

In on-policy methods, exploration is tightly coupled with policy improvement. The agent must explore using its current policy (e.g., via epsilon-greedy). As the policy improves, its exploration behavior changes.

Off-policy methods decouple exploration from learning. A highly exploratory or even random behavior policy (like a human demonstrator) can gather data, while the target policy is optimized for exploitation. This enables techniques like experience replay.

On-policy algorithms (e.g., REINFORCE, A2C, PPO) typically use each data sample once and discard it, as it becomes outdated when the policy changes. This can be less sample-efficient but often more stable.

Off-policy algorithms (e.g., Q-Learning, DDPG, SAC) can reuse past experiences stored in a replay buffer. This improves sample efficiency but introduces challenges like distributional shift, where the data in the buffer no longer matches the current policy's state distribution.

On-Policy Algorithms:

• SARSA: Learns the Q-value for the policy it is following.
• Proximal Policy Optimization (PPO): Uses a clipped objective to ensure updates stay close to the current policy.
• A3C/A2C: Asynchronous advantage-based policy gradient methods.

Off-Policy Algorithms:

• Q-Learning/DQN: Directly learns the optimal Q-function, independent of the behavior policy.
• Deep Deterministic Policy Gradient (DDPG): An actor-critic method for continuous action spaces.
• Soft Actor-Critic (SAC): Maximizes both reward and entropy for robust exploration.

Use on-policy learning when:

• You need stable, monotonic policy improvement.
• The cost of interaction is low, or you can generate new data easily (e.g., in high-fidelity simulators).
• The policy's exploration behavior must be explicitly controlled and updated.

Use off-policy learning when:

• Data collection is expensive, risky, or limited, requiring reuse of past experiences.
• You want to learn from pre-recorded datasets (offline RL).
• You need to learn from demonstrations or other agents (a form of imitation learning).

The distinction is rooted in the Bellman equation used for updates.

On-policy updates (like in SARSA) use the action actually taken by the current policy in the next state: Q(s, a) ← Q(s, a) + α [r + γ * Q(s', a') - Q(s, a)] where a' is from the policy.

Off-policy updates (like in Q-Learning) use the maximum valued action in the next state, regardless of what the behavior policy did: Q(s, a) ← Q(s, a) + α [r + γ * max_a' Q(s', a') - Q(s, a)]. This directly approximates the optimal Q-function.

A class of on-policy algorithms that optimize an agent's policy directly by ascending the gradient of expected reward with respect to the policy parameters. These methods, like REINFORCE, update the policy using trajectories generated by the current policy itself.

• Key Mechanism: Directly adjusts policy parameters to increase the probability of high-reward actions.
• On-Policy Nature: Requires fresh samples from the current policy for each update, making it less sample-efficient than off-policy methods.

A foundational model-free, off-policy algorithm that learns the action-value function (Q-function). It estimates the expected return for taking an action in a state and following the optimal policy thereafter.

• Off-Policy Core: Learns the value of the optimal policy (target policy) while using data generated by an exploratory behavior policy (e.g., epsilon-greedy).
• Update Rule: Uses the Bellman equation to iteratively improve Q-value estimates, decoupling the learning policy from the behavior policy.

A hybrid architecture that combines a policy network (Actor) and a value network (Critic). The actor selects actions, and the critic evaluates those actions by estimating the value function, providing a lower-variance learning signal.

• Can be On or Off-Policy: Implementations vary. Advantage Actor-Critic (A2C) is typically on-policy, while methods using a replay buffer (like some DDPG variants) incorporate off-policy learning.
• Core Benefit: The critic reduces the variance of policy gradient updates, leading to more stable training than pure policy gradient methods.

A critical technique that enables off-policy learning by storing an agent's experiences (state, action, reward, next state) in a buffer and later sampling random mini-batches for training.

• Breaks Temporal Correlation: Random sampling decorrelates sequential experiences, improving learning stability.
• Improves Sample Efficiency: Allows the reuse of past experiences multiple times, which is essential for data-efficient algorithms like Deep Q-Networks (DQN).

The fundamental dilemma where an agent must balance trying new actions to discover their effects (exploration) with choosing actions known to yield high rewards (exploitation). This tradeoff directly influences the design of the behavior policy.

• On-Policy Impact: The current policy must itself explore, often through stochasticity.
• Off-Policy Impact: The behavior policy (e.g., epsilon-greedy) handles exploration separately from the target policy being learned, providing more flexibility.

A central idea in reinforcement learning where an agent learns by bootstrapping—updating its estimate of a value based on other estimates. TD learning is the foundation for both on-policy and off-policy algorithms.

• On-Policy Example: SARSA uses TD learning to update the Q-values for the policy it is currently following.
• Off-Policy Example: Q-Learning uses TD learning to update towards the optimal Q-value, regardless of the action taken.
• Core Concept: Updates are made using the TD error, the difference between the current estimate and a more informed target estimate.