Trust Region Policy Optimization (TRPO)

Proximal Policy Optimization (PPO) is a successor algorithm to TRPO designed for greater simplicity and empirical performance. While TRPO uses a complex second-order optimization with a hard constraint on the KL divergence, PPO approximates this trust region using a clipped probability ratio in its objective function. This makes PPO easier to implement and tune, often achieving similar or better performance with first-order optimizers like Adam.

• Key Innovation: Replaces TRPO's constrained optimization with a clipped surrogate objective.
• Practical Impact: Became the de facto standard for policy gradient RL in domains like robotics and language model alignment (RLHF).

The Kullback-Leibler (KL) Divergence constraint is the mathematical mechanism that defines TRPO's trust region. It measures the difference between the probability distributions of the new policy and the old policy. By bounding this divergence, TRPO ensures policy updates are small and stable.

• Function: Acts as a step-size controller, preventing updates that could collapse performance.
• Formulation: The optimization is maximize expected reward subject to KL(old || new) ≤ δ.
• Impact: This constraint is the direct cause of TRPO's guaranteed monotonic improvement, though it requires computationally expensive second-order approximations.

The Natural Policy Gradient is an optimization method that forms the theoretical backbone of TRPO. Instead of using the standard gradient (which can be misleading in parameter space), it uses the natural gradient, which preconditions the gradient by the inverse of the Fisher Information Matrix. This accounts for the geometry of the policy space, leading to more direct updates.

• Relation to TRPO: TRPO can be viewed as a practical, approximate implementation of the natural policy gradient that enforces a trust region via a line search.
• Advantage: Provides invariance to parameterization, meaning the learning update is independent of how the policy is represented.

Actor-Critic methods are a foundational architecture in reinforcement learning where TRPO is typically applied. In this framework, the actor (the policy being optimized by TRPO) selects actions, while a separate critic (a value function) evaluates those actions. TRPO is a policy optimization algorithm that fits within the actor-critic paradigm.

• TRPO's Role: TRPO specifically optimizes the actor using the critic's value estimates to compute advantage functions.
• Synergy: The stability of TRPO's updates complements the value estimation of the critic, leading to more sample-efficient and reliable learning than pure policy gradient methods.

Monotonic improvement is the formal guarantee that TRPO was designed to provide. It ensures that each policy update is non-degrading, meaning the performance of the new policy is theoretically guaranteed to be at least as good as the old policy, given the trust region constraint is satisfied. This is a critical property for stable training in complex environments.

• Theoretical Basis: Derived from a lower bound on policy performance (the surrogate advantage).
• Practical Significance: Prevents the catastrophic performance collapses common in naive policy gradient methods, making TRPO suitable for high-stakes or expensive-to-simulate domains.

Model-Based Reinforcement Learning (MBRL) is a contrasting paradigm to model-free methods like TRPO. In MBRL, the agent learns an internal model of the environment dynamics and uses it for planning. TRPO is a model-free algorithm; it learns a policy directly from experience without building an explicit world model.

• Comparison: TRPO excels in settings where learning an accurate model is difficult, but can be less sample-efficient than MBRL when a good model is available.
• Hybrid Approaches: Modern research often combines trust region policy optimization with learned models to gain the sample efficiency of MBRL and the robustness of model-free policy search.