Policy Gradient

Policy gradient methods directly optimize the policy function π(a|s; θ) by adjusting parameters θ to maximize expected return, bypassing the need to learn a value function as an intermediary. This direct approach allows the policy to represent stochastic strategies where actions are sampled from a probability distribution, enabling smooth exploration in continuous action spaces like position sizing. The policy can be any differentiable function approximator, from a simple softmax for discrete actions to a Gaussian distribution for continuous control, providing flexibility across diverse trading scenarios.

The theoretical foundation of these methods is the policy gradient theorem, which expresses the gradient of expected return as:

∇J(θ) = E[∇log π(a|s; θ) · Q(s,a)]

This formulation eliminates the need to differentiate through the environment dynamics, requiring only the gradient of the log-probability of the selected action weighted by its quality. In practice, the true Q-value is often replaced with a sample return or advantage estimate to reduce variance, leading to variants like REINFORCE and actor-critic architectures.

Raw policy gradient estimates suffer from high variance, making learning unstable. Key mitigation strategies include:

• Baseline subtraction: Subtracting a state-dependent baseline b(s) from the return reduces variance without introducing bias, as E[∇log π · b(s)] = 0
• Advantage function: Using A(s,a) = Q(s,a) - V(s) instead of raw returns centers the signal around zero
• Generalized Advantage Estimation (GAE): Computes advantage as an exponentially-weighted average of n-step TD errors, trading off bias and variance through the λ parameter

These techniques are critical for stable training in noisy financial environments where reward signals exhibit high volatility.

Policy gradient methods excel in continuous action domains where value-based approaches like DQN require discretization. The policy typically outputs parameters of a probability distribution:

• Gaussian policies: Output mean μ(s) and standard deviation σ(s) for each action dimension, sampling actions from N(μ, σ²)
• Beta distributions: Bounded support on [0,1] useful for portfolio weight allocation
• Squashed Gaussian: Used in SAC with tanh transformation to enforce action bounds

For trading, this enables precise position sizing—an agent can output a continuous value representing the fraction of capital to allocate rather than selecting from coarse discrete buckets.

The REINFORCE algorithm represents the simplest policy gradient implementation, using Monte Carlo returns to update policy parameters after complete episodes:

θ ← θ + α · G_t · ∇log π(a_t|s_t; θ)

Where G_t is the total discounted return from timestep t. While unbiased, REINFORCE suffers from high variance due to trajectory randomness. In trading contexts, this manifests as unstable learning when individual episodes exhibit extreme profit or loss scenarios. Modern implementations augment REINFORCE with baselines and advantage estimation, evolving into the actor-critic family of algorithms.

Advanced policy gradient methods constrain updates to prevent catastrophic policy collapse from overly large parameter changes:

• TRPO (Trust Region Policy Optimization): Enforces a KL-divergence constraint between old and new policies, solving a constrained optimization problem at each update
• PPO (Proximal Policy Optimization): Simplifies TRPO by using a clipped surrogate objective that penalizes probability ratios deviating from 1, achieving similar stability with first-order optimization

These trust region methods are particularly valuable in financial applications where a single bad policy update could theoretically learn to place ruinously large bets before the next evaluation cycle.

A hybrid framework that directly addresses the high variance of pure policy gradients. The actor (policy network) selects actions, while the critic (value network) evaluates them. The critic provides a learned baseline, reducing gradient variance without introducing bias. In trading, the actor might decide to buy or sell, while the critic estimates the expected profit of that action given the current order book state.

A function defined as A(s,a) = Q(s,a) - V(s), quantifying how much better a specific action is compared to the average action in a given state. Using the advantage instead of raw returns in policy gradients significantly reduces variance. For a trading agent, a positive advantage indicates a buy order was superior to the current portfolio's expected baseline return.

A technique that computes the advantage as an exponentially-weighted average of multi-step temporal difference errors. Controlled by the λ (lambda) parameter, GAE smoothly interpolates between high-bias Monte Carlo estimates (λ=1) and high-variance one-step TD estimates (λ=0). This is critical in financial environments with noisy reward signals, allowing stable learning of long-horizon trading strategies.

A policy gradient method that constrains updates to a trust region using a clipped surrogate objective. PPO prevents destructively large parameter changes that could collapse trading performance. It is the dominant algorithm in production trading systems due to its simplicity and robustness. The clipping mechanism ensures a new policy does not deviate too far from the old one, avoiding catastrophic forgetting in volatile markets.

An online, differentiable approximation of the Sharpe ratio used as a direct reward signal. Instead of optimizing for raw profit, the agent maximizes risk-adjusted returns during training. This aligns the RL objective with the actual mandate of portfolio managers. The differential form allows gradient flow through the reward calculation, enabling end-to-end optimization of the strategy's risk-return profile.

A technique that adds a bonus reward proportional to the entropy of the policy distribution. This encourages the agent to maintain stochasticity and explore diverse action trajectories rather than collapsing to a deterministic strategy prematurely. In trading, entropy regularization prevents the agent from overfitting to a single market regime, preserving the ability to adapt when volatility regimes shift unexpectedly.