Actor-Critic

The Actor network parameterizes the policy π(a|s) and selects actions, while the Critic network estimates the value function V(s) or Q(s,a) to evaluate those actions. This separation allows the Actor to focus on action selection and the Critic to focus on accurate value estimation, reducing the variance inherent in pure policy gradient methods.

• Actor: Outputs action probabilities or continuous action values
• Critic: Outputs a scalar value estimate used to compute the advantage
• Shared layers: Often share early feature extraction layers to reduce parameters

The Critic enables computation of the advantage function A(s,a) = Q(s,a) - V(s), which measures how much better a specific action is compared to the average action in that state. By using the advantage instead of raw returns, the Actor receives a centered, lower-variance learning signal that dramatically stabilizes policy updates.

• Baseline subtraction: V(s) serves as a baseline to reduce gradient variance
• GAE integration: Generalized Advantage Estimation uses the Critic's value predictions to compute multi-step advantage estimates
• Temporal Difference error: δ = r + γV(s') - V(s) drives both Actor and Critic updates

The Critic learns via temporal difference (TD) learning, bootstrapping from its own value estimates of future states rather than waiting for episode completion. This enables online, step-by-step learning critical for continuous trading environments where episodes may be arbitrarily long or undefined.

• TD(0): Single-step bootstrapping for rapid updates
• TD(λ): Multi-step returns with eligibility traces for balancing bias-variance
• On-policy vs. off-policy: Actor-Critic can be adapted to either paradigm depending on whether the Critic evaluates the current or a target policy

Unlike value-based methods like DQN that require discretization, Actor-Critic architectures naturally handle continuous action spaces through the Actor's parameterized policy. For trading, this enables precise position sizing, limit order pricing, and execution scheduling without quantization error.

• Gaussian policies: Actor outputs mean and variance for continuous actions
• Deterministic variants: DDPG and TD3 output exact action values for stable control
• Stochastic exploration: Entropy regularization maintains exploration in continuous domains

Pure policy gradient methods like REINFORCE suffer from high variance due to Monte Carlo sampling of full episode returns. The Critic's value estimates provide a learned baseline that dramatically reduces gradient estimate variance without introducing bias, enabling faster convergence on optimal trading policies.

• Baseline property: E[∇log π · b(s)] = 0 ensures unbiased gradients
• State-dependent baseline: Critic provides per-state variance reduction
• Convergence speed: Typically 3-10x faster than vanilla policy gradient in trading environments

Modern Actor-Critic variants like Soft Actor-Critic (SAC) augment the objective with an entropy bonus that rewards the policy for maintaining stochasticity. This prevents premature convergence to suboptimal deterministic strategies, crucial in non-stationary financial markets where exploration must continue throughout training.

• Maximum entropy objective: J(π) = Σ E[r + αH(π(·|s))]
• Automatic temperature tuning: α can be learned to maintain target entropy levels
• Multi-modal policies: Entropy encourages discovering diverse profitable trading behaviors

The actor network outputs a continuous distribution over position sizes, enabling precise, fractional share allocation rather than discrete buy/sell/hold signals. This is critical for optimal execution algorithms where the agent must determine the exact number of shares to trade at each time step to minimize market impact. The critic network evaluates the Q-value of each potential allocation, providing a gradient signal that refines the sizing policy toward risk-adjusted return maximization.

Financial time series exhibit high noise-to-signal ratios, making pure policy gradient methods unstable. Actor-critic architectures compute the advantage function—the difference between the Q-value of a chosen action and the state value baseline—to dramatically reduce gradient variance. This allows the agent to distinguish whether a profitable trade resulted from skillful timing or random market drift, accelerating convergence in high-frequency time-series forecasting environments.

Markets transition between bull, bear, and sideways regimes, requiring agents to avoid premature policy collapse. Actor-critic methods like Soft Actor-Critic (SAC) incorporate entropy regularization directly into the objective function, rewarding the policy for maintaining stochasticity. This prevents the agent from overfitting to a single market condition and ensures it continues exploring alternative strategies when volatility surfaces shift unexpectedly.

Q-value overestimation is fatal in trading, where an agent may allocate excessive capital based on inflated value predictions. Twin Delayed DDPG (TD3) employs two independent critic networks, taking the minimum of their Q-value estimates to produce a conservative target. This architecture is particularly effective in portfolio optimization tasks where overestimating the value of a concentrated position can lead to catastrophic drawdowns.

Rather than using raw profit-and-loss, actor-critic trading agents can optimize the differential Sharpe ratio—an online, differentiable approximation of risk-adjusted returns. The critic learns to predict this metric, while the actor adjusts its policy to maximize it. This aligns the agent's objective with institutional performance benchmarks and naturally penalizes strategies with high volatility, addressing the exploration-exploitation trade-off in a risk-aware manner.

Traders never observe the full market state—hidden liquidity, latent intentions, and unobserved order flow create a Partially Observable MDP (POMDP). Actor-critic architectures can integrate recurrent layers or attention mechanisms into both networks, allowing the agent to maintain an implicit belief state over market conditions. The critic evaluates actions based on this inferred state, enabling the agent to trade effectively through information asymmetry.