Bellman Equation

The Bellman equation expresses the value of a state as the immediate reward plus the discounted expected value of the subsequent state. This recursive structure breaks an infinite-horizon sequential decision problem into a single-step optimization, enabling dynamic programming solutions. For a state s, the value function satisfies:

• V(s) = max_a [ R(s,a) + γ Σ P(s'|s,a) V(s') ]
• The equation assumes the agent follows the optimal policy thereafter
• This self-referential property is what makes bootstrapping possible in temporal difference learning

The Bellman equation formalizes the Principle of Optimality: an optimal policy has the property that whatever the initial state and decision, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. This means:

• Truncating an optimal trajectory at any point still yields an optimal sub-trajectory
• The value function V*(s) is the unique fixed point of the Bellman optimality operator
• Any policy that is greedy with respect to V* is guaranteed to be optimal

The Bellman operator is a contraction mapping in the space of value functions with respect to the max-norm. This mathematical property guarantees convergence of iterative solution methods:

• Applying the Bellman operator repeatedly shrinks the distance to the true value function by a factor of γ (the discount factor) at each iteration
• Value iteration converges to the unique fixed point regardless of the initial value function estimate
• The contraction property underpins the convergence proofs for Q-learning and fitted value iteration

The Bellman expectation equation describes the value of following a specific policy π, rather than the optimal policy. It replaces the max operator with an expectation over the policy's action distribution:

• V^π(s) = Σ_a π(a|s) [ R(s,a) + γ Σ_{s'} P(s'|s,a) V^π(s') ]
• This linear form enables policy evaluation through solving a system of linear equations
• The expectation form is the foundation for policy iteration, which alternates between policy evaluation and policy improvement until convergence to the optimal policy

The Bellman equation extends naturally to action-value functions Q(s,a), which evaluate state-action pairs rather than states alone. This variant is the direct foundation of model-free algorithms like Q-learning:

• Q*(s,a) = R(s,a) + γ Σ_{s'} P(s'|s,a) max_{a'} Q*(s',a')
• The Q-form eliminates the need for a transition model when selecting actions — the agent simply chooses the action with the highest Q-value
• Deep Q-Networks (DQN) approximate this function using neural networks trained to minimize the Bellman residual

The discount factor γ ∈ [0,1) serves multiple critical functions in the Bellman equation beyond simple time preference:

• Mathematical: Ensures the contraction property holds, guaranteeing convergence of iterative methods
• Economic: Represents the opportunity cost or uncertainty about future rewards in financial applications
• Practical: Values closer to 0 produce myopic agents focused on immediate rewards; values near 1 create far-sighted strategies
• In trading, γ is often calibrated to match the agent's investment horizon, with typical values between 0.95 and 0.99 for daily decision frequencies

The formal mathematical framework for modeling sequential decision-making in stochastic environments. An MDP is defined by a tuple (S, A, P, R, γ) where S is the state space, A is the action space, P(s'|s,a) is the state transition probability, R(s,a) is the reward function, and γ is the discount factor. The Bellman Equation provides the recursive decomposition of value functions within this framework.

• States must satisfy the Markov property: the future depends only on the current state, not the history
• In trading, states might include current position, portfolio weights, and market features
• The agent's goal is to find a policy π(a|s) that maximizes expected cumulative discounted reward

The TD error is the core learning signal that drives value function updates in reinforcement learning. It measures the difference between the current value estimate and a better estimate formed using an observed reward plus the discounted value of the next state.

• Formula: δ = R(s,a) + γV(s') - V(s)
• This is a bootstrapping method: it updates estimates based on other estimates
• The Bellman Equation defines the target that TD learning aims to achieve
• In trading, a large positive TD error indicates the agent received a better-than-expected return from an action

A model-free, off-policy temporal difference learning algorithm that directly learns the optimal action-value function Q*(s,a) without requiring a model of the environment. The Q-learning update rule is a direct application of the Bellman optimality equation:

• Update: Q(s,a) ← Q(s,a) + α[R + γ max_a' Q(s',a') - Q(s,a)]
• Off-policy: learns the optimal policy while following an exploratory behavior policy
• Converges to Q* under conditions of sufficient exploration and appropriate learning rate decay
• Forms the foundation for Deep Q-Networks used in discrete-action trading environments

The advantage function A(s,a) quantifies how much better a specific action is compared to the average action in a given state. It decomposes the action-value function using the Bellman Equation structure:

• Definition: A(s,a) = Q(s,a) - V(s)
• Positive advantage means the action is better than average; negative means worse
• Reduces variance in policy gradient estimation by subtracting the state baseline
• In trading, advantage indicates whether a buy/sell decision outperformed the expected portfolio value
• Central to Actor-Critic architectures and Generalized Advantage Estimation (GAE)

An extension of the MDP framework where the agent cannot directly observe the full environmental state. Instead, the agent receives observations that provide partial, noisy information about the true underlying state. The Bellman Equation generalizes to operate over belief states—probability distributions over possible states.

• Defined by the tuple (S, A, O, P, R, Z, γ) where Z is the observation function
• The agent must maintain and update a belief state using Bayesian filtering
• Financial markets are inherently partially observable: you cannot see all order book depth or other participants' intentions
• Solving POMDPs exactly is computationally intractable; approximate methods are required

The practice of engineering auxiliary reward signals to guide the agent toward desired behaviors when the environment's natural reward is sparse or delayed. This modifies the effective reward function while preserving the optimal policy if done correctly.

• Potential-based shaping: F(s,s') = γΦ(s') - Φ(s) guarantees policy invariance
• In trading, shaping rewards might include the Differential Sharpe Ratio or penalties for excessive turnover
• Must be carefully designed to avoid reward hacking, where the agent exploits the shaped reward without achieving the true objective
• Directly alters the right-hand side of the Bellman Equation's recursive decomposition