Dynamic Programming

The Bellman Equation is the recursive definition that formalizes the value of a state as the immediate reward plus the discounted value of the optimal future state. In portfolio theory, it expresses the trade-off between current consumption and future wealth.

• Value Function V(s): The maximum expected utility achievable from state s.
• Recursive Decomposition: V(s) = max [ R(s, a) + γ * V(s') ]
• Curse of Dimensionality: The computational bottleneck that arises when the state space (e.g., number of assets) grows exponentially.

The Principle of Optimality states that an optimal policy has the property that whatever the initial state and decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.

• Truncation Invariance: The tail of an optimal trajectory is itself optimal.
• Backward Induction: Solving a multi-period portfolio problem by starting at the terminal date and recursively stepping backward to the present.
• Path Independence: The optimal decision at time t depends only on the current state, not on the history of how you arrived there.

Unlike divide-and-conquer methods, DP exploits overlapping sub-problems—identical sub-problems that are solved multiple times. The algorithm stores these solutions to avoid redundant computation.

• Memoization: A top-down caching technique that stores the results of expensive function calls.
• Tabulation: A bottom-up approach that fills a table iteratively based on previously computed values.
• State Aggregation: In continuous-state portfolio problems, discretizing the state space (e.g., wealth grid) to make the problem computationally tractable.

Dynamic Programming provides the exact solution method for Markov Decision Processes (MDPs) with known transition probabilities. The MDP formalizes the environment for sequential asset allocation.

• State (S): The current wealth and market regime.
• Action (A): The portfolio weights allocated to each asset.
• Transition Probability (P): The probability of moving from one market state to another.
• Reward Function (R): The utility derived from consumption or terminal wealth.
• Policy (π): A mapping from states to actions that the DP algorithm optimizes.

Two fundamental DP algorithms for finding the optimal policy in a known MDP. They differ in computational efficiency and convergence properties.

• Value Iteration: Starts with an arbitrary value function and iteratively applies the Bellman optimality operator until convergence. It is robust but can be slow.
• Policy Iteration: Alternates between evaluating a fixed policy (solving linear equations) and improving the policy by acting greedily. It converges in fewer iterations but each iteration is computationally heavier.
• Howard's Policy Improvement: The theorem guaranteeing that a greedy policy with respect to the value function is strictly better unless the policy is already optimal.

The continuous-time analog of the Bellman equation, the HJB equation is a partial differential equation (PDE) central to stochastic control problems like Merton's portfolio problem.

• Merton's Problem: The classic application of HJB to determine optimal consumption and investment in continuous time.
• Ito's Lemma: A stochastic calculus tool required to derive the HJB equation from the dynamics of wealth.
• Viscosity Solutions: A generalized solution concept for the HJB equation when classical smooth solutions do not exist, ensuring the framework remains mathematically rigorous.

The recursive optimality condition that defines the value of a state as the immediate reward plus the discounted value of the best possible future state. In portfolio optimization, it decomposes a multi-period allocation problem into a sequence of single-period trade-offs.

• State variables: Current wealth, asset positions, market regime
• Control variables: Portfolio weights, rebalancing actions
• Value function V(s) captures expected utility-to-go from state s
• Curse of dimensionality: Computational cost grows exponentially with state space size

The formal framework underlying dynamic programming, defined by states, actions, transition probabilities, and rewards. Financial markets are modeled as MDPs where the Markov property holds: future states depend only on the current state, not the path taken to reach it.

• State space: Discretized market conditions and portfolio attributes
• Action space: Buy, sell, hold decisions with continuous sizing
• Transition kernel: Stochastic model of asset price evolution
• Reward function: Utility of consumption or terminal wealth

An extension of deterministic DP that incorporates random shocks to state transitions, essential for modeling uncertain asset returns. The expectation operator in the Bellman equation integrates over the probability distribution of future states.

• Handles Brownian motion and jump-diffusion price processes
• Requires discretization of continuous state variables for numerical solution
• Used in consumption-investment problems (Merton's model)
• Enables optimal execution with stochastic price impact

Techniques to overcome the curse of dimensionality by representing the value function with parametric models rather than tabular lookup. Neural networks and basis function expansions learn compact representations of V(s) from sampled trajectories.

• Deep Q-Networks: Neural value functions for high-dimensional states
• Least-Squares Monte Carlo: Regression-based approximation for American option pricing
• Tile coding: Discretization with overlapping receptive fields
• Enables DP in continuous state spaces with hundreds of dimensions

An algorithm that alternates between policy evaluation (computing the value function for a fixed policy) and policy improvement (greedily updating the policy based on the current value function). Converges to the optimal policy in finite MDPs.

• Guaranteed monotonic improvement at each iteration
• Often converges in fewer iterations than value iteration
• Computationally intensive per iteration due to exact policy evaluation
• Foundation for actor-critic methods in reinforcement learning

The continuous-time analog of the Bellman equation, expressed as a partial differential equation (PDE). In portfolio theory, the HJB equation characterizes the optimal value function when asset prices follow continuous-time stochastic processes.

• Governs Merton's portfolio problem with continuous rebalancing
• Solved analytically for specific utility functions (CRRA, CARA)
• Numerical solutions via finite difference methods for complex constraints
• Links stochastic control theory to dynamic asset allocation