Stochastic Optimal Control

The HJB equation is the continuous-time analog of the Bellman optimality principle, expressed as a nonlinear partial differential equation. It characterizes the value function—the minimum cost-to-go from any state—by balancing immediate running costs against expected future evolution. In optimal execution, the HJB equation explicitly trades off market impact cost against timing risk, yielding a deterministic optimal liquidation trajectory when solved analytically or numerically via finite-difference methods.

The value function V(t, x, q) maps the current state—time t, asset price x, and remaining inventory q—to the minimum expected cost of liquidating the remaining shares optimally. It serves as the sufficient statistic for decision-making: the optimal control at any instant is derived directly from its partial derivatives. In the Almgren-Chriss framework, the value function takes a quadratic form, enabling closed-form solutions for the optimal trading rate as a function of risk aversion and temporary impact parameters.

Stochastic optimal control is built on Itô calculus, which extends standard calculus to processes driven by Brownian motion. The Itô lemma provides the chain rule for stochastic differential equations, allowing the derivation of the HJB equation from the dynamic programming principle. Key components include:

• Drift term: Deterministic expected change in the state variable
• Diffusion term: Random fluctuation scaled by volatility
• Quadratic variation: The accumulated squared randomness that survives in the limit

The verification theorem establishes that any smooth solution to the HJB equation that satisfies terminal and boundary conditions is indeed the true value function, and the associated control policy is optimal. This bridges the gap between solving a PDE and proving optimality. In practice, it validates that the candidate solution derived analytically or numerically achieves the global minimum of the cost functional, eliminating concerns about local minima or spurious solutions.

Unlike deterministic optimal control, stochastic formulations explicitly model random disturbances in state dynamics. The key distinction lies in the cost functional:

• Deterministic: Minimizes a known trajectory cost
• Stochastic: Minimizes the expected value of future costs over all possible random paths This expectation operator fundamentally changes the optimal policy—introducing risk aversion that causes the controller to accelerate execution when uncertainty is high, rather than following a purely deterministic schedule.

The dynamic programming principle decomposes a multi-period optimization into a sequence of single-period problems by working backward from the terminal condition. At each instant, the optimal decision minimizes the sum of the immediate cost plus the expected value of future optimal costs. This recursive structure—"optimize today assuming optimal behavior tomorrow"—is the conceptual engine behind the HJB equation and enables tractable solutions to otherwise intractable sequential decision problems.

A cost measurement framework quantifying the difference between the decision price (when the trading intention is formed) and the final execution price. It captures:

• Explicit costs: Commissions, fees, and taxes
• Implicit costs: Market impact, delay cost, and missed trade opportunity cost

Often used as the objective function in stochastic optimal control formulations, where the goal is to minimize the expected shortfall subject to risk constraints.

A mathematical function that estimates the expected price movement caused by a trade of a specific size. Critical components include:

• Permanent impact: Information leakage that shifts the equilibrium price
• Temporary impact: Liquidity demand that dissipates as the order book replenishes
• Kyle's Lambda: The linear coefficient relating price change to signed order flow

These models serve as the state transition dynamics within stochastic optimal control problems, dictating how trading actions affect future prices.

An autonomous trading system trained via trial-and-error interaction with a market simulator to learn optimal order slicing and routing policies. Unlike classical stochastic control, RL agents can handle:

• High-dimensional state spaces including order book features
• Non-linear, non-convex cost functions
• Model-free optimization without explicit price dynamics

These agents approximate the value function and optimal policy that the HJB equation solves analytically in simpler settings.

The difference between the market price when the trading decision was made and the final average execution price achieved. It represents the total slippage incurred during implementation. In stochastic optimal control frameworks, arrival cost is often the running cost being minimized, with the optimal policy balancing:

• Immediate execution (low timing risk, high impact)
• Delayed execution (low impact, high timing risk)

The rate at which temporary price dislocation caused by a trade dissipates as the limit order book replenishes. This reflects the market's resilience and determines the transient component of execution cost. In stochastic control models, decay dynamics influence:

• The optimal time between child orders
• The aggressiveness of the liquidation schedule
• The memory embedded in the state space of the price process