Q-Learning

The mathematical framework that formalizes the environment in which Q-Learning operates. An MDP is defined by a tuple (S, A, P, R, γ) where S is the state space, A is the action space, P is the state transition probability matrix, R is the reward function, and γ is the discount factor. Q-Learning is designed to solve MDPs where the transition dynamics P and reward function R are unknown to the agent, making it a model-free method. In trading, the MDP state might represent current portfolio holdings and market microstructure features, while actions correspond to order placements.

The core learning signal that drives Q-value updates. The TD error δ is computed as:

δ = R(s,a) + γ * max_a' Q(s',a') - Q(s,a)

This represents the difference between the current Q-value estimate and the bootstrapped target—the immediate reward plus the discounted maximum future value. Unlike Monte Carlo methods that wait until episode completion, TD learning updates estimates based on other learned estimates, enabling online, incremental learning. In trading, this allows agents to update their strategy continuously as new market data arrives without waiting for position closure.

The recursive decomposition that Q-Learning approximates. The Bellman optimality equation for the action-value function states:

Q*(s,a) = E[R(s,a) + γ * max_a' Q*(s',a')]

This expresses the optimal Q-value as the expected immediate reward plus the discounted optimal future value. Q-Learning uses this as an update target without requiring the expectation over transitions—it samples from actual experience. The algorithm's convergence guarantee relies on the Bellman operator being a contraction mapping in the space of Q-functions, ensuring iterative updates converge to the unique fixed point Q*.

The fundamental dilemma that Q-Learning must resolve to learn effectively. The agent must balance:

• Exploitation: Selecting the action with the highest current Q-value to maximize immediate reward
• Exploration: Trying suboptimal actions to discover potentially better strategies

Q-Learning is an off-policy algorithm, meaning it can learn the optimal policy while following an exploratory behavior policy. Common exploration strategies include:

• Epsilon-greedy: Select random action with probability ε, greedy action otherwise
• Boltzmann exploration: Sample actions from a softmax distribution over Q-values

In trading, insufficient exploration leads to overfitting to historical regimes, while excessive exploration incurs unnecessary transaction costs.

The neural network extension of tabular Q-Learning that enables scaling to high-dimensional state spaces. DQN addresses three key challenges:

• Function approximation: A deep neural network parameterized by θ approximates Q(s,a;θ) instead of a lookup table
• Experience replay: Transitions (s,a,r,s') are stored in a replay buffer and sampled randomly to break temporal correlations and improve data efficiency
• Target networks: A separate frozen network Q(s,a;θ⁻) generates TD targets, with θ⁻ updated periodically to stabilize training

DQN was the breakthrough that enabled reinforcement learning to master complex tasks from raw sensory input. In trading, DQN can process order book embeddings and price histories as state representations.

A closely related concept that decomposes the action-value function to reduce variance in learning. The advantage function is defined as:

A(s,a) = Q(s,a) - V(s)

where V(s) is the state-value function representing the expected return from state s. The advantage quantifies how much better an action is compared to the average action in that state. While standard Q-Learning directly estimates Q-values, algorithms like Dueling DQN separately estimate V(s) and A(s,a) and combine them to compute Q-values. This architecture learns which states are valuable without requiring every action to be evaluated, improving sample efficiency in trading environments where most actions have similar outcomes.