Partially Observable MDP (POMDP)

The core innovation of a POMDP is the belief state, a sufficient statistic that compresses the entire history of actions and observations into a probability distribution over the true market regime. Rather than knowing the exact state, the agent maintains a vector b(s) representing the probability of being in each possible state s. This belief is updated recursively using a state estimator that applies Bayes' rule to incorporate new observations. In trading, this allows an agent to explicitly represent uncertainty about whether the market is trending, mean-reverting, or in a high-volatility regime.

The observation function O(o | s', a) defines the probability of perceiving observation o after taking action a and transitioning to state s'. This model captures sensor noise, partial information, and market microstructure opacity. Key aspects:

• Stochastic observations: The same underlying state can generate different observations due to noise
• Aliasing: Multiple true states may produce identical observations, forcing the agent to disambiguate through sequential information gathering
• In trading, the observation model might represent the probability of seeing a specific order book snapshot given the true liquidity regime

In a POMDP, the value function is defined over the continuous belief space rather than discrete states. The optimal value function V*(b) satisfies the Bellman equation adapted for partial observability, where the expectation is taken over both state transitions and observations. This transforms the problem into a fully observable MDP in belief space, but with a continuous, high-dimensional state space. Practical solutions often use point-based value iteration or policy search methods that sample reachable beliefs rather than covering the entire simplex.

A defining characteristic of POMDPs is that agents may take actions purely to reduce uncertainty rather than to gain immediate reward. These information-gathering behaviors emerge naturally from the belief-state framework:

• Active sensing: Placing small probing orders to reveal hidden liquidity
• Delayed execution: Waiting for more informative observations before committing capital
• Exploration bonuses: Intrinsic motivation to visit uncertain regions of belief space This contrasts with fully observable MDPs, where all actions are directed solely at reward accumulation.

Unlike MDPs where the current state is a sufficient statistic for decision-making, POMDP policies are functions of the entire action-observation history. The belief state compresses this history losslessly, but the policy π(b) still depends on all past interactions. This enables sophisticated behaviors:

• Dual control: Balancing exploitation with deliberate experimentation to improve future state estimates
• Hedging against uncertainty: Taking positions that perform adequately across multiple plausible market regimes
• Sequential hypothesis testing: Gradually accumulating evidence before committing to a directional bet

POMDPs are particularly suited to financial markets where the true state is inherently hidden:

• Regime detection under noise: Inferring whether the market is trending or mean-reverting from noisy price observations
• Adversarial environments: Modeling other market participants as having private information that must be inferred from order flow
• Optimal execution with hidden liquidity: Learning when and where to place orders when the full depth of the order book is not visible
• Portfolio allocation with latent factors: Maintaining a belief over unobserved macroeconomic regimes that drive cross-asset correlations

A probability distribution over all possible true market states, maintained by the agent when it cannot directly observe the environment. In a POMDP, the agent never sees the true state—instead, it updates its belief using Bayesian filtering on incoming observations like price ticks or order flow. This transforms the partially observable problem into a fully observable MDP over the continuous belief space, enabling standard planning algorithms to operate on the agent's subjective uncertainty.

The foundational framework from which POMDPs extend. An MDP assumes the agent has perfect, noise-free observation of the full environmental state at every timestep. In trading, this would mean seeing every hidden limit order and every participant's intention—an unrealistic assumption. POMDPs relax this by introducing an observation function that maps hidden states to noisy, partial sensor readings, making them far more applicable to real financial markets.

A recursive Bayesian state estimator that is the optimal belief update mechanism for linear-Gaussian POMDPs. In algorithmic trading, Kalman filters are used to estimate unobserved variables like the true mid-price from noisy bid-ask quotes, or to track a latent alpha factor that is only indirectly observed through correlated asset returns. The filter maintains a mean and covariance that exactly represents the belief state.

A scalable online planning algorithm for large POMDPs that combines Monte Carlo tree search with particle filtering. Instead of planning over the full continuous belief space, POMCP represents the belief as a set of unweighted particles—sampled hypotheses of the true state. Each particle is a potential market configuration, and the algorithm simulates action-observation trajectories to estimate Q-values, enabling real-time decision-making in complex trading environments.

A statistical model where the system transitions through unobserved (hidden) states that generate observable emissions. HMMs are the simplest form of POMDP with no actions or rewards—purely a perception model. In quantitative finance, HMMs are widely used for regime detection, classifying market conditions (bull, bear, sideways) from price data, which can then inform a POMDP-based trading agent's policy.

A market simulation where the underlying data-generating process transitions between distinct latent regimes—such as low-volatility trending, high-volatility mean-reverting, or crisis conditions. The agent cannot directly observe the current regime, making this a canonical POMDP problem. Successful trading agents must infer the regime from price action and volume patterns, then adapt their strategy accordingly, balancing aggressive positioning against capital preservation.