Order Book Embedding

Unlike flat feature vectors that treat each price level independently, embeddings capture the topological structure of the order book. Convolutional or attention-based encoders process the book as a structured grid, learning that liquidity at adjacent price levels is correlated. This preserves the spatial locality of supply and demand, allowing the agent to recognize patterns like iceberg orders, support/resistance walls, and liquidity gaps that span multiple levels.

A raw Level-3 order book can contain thousands of price levels with dozens of features each, creating a state space too large for efficient reinforcement learning. Embeddings compress this into a low-dimensional latent vector (typically 64-256 dimensions) while retaining critical microstructure information. This compression enables faster policy network training and reduces the curse of dimensionality that plagues value function approximation in high-frequency trading environments.

Advanced embedding architectures incorporate recurrent or transformer layers to encode not just the current book snapshot but its recent evolution. By processing sequences of order book states, the embedding captures order flow toxicity, quote stuffing detection, and momentum signals that are invisible in static snapshots. This temporal context is critical for distinguishing transient liquidity events from genuine supply/demand shifts.

Well-trained order book embeddings function as universal market state encoders that can be transferred across related trading instruments. An embedding model pre-trained on liquid equity order books can be fine-tuned for futures or foreign exchange markets with minimal adaptation. This representation learning approach reduces the need for hand-crafted features and allows agents to leverage common microstructure patterns across asset classes.

Embeddings are often trained using self-supervised contrastive objectives rather than end-to-end with the trading policy. The model learns to pull together representations of order books that precede similar price movements and push apart those preceding divergent outcomes. This decoupling ensures the embedding captures causally relevant structure rather than spurious correlations, producing a robust observation space for downstream reinforcement learning.

Sophisticated embedding architectures process the order book at multiple temporal and spatial resolutions simultaneously. Fine-grained levels capture immediate microstructure while coarse aggregations model broader liquidity profiles. This hierarchical encoding mirrors how human market makers perceive the book—tracking both the tick-by-tick queue dynamics and the macro-level depth profile—enabling agents to execute both aggressive and passive strategies from a unified representation.

The mathematical framework that necessitates order book embeddings. In a POMDP, the agent cannot directly observe the full market state—only noisy, high-dimensional snapshots like the limit order book. The embedding serves as a learned belief state, compressing raw LOB data into a compact representation that captures the underlying latent market dynamics. Without this compression, the agent's observation space would be too large for efficient learning.

A probability distribution over possible true market states maintained by an agent in a partially observable environment. Order book embeddings function as a learned, compressed belief state. Rather than explicitly modeling the distribution over latent variables like fair price or order flow toxicity, the embedding network learns to encode the sufficient statistics of the belief state directly from LOB data. This is updated recursively as new order book snapshots arrive, providing a Markovian representation for downstream policy networks.

An RL model trained to minimize the adverse price movement caused by its own order execution. The agent's observation space critically depends on order book embeddings to understand:

• Liquidity distribution across price levels
• Order book imbalance indicating short-term price pressure
• Hidden liquidity patterns from historical embedding sequences By learning optimal trade scheduling from embedded LOB states, the agent reduces implementation shortfall and avoids signaling its intentions to other market participants.

The difference between the predicted value of a state and the updated estimate incorporating an observed reward and the value of the subsequent state. In the context of order book embeddings:

• The value network takes the embedded LOB vector as input
• High TD error on specific embedding regions signals surprising market events or regime changes
• Prioritized experience replay uses TD error magnitude to oversample transitions where the embedding representation failed to predict the outcome, accelerating learning on rare but critical market scenarios

A market simulation where the underlying data-generating process transitions between distinct states like bull, bear, or sideways markets. Order book embeddings must be regime-aware to be effective:

• Embedding vectors should cluster differently for high-volatility vs. low-volatility regimes
• A single embedding space must encode both the microstructure state and the macro regime context
• Agents trained with domain randomization across regimes learn embeddings that generalize rather than overfitting to a single market condition