Market Impact Agent

The agent's primary objective is to minimize the difference between the decision price and the final execution price. It learns to balance urgency against market impact by decomposing large parent orders into smaller child orders. The reward function typically penalizes the implementation shortfall, which includes both explicit costs (commissions, fees) and implicit costs (slippage, spread crossing). By modeling the price impact function, the agent predicts how its own trading will move the market and adjusts participation rates accordingly.

The agent's observation space is constructed from high-fidelity market microstructure data. Key features include:

• Limit Order Book (LOB) imbalance: Ratio of bid to ask volume at multiple price levels
• Recent trade flow: Signed volume and trade aggressor flags
• Spread and depth: Current bid-ask spread and cumulative volume at each tick
• Private inventory: Remaining shares to execute and current P&L
• Time remaining: Normalized time until the execution deadline

These features are often compressed into an order book embedding using a convolutional or recurrent neural network before being fed to the policy network.

The agent outputs a continuous or discrete action representing its trading decision at each time step. Common action parameterizations include:

• Participation rate: Percentage of market volume to capture in the next interval
• Limit order placement: Price level and size for passive liquidity provision
• Market order size: Aggressive quantity to execute immediately
• Order type selection: Binary choice between limit and market orders

In continuous action spaces, algorithms like Soft Actor-Critic (SAC) or Twin Delayed DDPG (TD3) are preferred for their sample efficiency and stability.

The reward function must capture risk-adjusted execution quality. A common approach uses the differential Sharpe ratio, an online, differentiable approximation that allows gradient-based optimization of the Sharpe ratio directly. The agent receives a reward proportional to the trade's excess return per unit of risk. Transaction cost penalization is critical: the reward is reduced by explicit fees and estimated slippage to prevent the agent from learning unrealistic high-frequency churning strategies that would be unprofitable in live trading.

To build robust policies, agents are trained in adversarial market simulations where a separate adversary model learns to create challenging market conditions. This generative adversarial approach prevents overfitting to historical data. The adversary may manipulate the order book arrival process or simulate regime-switching environments where volatility and liquidity parameters shift between distinct states. Domain randomization further enhances robustness by varying simulation parameters like tick size, latency, and fill probability during training.

Real markets are Partially Observable Markov Decision Processes (POMDPs) because the agent cannot observe hidden liquidity, iceberg orders, or other participants' intentions. The agent maintains a belief state—a probability distribution over possible hidden market states—updated via Bayesian filtering. Architecturally, this is implemented using recurrent neural networks (LSTMs or GRUs) within the policy network, allowing the agent to integrate temporal context and infer latent market conditions from sequential observations of order flow and price movements.

The formal mathematical framework underlying the Market Impact Agent's learning. An MDP is defined by a tuple (S, A, P, R).

• S (State): The agent's observation, such as the current limit order book snapshot and remaining inventory.
• A (Action): The agent's decision, typically the quantity to execute in the next time step.
• P (Transition): The probability of the market moving to a new state given the action.
• R (Reward): The negative cost of execution, including immediate slippage and future impact.

The agent's goal is to learn a policy π(s) that maximizes cumulative discounted reward.

A critical engineering component of the agent's reward function. Without it, the agent learns unrealistic strategies that ignore real-world frictions.

• Explicit Costs: Commissions, exchange fees, and regulatory charges are subtracted directly from the reward.
• Implicit Costs: The agent's own price impact is modeled as a penalty proportional to the square root of the trade size or via a linear permanent impact term.
• Slippage: The difference between the expected price of a trade and the price at which the trade is actually executed, heavily penalized to prevent aggressive liquidity consumption.

A technique for compressing the high-dimensional state of the limit order book into a dense, low-dimensional vector for the agent's neural network.

• Spatial Structure: Captures the shape of the bid and ask curves across multiple price levels.
• Temporal Dynamics: Recurrent neural networks or attention mechanisms encode how the book has evolved over recent time steps.
• Learned Representation: The embedding is trained end-to-end with the agent's policy, discovering the most predictive features for minimizing impact without manual feature engineering.

A training methodology where the market environment itself is a learning agent that adapts to exploit the trading agent's weaknesses.

• Generative Adversarial Networks (GANs) can produce realistic synthetic order flow that reacts adversarially to the agent's orders.
• Multi-Agent RL pits the execution agent against other profit-seeking agents, forcing it to learn robust, stealthy execution strategies.
• This prevents overfitting to static historical data and produces policies that are resilient to strategic, reactive market participants.

A reward signal that directly optimizes for risk-adjusted returns in an online, differentiable manner. Unlike terminal Sharpe ratio, it can be used as a per-step reward.

• Online Estimation: Maintains exponential moving averages of the first and second moments of returns.
• Differentiability: The derivative of the Sharpe ratio with respect to the policy parameters can be computed, enabling direct gradient-based optimization.
• Risk Penalization: Naturally discourages strategies with high volatility, even if they have high mean returns, aligning the agent's objective with institutional risk mandates.