Market Environment Wrapper

Conforms market simulators to the OpenAI Gym or PettingZoo API standards, exposing reset() and step() methods. This abstraction allows any RL algorithm—from PPO to SAC—to interact with market data without modification.

• Returns observations as NumPy arrays or PyTorch tensors
• Accepts discrete (buy/sell/hold) or continuous (position sizing) action spaces
• Emits scalar reward signals after each step

Transforms raw market microstructure into structured feature vectors. The wrapper computes technical indicators (RSI, MACD, Bollinger Bands), normalizes price data, and constructs order book embeddings—compact vector representations of bid-ask liquidity across price levels.

• Supports multi-modal observations combining OHLCV, volume profiles, and sentiment
• Handles lookback windows to provide temporal context
• Optionally includes private variables like current position and P&L

Maps abstract RL actions to executable market orders. A continuous action in [-1, 1] might represent a percentage of available capital to allocate, while discrete actions map to specific order types.

• Converts agent output to limit orders, market orders, or order cancellations
• Enforces position limits and short-selling constraints
• Handles order sizing relative to current portfolio value

Provides pluggable reward calculators that translate portfolio performance into scalar feedback. Common implementations include differential Sharpe ratio for risk-adjusted returns and profit-and-loss with transaction cost penalization.

• Prevents reward hacking by incorporating slippage models
• Supports multi-objective rewards balancing returns, drawdown, and turnover
• Enables custom reward shaping for specific trading objectives

Controls the temporal boundaries of trading simulations. The reset() method initializes the environment to a random or specific historical timestamp, while terminal conditions define when an episode ends.

• Supports fixed-length episodes (e.g., 252 trading days)
• Implements early termination on stop-loss breaches or maximum drawdown
• Shuffles starting points across historical data for robust training

Extends the wrapper to handle portfolios of correlated instruments or competitive trading scenarios. In multi-agent configurations, each agent observes a partial view of the market and competes for liquidity.

• Manages vectorized environments for parallel training across assets
• Implements turn-based or simultaneous action execution
• Tracks per-agent portfolio states and attribution

Defines the state representation the agent receives from the market. This is typically a structured vector or tensor containing normalized features.

• Price Data: OHLCV bars, mid-prices, micro-prices.
• Order Book State: Multi-level bid/ask volumes and depths.
• Private Variables: Current portfolio holdings, cash balance, and unrealized PnL.
• Alternative Data: Sentiment scores or correlation matrices.

Specifies the set of valid trading decisions the agent can execute at each step. The design directly impacts strategy complexity.

• Discrete: Simple actions like {BUY, SELL, HOLD} or fixed-size order placements.
• Continuous: A vector representing precise order quantities or target portfolio weights, often constrained to [-1, 1].
• Parametric: Actions that specify order type, price limit, and duration simultaneously.

The scalar feedback signal that encodes the trading objective. This is the only mechanism for the agent to learn what constitutes a 'good' trade.

• Profit & Loss (PnL): The change in portfolio value, often the most direct signal.
• Differential Sharpe Ratio: A risk-adjusted metric that accounts for return volatility.
• Transaction Cost Penalization: Subtracts explicit fees and estimated slippage to prevent unprofitable churning.

The mechanism that returns the environment to an initial state at the start of each episode. This is crucial for preventing overfitting to a single market trajectory.

• Random Sampling: Selects a random historical start date and initializes the portfolio with a fixed cash amount.
• Regime Conditioning: Ensures the agent trains on a balanced mix of bull, bear, and sideways market regimes.
• State Randomization: Introduces minor noise to initial prices or spreads to improve policy robustness.

The core execution loop that advances the environment by one discrete time increment (e.g., 1 second, 1 day). It processes the agent's action and returns the next state.

• Market Simulation: Fetches the next price tick or bar from the historical replay or synthetic generator.
• Order Matching: Simulates the fill logic based on the current order book depth and the agent's limit price.
• Portfolio Update: Recalculates cash, holdings, margin requirements, and the new net asset value.

The standard API specification (formerly OpenAI Gym) that the wrapper implements to ensure compatibility with RL libraries like Stable-Baselines3 or RLlib.

• env.step(action): Returns (observation, reward, terminated, truncated, info).
• env.reset(): Returns the initial observation and an info dictionary.
• env.action_space: A gym.spaces object defining the structure and bounds of valid actions.
• env.observation_space: A gym.spaces object defining the shape and data type of the state vector.