Reward Shaping

The gold standard for policy invariance. Auxiliary rewards are defined as the difference in a state potential function: F(s, s') = γΦ(s') - Φ(s). This mathematical structure guarantees the optimal policy of the original MDP is preserved, preventing the agent from learning unintended behaviors like reward hacking. In trading, Φ(s) often represents a heuristic estimate of portfolio value or a risk-adjusted distance to a target allocation.

A practical shaping technique that rewards the agent for reducing the gap between the current state and a desired terminal condition. Common implementations include:

• Inventory penalty: Penalize the agent for holding non-zero positions near market close to encourage flat books.
• Profit threshold bonus: Provide a sparse supplementary reward when cumulative P&L crosses a predefined target.
• Drawdown proximity: Apply negative shaping when the portfolio approaches a maximum allowable loss limit.

Intrinsic motivation signals that reward the agent for discovering novel or unpredictable states rather than exploiting known rewards. Implemented via an Intrinsic Curiosity Module (ICM) that predicts the next state embedding; the prediction error becomes the shaping bonus. In market environments, this prevents the agent from fixating on a single volatile asset and encourages the discovery of diverse, uncorrelated alpha sources.

Breaking a complex trading objective into a hierarchy of simpler sub-goals, each with its own shaped reward. A master policy selects high-level goals while sub-policies execute primitive actions. Example hierarchy:

• Level 1: Allocate capital across asset classes (equities, fixed income, commodities).
• Level 2: Select specific instruments within each class.
• Level 3: Execute optimal entry and exit timing. Shaping at each level provides dense feedback, dramatically accelerating learning compared to a monolithic sparse reward.

Directly modifying the reward function to penalize volatility and tail risk beyond raw returns. Instead of maximizing simple profit, the shaped reward incorporates:

• Differential Sharpe ratio: An online, differentiable approximation of the Sharpe ratio that serves as a direct reward signal.
• Sortino bonus: Penalizes only downside deviation, ignoring upside volatility.
• Calmar ratio shaping: Rewards the agent based on the ratio of annualized return to maximum drawdown. This ensures the learned policy optimizes for risk-adjusted performance from the start, not just raw P&L.

Combining Inverse Reinforcement Learning (IRL) with reward shaping by extracting a potential function from expert trader demonstrations. The process:

1. Collect trajectories of optimal execution from human traders or a heuristic oracle.
2. Learn a state-value function that explains the expert's behavior.
3. Use this learned value function as the shaping potential Φ(s). This bootstraps the agent with institutional knowledge, reducing random exploration in high-stakes environments where real-world mistakes are costly.

An online, differentiable approximation of the Sharpe ratio used as a direct reward signal. Instead of optimizing raw profit, the agent maximizes risk-adjusted returns by computing the derivative of the Sharpe ratio with respect to policy parameters. This acts as an auxiliary reward that incorporates volatility awareness directly into the gradient update, preventing the agent from learning high-variance strategies that look profitable but carry unacceptable tail risk.

A critical reward shaping technique that subtracts explicit costs from the agent's profit signal. The reward function incorporates:

• Commissions and fees per trade
• Slippage estimates based on order size
• Bid-ask spread costs

Without this penalization, agents learn unrealistic high-frequency churning strategies that appear profitable in simulation but fail catastrophically in live markets due to unmodeled friction.

A function quantifying how much better a specific action is compared to the average action in a given state. In reward shaping, the advantage function serves as a variance reduction mechanism: by subtracting the state baseline from the shaped reward, the agent learns which actions truly contribute to excess returns rather than being misled by market-wide movements. This is foundational to Generalized Advantage Estimation (GAE).

A technique that adds a bonus reward proportional to the entropy of the policy distribution. This shaped reward encourages the agent to maintain stochasticity and explore diverse action trajectories rather than prematurely converging to a suboptimal deterministic strategy. In trading, this prevents the agent from overfitting to a single market regime and promotes discovery of robust, adaptive behaviors across bull, bear, and sideways conditions.

A reinforcement learning model trained to minimize the adverse price movement caused by its own order execution. The reward function is shaped to penalize implementation shortfall—the difference between the decision price and the final execution price. By incorporating a market impact model into the reward signal, the agent learns optimal trade scheduling that slices large orders to minimize information leakage and price slippage.

A framework where the agent infers the underlying reward function from expert demonstrations rather than receiving an explicit shaped reward. In trading, IRL is used to model human trader behavior by observing their executed orders and recovering the implicit objectives they were optimizing. This learned reward function can then be used to train new agents that replicate expert decision-making without manual reward engineering.