Markov Decision Process (MDP)

The state is a complete snapshot of the environment at a specific time step. It must satisfy the Markov property: the future is conditionally independent of the past given the present state.

• Discrete States: Finite set of distinct configurations (e.g., grid cells in a maze).
• Continuous States: Infinite values in a range (e.g., portfolio value, volatility).
• Market Example: A state vector might include current price, position size, and a 14-period RSI value.
• Representation: Raw market data is often transformed into a compact feature vector or order book embedding before being fed to the agent.

The set of all possible moves the agent can execute in a given state. The agent's policy maps states to a probability distribution over these actions.

• Discrete Actions: Finite set of choices (e.g., Buy, Sell, Hold).
• Continuous Actions: Real-valued vectors (e.g., order size as a fraction of portfolio, limit price offset in basis points).
• Trading Context: A continuous action might be a vector [0.7, -0.2] indicating buy 70% of max position for asset 1 and short 20% for asset 2.
• Constraints: Actions are often bounded to prevent short-selling violations or exceeding available capital.

The dynamics model defining the probability of moving from state s to state s' after taking action a. This captures the environment's stochasticity.

• Notation: P(s' | s, a) — the conditional probability of the next state given the current state and action.
• Model-Free vs. Model-Based: In model-free RL (e.g., DQN, PPO), the agent does not learn an explicit transition model. In model-based RL, the agent learns or is given P.
• Market Reality: True market transition dynamics are unknown and non-stationary. Agents typically operate model-free, learning from sampled experience rather than a predefined P.

A scalar signal R(s, a, s') emitted by the environment after each transition. The agent's sole objective is to maximize the expected cumulative discounted reward.

• Profit-Driven: Simple reward can be the change in portfolio mark-to-market value.
• Risk-Adjusted: Use the Differential Sharpe Ratio to directly optimize for risk-adjusted returns in an online fashion.
• Transaction Cost Penalization: Subtract explicit fees and estimated slippage from the reward to prevent the agent from learning unrealistic high-frequency churning.
• Sparsity: Rewards can be dense (every step) or sparse (only at episode end). Dense rewards often accelerate learning but require careful engineering to avoid unintended behaviors.

A parameter γ ∈ [0, 1] that determines the present value of future rewards. It controls the agent's planning horizon.

• γ ≈ 0: Myopic agent. Prioritizes immediate reward only. Equivalent to a scalper seeking instant profit.
• γ ≈ 1: Farsighted agent. Values distant rewards almost as much as immediate ones. Suitable for long-term portfolio growth.
• Mathematical Role: Ensures the infinite sum of rewards converges to a finite value in continuing tasks.
• Trading Calibration: A γ of 0.99 is common for daily trading, while 0.999 might be used for longer-horizon strategic asset allocation.

The agent's brain — a mapping from states to actions. It defines the behavior at every step.

• Deterministic Policy: a = μ(s). Always outputs the same action for a given state.
• Stochastic Policy: π(a | s). Outputs a probability distribution over actions, enabling exploration.
• Parameterization: In deep RL, the policy is a neural network with weights θ, denoted π_θ.
• Optimization: Algorithms like PPO and SAC directly adjust θ to maximize expected return. The policy is the primary output of the training process.