Markov Decision Process (MDP)

The defining assumption that the future depends only on the present state, not the history of how you arrived there. Formally: P(s_{t+1} | s_t, a_t) = P(s_{t+1} | s_t, a_t, s_{t-1}, a_{t-1}, ..., s_0). This memoryless property makes computation tractable.

• State Sufficiency: The state representation must capture all relevant information for decision-making
• Practical Implication: In a digital twin of a warehouse, the state must encode current inventory levels, pending orders, and fleet positions—not the full 3-year history
• Violation Consequence: If the Markov property doesn't hold, the agent makes suboptimal decisions because it lacks critical context

The complete set of all possible configurations the agent can encounter. States can be discrete (e.g., inventory level is 0, 1, 2... units) or continuous (e.g., a truck's GPS coordinates).

• Curse of Dimensionality: State spaces explode combinatorially—a supply chain with 100 SKUs and 50 locations yields an astronomically large state space
• Feature Representation: Raw sensor data is often abstracted into a compact feature vector (e.g., average lead time, stockout risk score)
• Observability: In fully observable MDPs, the agent sees the true state; in Partially Observable MDPs (POMDPs), it receives noisy observations

The set of all possible decisions available to the agent at each state. Actions can be deterministic (a single choice) or stochastic (a probability distribution over choices).

• Discrete Actions: Reorder 0, 10, or 50 units; route a truck to Dock A, B, or C
• Continuous Actions: Set reorder quantity to any real number between 0 and 1000; adjust conveyor belt speed from 0.5 to 3.0 m/s
• Constrained Actions: Real-world logistics imposes hard constraints—a truck cannot be assigned two conflicting routes simultaneously
• Action Masking: In reinforcement learning, invalid actions are masked to prevent the agent from selecting impossible moves

The probability of landing in state s' after taking action a in state s. This encodes the environment's stochastic dynamics—the same action may yield different outcomes due to uncertainty.

• Deterministic Transitions: In a simplified simulation, ordering 100 units always increases inventory by exactly 100 after a fixed lead time
• Stochastic Transitions: In reality, ordering 100 units may result in receiving 80–120 units after 2–5 days, modeled as a probability distribution
• Model-Based vs. Model-Free: If P is known, the agent can plan; if unknown, the agent must learn it through interaction (reinforcement learning)
• Digital Twin Integration: A high-fidelity digital twin serves as an approximate transition model for offline planning

The scalar feedback signal that defines the objective. The agent's goal is to maximize cumulative reward over time, not just immediate reward. Reward shaping is the art of encoding business objectives into this function.

• Immediate Reward: +$500 for on-time delivery, -$200 for stockout, -$0.10 per unit-mile for transportation cost
• Delayed Reward: A decision to build inventory now may incur holding costs but enables a large reward from fulfilling a bulk order next month
• Sparse vs. Dense: Sparse rewards (only at goal) make learning slow; dense rewards (shaped feedback at each step) accelerate convergence
• Multi-Objective: In supply chains, rewards often combine cost, service level, and sustainability metrics into a weighted sum

A policy is the agent's strategy—a mapping from states to actions (or action probabilities). The value function V(s) estimates expected cumulative reward from state s under a given policy, while the action-value function Q(s,a) estimates it for a specific action.

• Optimal Policy π*: The strategy that maximizes expected cumulative reward from any starting state
• Bellman Equation: The recursive relationship V(s) = max_a [R(s,a) + γ Σ P(s'|s,a) V(s')] that underpins all solution methods
• Discount Factor γ: A parameter (0 to 1) that trades off immediate vs. future rewards; γ=0.95 means rewards 20 steps away are worth ~36% of immediate rewards
• Exploitation vs. Exploration: The agent must balance using known good actions (exploit) with trying new ones (explore) to avoid local optima