Markov Decision Process (MDP)

The finite set of all possible situations the agent can encounter. In logistics, a state encodes warehouse inventory levels, vehicle locations, and pending orders. The Markov property dictates that the state captures all relevant history—the future depends only on the present state, not the sequence that preceded it.

• Fully Observable: Agent sees the exact state (standard MDP)
• Partially Observable: Agent receives noisy observations (extends to POMDP)
• Representation: Can be tabular (discrete grids) or continuous (neural embeddings)
• Example: A state tuple (truck_position, fuel_level, package_status)

The finite set of decisions available to the agent at each timestep. Actions transition the environment from one state to the next. In reinforcement learning for logistics, actions might include dispatching a truck, reordering inventory, or rerouting a shipment.

• Discrete: A fixed menu of choices (e.g., {north, south, east, west})
• Continuous: Real-valued controls (e.g., steering angle, acceleration)
• Constraint Handling: Invalid actions (e.g., shipping to a full warehouse) are masked
• Example: Action dispatch_truck_A_to_hub_3 triggers a state transition

The probability kernel P(s' | s, a) defining the environment's stochastic dynamics. Given a current state and action, it specifies the likelihood of landing in each possible successor state. This captures real-world uncertainty—a dispatched truck may arrive on time or encounter traffic.

• Deterministic: One guaranteed outcome per action (rare in practice)
• Stochastic: A distribution over outcomes (standard MDP assumption)
• Model-Based RL: Agent learns or is given P to plan ahead
• Model-Free RL: Agent learns optimal behavior without explicit knowledge of P

The scalar feedback signal R(s, a, s') that defines the agent's objective. The agent's sole purpose is to maximize the cumulative discounted reward over time. In supply chains, rewards encode business KPIs: on-time delivery yields positive reward, while late shipments or stockouts incur penalties.

• Dense Rewards: Frequent feedback at every step (e.g., distance traveled)
• Sparse Rewards: Feedback only at terminal states (e.g., delivery completed)
• Reward Shaping: Adding intermediate bonuses to accelerate learning
• Discount Factor (γ): Balances immediate vs. future rewards, typically 0.95–0.99

The agent's strategy—a mapping from states to actions (or action distributions). The policy is the output of the learning process. A deterministic policy π(s) = a always selects the same action, while a stochastic policy π(a|s) samples from a distribution, enabling exploration.

• *Optimal Policy (π)**: Maximizes expected cumulative reward from any start state
• Value-Based: Derives policy implicitly from action-value estimates (Q-Learning)
• Policy-Based: Directly parameterizes and optimizes the policy (Policy Gradient)
• Example: π(high_inventory, low_demand) = hold_or_redirect

Predicted long-term return used to evaluate states and actions. The state-value function V(s) estimates expected cumulative reward from state s under policy π. The action-value function Q(s, a) estimates the same from taking action a in state s. These functions drive the Bellman optimality equations.

• Bellman Equation: Q(s,a) = R(s,a) + γ * max Q(s', a')
• Temporal Difference (TD): Updates estimates using bootstrapped future predictions
• Advantage Function: A(s,a) = Q(s,a) - V(s) measures relative action quality
• Convergence: Guaranteed for tabular MDPs with sufficient exploration