Multi-Agent Reinforcement Learning (MARL)

The mathematical bedrock of single-agent RL. An MDP formally defines sequential decision-making through a tuple: states, actions, transition probabilities, and rewards. In logistics, a state might be 'inventory level at warehouse A,' an action is 'dispatch truck to warehouse B,' and the reward is the negative of transportation cost. MARL extends this framework to stochastic games, where the transition dynamics and rewards depend on the joint actions of all agents, creating a moving-target problem for each learner.

A critical extension of the MDP for real-world logistics where no single agent has a god's-eye view. In a POMDP, an agent receives an observation that is a noisy or incomplete function of the true underlying state. For example, a delivery drone may only see local weather, not a storm 50 miles away. Agents must maintain a belief state—a probability distribution over possible true states—to act optimally. This formalizes the challenge of decentralized execution in MARL.

The dominant paradigm for overcoming non-stationarity in cooperative MARL. During training, agents have access to global information—the full state, other agents' actions, and shared rewards—often through a centralized critic. During execution, agents act using only their local observations. Key architectures include:

• MADDPG: Extends DDPG with a centralized critic per agent
• QMIX: A monotonic value function factorization that ensures decentralized policies are consistent with a centralized optimum
• COMA: A counterfactual multi-agent policy gradient that marginalizes out a single agent's action to compute a dedicated baseline

The fundamental difficulty of determining which agent's past action contributed to a shared global reward. In a warehouse where multiple robots collaborate to pack an order, a delayed reward signal makes it ambiguous whether robot A's retrieval or robot B's sorting was the bottleneck. Solutions include:

• Difference Rewards: Shaping an agent's reward as the difference between the global reward and the reward the system would have achieved without that agent
• Shapley Value: A game-theoretic approach that fairly distributes credit based on an agent's marginal contribution to every possible coalition
• COMA: Uses a counterfactual baseline to isolate an individual agent's advantage

The central dilemma amplified in multi-agent settings. An agent must decide whether to exploit its current best-known policy or explore new actions to discover better strategies. In MARL, exploration is complicated by the co-adaptation of other agents. If all agents explore simultaneously, the environment becomes excessively noisy. Structured exploration strategies include:

• Parameter-space noise: Adding noise to policy weights rather than action selection for temporally consistent exploration
• Curiosity-driven exploration: Using intrinsic rewards based on prediction error of a learned dynamics model to incentivize visiting novel joint states

An approach that decomposes a complex logistics task into a hierarchy of subtasks, enabling learning at multiple temporal scales. A high-level meta-controller selects a sub-goal (e.g., 'rebalance inventory in region X'), while a low-level controller executes primitive actions to achieve it (e.g., specific truck dispatch commands). This is critical for MARL in supply chains where planning horizons range from minutes (traffic re-routing) to months (warehouse stocking). Options Framework is a common formalization, defining temporally extended actions with initiation and termination conditions.