Centralized Training Decentralized Execution (CTDE)

During the training phase, agents have unrestricted access to the full global state of the environment, including the positions, actions, and observations of all other agents. This eliminates the non-stationarity problem—where other agents' changing policies make the environment appear dynamic from a single agent's perspective. By training with omniscient critic networks, agents learn optimal cooperative behaviors without the instability caused by partial observability during the learning process.

At execution time, each agent operates using only its own local observations and partial state information. No communication with a central controller is required, and agents do not share information with each other unless explicitly designed to do so. This decoupling ensures:

• Scalability: Adding agents does not increase per-agent computational load
• Robustness: No single point of failure
• Low Latency: Decisions are made locally without network round-trips
• Privacy: Sensitive local data never leaves the agent

CTDE is typically implemented through an Actor-Critic architecture where:

• The Critic is centralized and receives global state information during training to accurately estimate value functions
• The Actors are decentralized and map local observations to actions

This design leverages the Centralized Training with Decentralized Execution principle: the critic provides stable learning signals using global knowledge, while actors learn policies that depend only on locally available information. Common algorithms include MADDPG, COMA, and QMIX.

In homogeneous agent settings, CTDE often employs parameter sharing—all agents use identical neural network weights. This dramatically reduces the number of trainable parameters and enables sample-efficient learning across agents. During execution, agents differentiate their behavior through their unique local observations rather than distinct model weights. For heterogeneous fleets, separate parameter groups can be maintained while still benefiting from centralized critic training.

CTDE frameworks directly tackle the multi-agent credit assignment problem—determining which agent's actions contributed to a shared team reward. Centralized critics can compute counterfactual baselines or learn joint action-value functions that decompose team rewards into per-agent contributions. Techniques include:

• COMA: Uses a counterfactual baseline marginalizing out individual agent actions
• VDN/QMIX: Decomposes the joint Q-function into agent-specific utility functions
• Value Decomposition Networks: Learn additive or monotonic mixing of individual value estimates

CTDE is particularly suited to warehouse robotics and fleet coordination where:

• Training occurs in a digital twin simulation with full observability of all robots, inventory positions, and order queues
• Execution happens on individual autonomous mobile robots (AMRs) using only onboard LiDAR, cameras, and local task assignments

This enables training policies for collision avoidance, task allocation, and congestion management that generalize to real-world partial observability. The paradigm bridges the sim-to-real gap by ensuring policies do not depend on simulation-only information.

The foundational framework where multiple autonomous agents learn and interact simultaneously within a shared environment. CTDE is a specific paradigm within MARL designed to solve the non-stationarity problem—where one agent's changing policy makes the environment appear dynamic to others. In logistics, MARL enables fleets of autonomous mobile robots to coordinate without centralized control during execution.

• Key challenge: Non-stationarity breaks standard RL convergence guarantees
• CTDE solution: Agents access global state during training only
• Logistics example: Warehouse robots learning to avoid congestion without a central dispatcher

An extension of the Markov Decision Process where agents cannot directly observe the full environmental state. CTDE directly addresses the decentralized POMDP (Dec-POMDP) challenge: each agent receives only local observations during execution. In supply chain control towers, this mirrors reality—a truck only knows its own GPS location and nearby traffic, not the entire fleet's status.

• Belief state: Agent maintains a probability distribution over possible true states
• Observation function: Maps true state to what the agent can perceive
• CTDE alignment: Centralized training resolves partial observability; decentralized execution respects real-world sensor limitations

A hybrid RL architecture that combines policy gradient methods (the actor) with a learned value function (the critic). In CTDE implementations, the critic often has access to global information during training while the actor uses only local observations—a pattern known as centralized critic with decentralized actors. This architecture is the most common instantiation of CTDE in practice.

• Actor: Selects actions based on local observations only
• Critic: Estimates value using global state during training, discarded at execution
• Advantage: Dramatically reduces variance while maintaining decentralized execution capability

The challenge of determining which past actions in a sequence were responsible for a received outcome. In cooperative multi-agent systems, this becomes the multi-agent credit assignment problem: identifying which agent's actions contributed to a shared reward. CTDE frameworks use centralized critics to solve this by evaluating individual contributions against the global state.

• Counterfactual baseline: Comparing reward with and without an agent's action
• Difference rewards: Isolating an agent's marginal contribution
• Logistics example: Determining which truck's rerouting decision prevented a delivery delay across the network

A technique that stores agent experiences—state, action, reward, next state tuples—in a replay buffer and randomly samples them for training. In CTDE, experience replay must be carefully managed because the experiences of multiple agents are temporally correlated. Centralized training leverages this by storing all agents' experiences together, breaking correlations that would destabilize independent learning.

• Prioritized experience replay: Samples important transitions more frequently
• CTDE benefit: Enables off-policy learning from historical fleet data
• Practical use: Training delivery route optimizers on months of historical logistics data

The fundamental dilemma between gathering new knowledge about the environment and leveraging existing knowledge to maximize immediate reward. In CTDE, this challenge is compounded across agents: coordinated exploration prevents agents from interfering with each other's learning. Centralized training enables parameter sharing and coordinated exploration strategies that would be impossible with fully decentralized learners.

• Epsilon-greedy: Simple exploration by random action selection with decreasing probability
• Entropy regularization: Encourages exploration by penalizing overconfident policies
• CTDE advantage: Global training coordinates exploration to avoid redundant or conflicting agent behaviors