Centralized Training Decentralized Execution (CTDE)

The defining architectural split of CTDE. During centralized training, a critic network has access to the full global state—including the positions, intentions, and observations of all agents—as well as joint actions. This critic computes a value function that accurately assesses the contribution of a single agent's action to the team's overall objective. The actor network, however, is conditioned only on the agent's own local observation history. At execution time, the critic is discarded entirely, and the agent acts autonomously using only its local policy. This is typically implemented via Actor-Critic algorithms with parameter sharing among homogeneous agents.

In standard multi-agent reinforcement learning, the environment appears non-stationary from the perspective of any single agent because other agents are simultaneously learning and changing their behavior. This violates the Markov assumption and causes training instability. CTDE addresses this by conditioning the centralized critic on the joint observation-action space. From the critic's omniscient viewpoint, the world is stationary because the dynamics of all agents are accounted for. This stabilization is critical for learning convergent spectrum access policies where multiple secondary users must coordinate without explicit communication.

CTDE naturally supports parameter sharing among homogeneous agents, where all agents use copies of the same neural network. During training, experiences from all agents are aggregated into a single replay buffer, dramatically improving sample efficiency. The shared policy learns a generalizable strategy conditioned on each agent's unique local observation. This allows a trained policy to be deployed to an arbitrary number of cognitive radios without retraining, enabling truly scalable dynamic spectrum access networks. The approach is foundational to algorithms like MADDPG and QMIX.

A key challenge in CTDE is ensuring that the decentralized actors learn coordinated behavior without explicit communication. Value decomposition techniques address this by factorizing the centralized joint action-value function into per-agent utility functions. The IGM (Individual-Global-Max) principle ensures that the optimal joint action is equivalent to the collection of each agent's greedy local action. Algorithms like VDN (additive decomposition) and QMIX (monotonic mixing network) enforce this constraint, enabling agents to learn implicit coordination protocols for collision-free spectrum access.

A direct practical benefit of CTDE for spectrum access is zero communication overhead during execution. Agents do not need to exchange coordination messages, which would consume scarce spectrum resources and introduce latency. Instead, coordination is implicitly encoded in the learned policy. For example, two cognitive radios trained under CTDE can learn to select orthogonal channels without ever negotiating, simply by observing the same environmental cues—such as time-of-day or detected energy patterns—that were present during centralized training.

CTDE is inherently suited for sim-to-real transfer. The centralized training phase can be conducted entirely in a high-fidelity RF digital twin or spectrum simulation environment where global state information is trivially available. The resulting decentralized policies, which rely only on local sensor observations, can then be deployed directly onto physical cognitive radios operating in the real electromagnetic environment. This decoupling allows for extensive training on rare interference events and adversarial jamming scenarios that would be impractical or dangerous to replicate in live spectrum.

The broader field encompassing CTDE. In MARL, multiple agents learn concurrently in a shared environment. The core challenge is non-stationarity: from a single agent's perspective, the environment dynamics appear to change as other agents update their policies. CTDE directly addresses this by allowing agents to condition on global state during training, making the learning problem stationary, while preserving decentralized scalability at execution time.

The mathematical framework that rigorously models the decentralized execution phase. A POMDP extends the Markov Decision Process by acknowledging that an agent never sees the true global state. Instead, it receives a local observation that is probabilistically related to the underlying state. CTDE agents are trained to map these partial, noisy observations—such as local spectrum sensing data—to optimal actions without requiring global knowledge at runtime.

The formal model for fully cooperative multi-agent teams under partial observability. In a Dec-POMDP, agents act based on individual observation histories to maximize a single shared reward. Solving Dec-POMDPs optimally is NEXPTIME-complete, motivating the CTDE paradigm as a practical, learning-based approximation. CTDE uses centralized critics to estimate the joint value function, bypassing the intractable exact solution.

The neural network architecture underlying most CTDE implementations. The architecture splits the agent into two components:

• Actor: The decentralized policy network that maps local observations to actions. Only this component is deployed at execution.
• Critic: The centralized value network that evaluates actions using global state. This component is discarded after training. This separation is the direct mechanism enabling the CTDE paradigm.

A technique frequently combined with CTDE to train homogeneous agents. Instead of learning N distinct policies, a single policy network is trained using the aggregated experiences of all agents. Agent-specific conditioning—appending a one-hot agent ID or local context to observations—prevents identical behavior. This dramatically reduces the number of trainable parameters and accelerates convergence in large-scale spectrum access networks.

A seminal CTDE algorithm that addresses the credit assignment problem: determining which agent's action contributed to a shared reward. COMA uses a centralized critic to compute a counterfactual baseline—what the joint reward would have been if a single agent had taken a different action—marginalizing out the contributions of other agents to isolate individual impact.