Centralized Training Decentralized Execution (CTDE)

During the training phase, each agent's critic network receives the full global state—including observations, actions, and positions of all other agents. This extra information acts as a 'training wheels' mechanism, allowing the critic to accurately assess the value of an agent's action within the broader team context. The centralized critic can learn to assign credit even when an agent's local observation is ambiguous, dramatically accelerating the learning of cooperative behaviors. This approach directly addresses the non-stationarity problem inherent in independent multi-agent learning.

Once trained, the agent's actor (policy) network is deployed independently. It maps only the agent's own local sensory input directly to actions without querying a central controller or communicating with peers. This ensures:

• Zero communication overhead during decision-making
• Fault tolerance: failure of one agent does not crash the system
• Scalability: adding more agents does not increase per-agent latency
• Privacy: agents do not need to broadcast sensitive local state

CTDE is most commonly implemented using the actor-critic architecture. The critic is centralized and takes global state as input to compute a value estimate; the actor is decentralized and takes only local observations to output a probability distribution over actions. This decomposition is the foundation of algorithms like MADDPG (Multi-Agent Deep Deterministic Policy Gradient) and COMA (Counterfactual Multi-Agent Policy Gradients). After training, the centralized critic is discarded—only the lightweight actor networks are deployed to agents.

When agents are functionally identical—such as multiple base stations in a cellular network—CTDE enables parameter sharing. A single policy network is trained using the aggregated experiences of all agents, with agent-specific context provided via an agent ID embedding or local observation. This dramatically improves sample efficiency, as every agent's experience contributes to improving the shared policy. During execution, each agent runs an identical copy of the policy but conditions on its unique local observation, producing agent-specific behaviors.

A core challenge in cooperative multi-agent systems is determining which agent's action contributed to a shared team reward. CTDE enables counterfactual baselines—comparing the global reward received to what the reward would have been had the agent taken a different action. Algorithms like COMA compute a separate baseline for each agent by marginalizing out that agent's action, providing a precise learning signal that isolates individual contributions from collective outcomes and prevents lazy agent problems.

In AI-enhanced Radio Access Networks, CTDE maps naturally to the distributed infrastructure. During offline training in a digital twin or network simulator, a centralized critic has access to full network telemetry—all UE positions, channel states, and cell loads. During live deployment, each base station's agent executes independently using only its own local SINR measurements, buffer status reports, and connected user equipment CQIs. This enables coordinated interference management and load balancing without real-time inter-gNB signaling, directly reducing X2/Xn interface overhead.

The foundational framework for CTDE, where multiple autonomous agents learn simultaneously in a shared environment. Unlike single-agent RL, MARL must address non-stationarity—where one agent's policy change alters the environment dynamics for others. CTDE is a specific MARL paradigm designed to mitigate this by providing global context during training while maintaining decentralized execution.

A hybrid RL architecture that naturally supports CTDE. The actor is the decentralized policy that maps local observations to actions. The critic is a centralized value function that evaluates actions using global state information. During training, the critic provides a low-variance learning signal to the actor. At execution, only the actor is deployed, enabling decentralized operation without communication overhead.

The Decentralized Partially Observable Markov Decision Process is the formal mathematical framework underlying CTDE. It models scenarios where multiple agents operate under partial observability—each agent receives only a local observation of the true global state. The framework defines how joint actions produce joint observations and a shared reward, capturing the coordination problem CTDE algorithms are designed to solve.

A technique critical for stabilizing CTDE training. Past transitions—comprising global state, local observations, actions, and rewards—are stored in a replay buffer and randomly sampled in mini-batches. This breaks harmful temporal correlations in sequential data. In CTDE, the replay buffer must store both the centralized critic's global state and each agent's local observation-action pairs to enable effective off-policy learning.

The primary application domain for CTDE in wireless networks. RRM encompasses the algorithms that allocate power, spectrum, and time slots across base stations and user equipment. CTDE enables RRM agents—each controlling a single cell or user group—to learn globally optimal allocation strategies during training while executing decisions independently using only local channel state information and buffer status reports.

A key coordination challenge that CTDE addresses in dense cellular deployments. When multiple base stations transmit simultaneously, co-channel interference degrades signal quality. A CTDE approach trains each base station's scheduling agent with access to global interference maps, but at execution, each agent selects beamforming vectors and power levels using only local SINR measurements and user locations, eliminating the need for real-time inter-cell coordination.