State Space

The dimensionality of a state space is the number of variables used to describe the environment. High-dimensional spaces suffer from the curse of dimensionality, where the volume grows exponentially, making exploration sparse and value function approximation computationally intractable. In RAN optimization, raw IQ samples or full grid channel matrices create massive state spaces that require deep neural networks for effective feature extraction.

A state is Markovian if it summarizes all historical information necessary to predict the future. Formally, the probability of the next state depends only on the current state and action: P(s_{t+1} | s_t, a_t). A sufficient state statistic eliminates the need for memory of past observations. In wireless systems, the current Channel State Information (CSI) and buffer status often form a sufficient Markov state for scheduling decisions.

State spaces are categorized by their mathematical nature:

• Discrete: Finite, countable states (e.g., a grid world, quantized SINR levels). Solvable via tabular methods.
• Continuous: Infinite, real-valued states (e.g., raw voltage readings, GPS coordinates). Require function approximators like neural networks.
• Hybrid: Mix of discrete modes and continuous parameters, common in network slicing where a slice type is discrete but resource allocation is continuous.

In a fully observable MDP, the agent's observation equals the true environmental state. In a Partially Observable MDP (POMDP), the agent receives an incomplete observation. RAN agents often operate in POMDPs because they cannot observe the exact interference at every user equipment or the internal buffer states of neighboring base stations. Solutions involve maintaining a belief state or using recurrent neural networks to infer hidden context.

Raw network telemetry must be transformed into a learnable representation. Critical preprocessing steps include:

• Normalization: Scaling metrics like SINR (dB) and throughput (Mbps) to a zero-mean, unit-variance range to stabilize neural network training.
• Temporal Aggregation: Averaging channel quality indicators over a sliding window to reduce noise.
• Topological Encoding: Using Graph Neural Networks to embed the physical cell layout directly into the state vector, capturing neighbor interference patterns.

To combat the curse of dimensionality, complex global states are decomposed into factored representations. Instead of a single monolithic state for a multi-cell network, the state is factored into local cell states plus an abstract coordination signal. This decentralized factorization enables Centralized Training Decentralized Execution (CTDE) architectures, where local policies operate on compact, relevant subsets of the global state.

The complementary set of all possible moves available to the agent at each time step. The state space defines where the agent is, while the action space defines what it can do. Together they form the input-output dimensionality of the policy:

• Discrete: A finite set (e.g., increase, decrease, maintain power)
• Continuous: A range of real values (e.g., transmit power from -20 dBm to 23 dBm)
• Hybrid: A combination of discrete and continuous parameters

In partially observable environments, the agent receives an observation rather than the true state. The state space represents the complete, ground-truth configuration of the environment, while the observation space is the agent's limited, possibly noisy view. This distinction is critical in wireless networks where an agent may only see local SINR measurements, not the global interference map.

The process of selecting and transforming raw telemetry into a compact, informative state vector. Effective state design reduces dimensionality and improves learning speed. Common RAN state features include:

• Per-UE metrics: SINR, CQI, buffer status, QoS class identifier
• Cell-level metrics: PRB utilization, active users, transmit power
• Topological features: Neighbor cell loads, adjacency matrices for graph neural networks

A fundamental challenge where the state space grows exponentially with the number of dimensions, making exploration and learning intractable. In RAN optimization, a state with 50 cells, each with 10 features, creates a massive combinatorial space. Function approximation via deep neural networks is the primary tool for overcoming this curse by generalizing across similar states.

A critical engineering step that transforms raw state values to a consistent scale before feeding them to a neural network. Without normalization, features with large magnitudes (e.g., throughput in Mbps) can dominate features with small magnitudes (e.g., packet loss ratio). Common techniques include:

• Z-score normalization: (x - μ) / σ
• Min-max scaling: (x - min) / (max - min)
• Running statistics: Exponentially weighted moving averages for non-stationary environments