Reinforcement Learning (RL)

The fundamental interaction cycle where an agent observes the current state of the environment, selects an action based on its policy, and receives a scalar reward signal along with the next state. This closed-loop feedback system is continuous and sequential. Unlike supervised learning which uses static labeled datasets, the RL agent's actions directly influence the distribution of future data it will encounter, creating a moving target problem.

• State (S_t): A representation of the environment at time t (e.g., grid voltage magnitudes, transformer temperatures, renewable generation levels).
• Action (A_t): A control decision (e.g., adjust a capacitor bank setting, curtail a battery inverter, open a tie switch).
• Reward (R_{t+1}): A scalar feedback signal evaluating the immediate quality of the action (e.g., negative value for voltage violation, positive for loss reduction).

All goals and purposes in RL can be formalized as the maximization of the expected cumulative reward. The agent is not told what to do but is incentivized through a carefully engineered reward function. The objective is to learn a policy that maximizes the return (G_t), which is the discounted sum of future rewards.

• Return: G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + ...
• Discount Factor (γ ∈ [0,1]): Determines the present value of future rewards. A γ close to 0 makes the agent myopic (prioritizing immediate grid stability); a γ close to 1 makes it farsighted (prioritizing long-term asset health and efficiency).
• Credit Assignment Problem: The challenge of determining which past actions were responsible for a delayed reward signal, such as a transformer failure occurring hours after a sustained overload.

The agent must balance exploitation (selecting actions known to yield high reward based on current knowledge) with exploration (trying suboptimal or uncertain actions to discover potentially better strategies). This is a fundamental trade-off with no purely optimal solution.

• Epsilon-Greedy: A simple strategy where the agent selects a random action with probability ε, otherwise exploits. In grid control, this might mean occasionally testing a non-standard feeder configuration.
• Upper Confidence Bound (UCB): Selects actions based on both their estimated value and the uncertainty in that estimate, naturally decaying exploration as knowledge increases.
• Boltzmann Exploration: Selects actions probabilistically according to a softmax distribution over estimated values, controlled by a temperature parameter. Critical for safe exploration in physical infrastructure where random actions could cause violations.

RL problems are formally modeled as Markov Decision Processes, which provide the mathematical foundation for sequential decision-making under uncertainty. An MDP is defined by the tuple (S, A, P, R, γ).

• Markov Property: The future is conditionally independent of the past given the present state. The current state must contain all relevant information for decision-making.
• Transition Function P(s'|s,a): Defines the probability of transitioning to state s' after taking action a in state s. In grid contexts, this captures the stochasticity of load behavior and renewable generation.
• Policy π(a|s): A mapping from states to a probability distribution over actions. The goal of RL is to find the optimal policy π* that maximizes expected return.
• Value Functions: The state-value function V(s) estimates expected return from state s following policy π. The action-value function Q(s,a) estimates expected return from taking action a in state s and then following π.

RL algorithms are categorized by whether they learn or use an explicit model of the environment's dynamics.

• Model-Based RL: The agent learns (or is given) a model of P(s'|s,a) and R(s,a). It can then use planning to simulate future trajectories and improve its policy without real-world interaction. This is highly sample-efficient and aligns with Model Predictive Control (MPC) approaches already used in grid optimization.
• Model-Free RL: The agent learns a policy or value function directly from experience without modeling the environment. Policy Gradient methods and Q-Learning are common examples. These are simpler to implement but typically require significantly more interaction data.
• Dyna Architecture: A hybrid approach that learns a model from experience and uses it to generate simulated experience for accelerating model-free learning.

This distinction defines the relationship between the policy being learned and the policy used to generate experience.

• On-Policy: The agent learns the value of the policy it is currently executing. SARSA (State-Action-Reward-State-Action) is a classic on-policy algorithm. The policy being evaluated and improved is the same one generating behavior. This is generally more stable and safer for online grid control.
• Off-Policy: The agent learns the value of an optimal policy while following a different, often exploratory, behavioral policy. Q-Learning is the canonical off-policy algorithm. This enables learning from historical operational data or demonstrations, which is critical for leveraging existing SCADA archives.
• Experience Replay: A technique used in Deep Q-Networks (DQN) that stores past transitions in a replay buffer and samples them randomly for training, breaking temporal correlations and improving data efficiency.

RL agents learn to commit and dispatch generators in real-time under high renewable uncertainty, replacing traditional solvers that struggle with stochasticity.

• State Space: Generator status, load forecasts, wind/solar output, ramp rates.
• Action Space: Generator setpoints and commitment decisions.
• Reward Function: Minimize fuel cost plus penalty for constraint violations.
• Key Benefit: Adapts to non-stationary distributions without explicit re-modeling, handling the stochastic programming problem implicitly through learned value functions.

Deep RL controllers coordinate capacitor banks, voltage regulators, and smart inverters to minimize reactive power losses while maintaining ANSI C84.1 voltage limits.

• Challenge: The combinatorial action space of discrete capacitor switching and continuous inverter setpoints creates a mixed-integer problem.
• RL Approach: A Deep Q-Network (DQN) or Soft Actor-Critic (SAC) learns a policy that balances exploration of switching configurations with exploitation of known low-loss states.
• Result: Up to 4% reduction in distribution losses compared to rule-based Volt-VAR Control (VVC) schemes.

RL agents schedule EV charging sessions to flatten the net load curve and prevent distribution transformer overloads without violating driver departure constraints.

• Formulation: A Constrained Markov Decision Process (CMDP) where the agent maximizes charging completion while respecting transformer thermal limits.
• State: Time of day, transformer temperature, State of Charge (SoC) per vehicle, electricity price.
• Reward Shaping: Positive reward for completed charges, negative reward for transformer overload or peak demand exceedance.
• Deployment: Often trained offline using historical data and deployed via Digital Twin simulation before real-world rollout.

RL-based controllers for Battery Energy Storage Systems (BESS) provide fast Automated Generation Control (AGC) response, outperforming PID controllers in non-linear operating regimes.

• Agent Objective: Track the Area Control Error (ACE) signal while minimizing battery degradation.
• State: System frequency deviation, ACE, battery SoC, state of health.
• Action: Charge/discharge power command.
• Reward: Negative absolute ACE minus a weighted penalty for depth-of-discharge cycles that accelerate degradation.
• Architecture: Typically a Deep Deterministic Policy Gradient (DDPG) or Twin Delayed DDPG (TD3) for continuous action control.

Following a fault, an RL agent learns to dynamically reconfigure distribution switches to restore power to the maximum number of customers while respecting radiality and thermal constraints.

• Problem Type: Sequential decision-making on a graph topology, naturally suited for Graph Neural Network (GNN)-based policy networks.
• State: Switch statuses, feeder loading, fault location, priority customer flags.
• Action: Open/close a specific tie or sectionalizing switch.
• Reward: Number of customers restored minus a penalty for constraint violations or excessive switching operations.
• Advantage: Finds near-optimal restoration sequences in milliseconds, critical for Fault Detection Isolation and Recovery (FDIR) schemes.

RL agents represent prosumers or aggregators in Transactive Energy markets, learning optimal bidding strategies to maximize revenue from rooftop solar, batteries, and flexible loads.

• Multi-Agent RL (MARL): Each prosumer is an independent agent learning in a non-stationary environment shaped by other agents' strategies.
• State: Local generation, load, storage SoC, market clearing prices, time of day.
• Action: Bid price and quantity for the next market interval.
• Algorithm: Proximal Policy Optimization (PPO) with centralized training and decentralized execution to handle partial observability.
• Outcome: Converges to a Nash Equilibrium where no agent can unilaterally improve its payoff.