Deep Reinforcement Learning for VVO

Unlike Model Predictive Control (MPC) , DRL agents do not require an explicit, pre-calibrated mathematical model of the distribution network. The agent learns the optimal mapping from grid states to control actions purely through trial-and-error interaction with a simulation or the live environment.

• Eliminates model bias: Avoids errors from inaccurate line impedance or load models.
• Adapts to topology changes: Automatically adjusts to feeder reconfiguration without manual re-engineering.
• Key algorithms: Deep Q-Networks (DQN) for discrete tap changes; Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) for continuous reactive power dispatch.

The agent's behavior is shaped by a carefully designed reward function that encodes operational objectives and constraints. This scalar signal guides the neural network toward policies that balance competing goals.

• Primary objectives: Minimize active power losses and maintain voltage within ANSI C84.1 Range A limits.
• Secondary objectives: Penalize excessive Load Tap Changer (LTC) operations to extend equipment life; minimize reactive power flow at the substation to improve power factor.
• Constraint handling: Large negative rewards (penalties) are assigned for voltage limit violations, effectively encoding hard constraints as soft penalties in the optimization.

The agent's observation vector must capture the complete electrical state of the feeder without overwhelming the neural network with dimensionality. Effective state representation is critical for convergence.

• Direct measurements: Voltage magnitudes at critical nodes, real and reactive power flows on line segments, and current tap positions of regulators.
• Temporal context: Stacking the last k measurements provides the agent with a sense of system dynamics and load trends.
• Topology encoding: Binary flags indicating the status of tie and sectionalizing switches allow the agent to reason about the current network configuration.

VVO involves a hybrid action space: discrete tap changes and continuous reactive power setpoints. DRL architectures must be designed to handle this mixed control problem effectively.

• Discrete actions: LTC tap raises and lowers are treated as discrete integer steps, often using a Dueling DQN architecture.
• Continuous actions: Smart inverter reactive power dispatch is a continuous value, typically handled by an actor-critic method like SAC.
• Hybrid approaches: A common pattern uses a hierarchical agent where a high-level policy selects discrete device settings and a low-level policy handles continuous modulation.

Training a DRL agent directly on a live distribution grid is unsafe and impractical. The agent is trained in a high-fidelity digital twin simulation and then deployed to the physical system.

• Training environment: Uses an unbalanced three-phase load flow solver (e.g., OpenDSS) to simulate the feeder response to control actions.
• Domain randomization: During training, load profiles and solar generation patterns are randomized to force the agent to learn a robust policy that generalizes to unseen conditions.
• Online fine-tuning: Once deployed, the agent can continue learning via Online Feedback Optimization (OFO) , using live measurements to refine its policy without a full system model.

A pure DRL policy can occasionally output unsafe actions during exploration or due to model inaccuracies. A deterministic safety layer is placed between the agent's output and the physical devices.

• Action masking: The safety layer invalidates any action that would cause an immediate voltage violation, forcing the agent to select a safe alternative.
• Tap change blocking: Prevents tap operations if the measured current exceeds the LTC's rated interrupting capacity.
• Deadband enforcement: Overrides the agent's command if the voltage deviation is within a configurable deadband, preventing unnecessary equipment hunting.