Posted on Leave a comment

Electrical System for Varying Topologies using Graph based DRL

Reference Paper

Zhao, Y., Liu, J., Liu, X., Yuan, K., Ren, K., & Yang, M. (2022, December). A graph-based deep reinforcement learning framework for autonomous power dispatch on power systems with changing topologies. In 2022 IEEE Sustainable Power and Energy Conference (iSPEC) (pp. 1-5). IEEE.   https://doi.org/10.1109/iSPEC54162.2022.10033001

Overview

Modern energy systems face increased complexity due to growing distributed power supply and energy load variability. Traditional autonomous policies struggle with this changing topology. To address this, the author proposes a framework combining Convolutional Graphics Network with DRL and GraphSAGE. it lies in the category of graph based DRL. This ensures adaptability to evolving topologies caused by emergencies, maintenance, and grid development. The Policy Proximal Optimization (PPO) algorithm facilitates effective power dispatch by recognizing network characteristics. This approach feeds graphical structured data into DRL to optimize outcomes.

 limitations of previous existing methods: 

  •  Traditional OPF policies overlook power system expansion and operational variations.
  • Some studies integrate DRL into OPF using PPO for distributed networks but are limited by fixed network topology.
  • To address this, recent research employs GCN in DQN for voltage control in varying topologies.
  • However, these approaches often use fixed-size input matrices, limiting their adaptability to new bus additions.

 This research proposes a unique graph based DRL framework for autonomous power dispatch that takes topology changes into account.  This work presents the Markov Decision Problem (MDP), a discrete-time control process model that is expressed as follows: 

min c_P,i =+α * P_i + β * P_A + γ * Σ loss_dis,i,j + δ * Σ loss_dis,i,k

This formula seems to represent a cost function for power distribution considering power values and losses between nodes.

 Graph based DRL Methodology

graph based DRL framework as shown in Figure 1 (Zhao et al., 2022)  

                  The suggested DRL architecture for electricity dispatch based on graphs. The critic NN’s architecture and the actor NN’s structure (which also constitutes IL’s structure). There are three branches to the suggested structure. In branch 2, the DRL agent interacting with the environment in the electricity dispatch issue utilizes the PPO algorithm.

The PPO neural networks include the Graph SAGE method to capture the characteristics of dynamic topologies. But starting from scratch might result in worse training outcomes. Therefore, in branch 1, the NN parameters of the PPO are initialized using historical data and expert knowledge. Branch 3 demonstrates how the PPO agent is updated during training by using a replay buffer.

 Every time step t, where it is represented as follows, the reinforcement learning (RL) agent engages with the environment and receives a reward t r:  Author sets a Multiple running cases, and the actions receive the highest reward are chosen through search to create the IL dataset.  

  • Architecture: Three-branch, graph-based DRL architecture.
  • Actor-Critic NN: Captures dynamic topologies by using GraphSAGE and the PPO method.
  • Initialization: Expert knowledge and historical data are used to initialize the PPO’s NN parameters.
  • Training: A replay buffer is used to update the PPO agent.
  • Case Study: Performed using a modified IEEE 118-bus design.

Every time step t, where it is represented as follows, the reinforcement learning (RL) agent engages with the environment and receives a reward t r: 

Conclusion

  • A graph based DRL structure for autonomous power dispatch was suggested in the paper, taking changes in power system topology into consideration.  
  • In this study, the Graph SAGE method is combined with DRL and makes use of imitation learning.  
  • In the proposed case study demonstrates that the conventional dense based PPO algorithm compared with suggested PPO algorithm, the suggested graph-based PPO approach is more successful when dealing with shifting topologies.  

Sakthivel R

I am a First-year M.Sc., AIML student enrolled at SASTRA University

I possess a high level of proficiency in a variety of programming languages and frameworks including Python. I have experience with cloud and database technologies, including SQL, Excel, Pandas, Scikit, TensorFlow, Git, and Power BI

Leave a Reply

Your email address will not be published. Required fields are marked *