Posted on Leave a comment

GRL for line flow control


Xu, P., Pei, Y., Zheng, X., & Zhang, J. (2020, October). A simulation-constraint graph reinforcement learning method for line flow control. In 2020 IEEE 4th Conference on Energy Internet and Energy System Integration (EI2) (pp. 319-324). IEEE.


System uncertainty resulting from the unknown strategies of other producers presents challenges to the search for the best bidding strategies in power markets. Despite its popularity, distributed optimization cannot withstand this unpredictability. While deep reinforcement learning (DRL) has potential, it does not integrate spatially. A semi-distributed learning strategy combining a graph convolutional neural network (GCN) and DRL is presented in this research. Generation units adjust bids to handle uncertainty by using input from the surroundings. Units can understand system structure and improve tactics thanks to GCN’s inputs of state and node connections. When compared to standard DRL, evaluation on IEEE 30-bus and 39-bus systems highlights better generalization and profit potential.

Issues in previous methods:

  • Previous RL algorithms did not take network topology into account.
  • Difficulties with distributed decision-making in intricate network systems, such as dynamic uncertainty and non-cooperative behavior.
  • Past studies that concentrated on conventional centralized decision-making, which raised issues with privacy and increased computing load.
  • The challenge of locating Nash Equilibrium in situations including dispersed decision-making and dynamic uncertainty.
  •  Because strategy and state spaces are continuous, standard RL algorithms provide computational difficulties.
  •  Prior research on power market bidding did not specifically address system concerns
  •  Previous RL approaches’ limited capacity to generalize across different system topologies.
  •  Previous research missing full definition of bidding methods as a two-level optimization issue.


The proposed methodology tackles incomplete information challenges in a bi-level optimization problem, where generation units aim to maximize revenue without knowledge of competitors’ bids or market prices. A novel Deep Reinforcement Learning (DRL) algorithm enables units to learn competitors’ strategies through feedback. Units submit bids to the ISO, which then clears the market and determines prices. Units adjust bids based on feedback until convergence to maximum profit. DRL employs Actor-Critic networks, with state space comprising historical data like price and demand. Action involves bidding strategies, and rewards are based on revenue minus generation costs. The algorithm integrates Graph Convolutional Neural Networks (GCN) to incorporate system topology. GCN-based Critic networks use demand graph and market prices to update weights, enhancing learning efficiency and system awareness.

The architecture of the proposed method from the study by Xu, P., Pei, Y., Zheng, X., & Zhang, J.(2020, October)


  • Using GCN and DRL together enhances bidding tactics.
  • Addresses uncertainty in the system and missing information.
  • Improves decision-making by enabling the learning of system topology.
  • DRL with GCN performs better than traditional DRL because it is more flexible to changing topologies.
  • Future studies should incorporate uncertainties related to wind turbines and assess on broader networks.

Sakthivel R

I am a First-year M.Sc., AIML student enrolled at SASTRA University

I possess a high level of proficiency in a variety of programming languages and frameworks including Python. I have experience with cloud and database technologies, including SQL, Excel, Pandas, Scikit, TensorFlow, Git, and Power BI

Leave a Reply

Your email address will not be published. Required fields are marked *