Skip to main content
SpringerLink
Log in

Multi-objective deep reinforcement learning for emergency scheduling in a water distribution network

  • Regular research paper
  • Published:
Memetic Computing Aims and scope Submit manuscript

Abstract

In recent years, water contamination incidents have happened frequently, causing serious losses and impacts on society. Therefore, how to quickly respond to emergency pollution incidents is a widespread concern for academic and industry scientists. In this paper, aiming to deal with the uncertain environment and randomness of water demand, we present a method based on a deep reinforcement learning emergency scheduling algorithm combined with Lamarckian local search. This method can effectively schedule water valves and fire hydrants to isolate contaminated water and reduce the residual concentration of contaminants in water distribution networks (WDNs). Furthermore, two optimization objectives are formulated, and then multi-objective optimization and deep reinforcement learning are combined to solve this problem. A real-world WDN is employed and simulation results show that our proposed algorithm can effectively isolate contamination and reduce the risk exclosure to customers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Gaffield SJ (xxxx) Securing our water supply: protecting a vulnerable resource

  2. Hrudey S, Payment P, Huck P, Gillham R, Hrudey E (2003) A fatal waterborne disease epidemic in walkerton, ontario: comparison with other waterborne outbreaks in the developed world. Water Sci Technol 47(3):7–14

    Article  Google Scholar 

  3. Schwab K (2017) The Global Risks Report 2017,12th Edition, World Economic Forum, http://reports.weforum.org/global-risks-2017/. Accessed 3 Nov 2017

  4. Hu C, Li M, Zeng D, Guo S (2018) A survey on sensor placement for contamination detection in water distribution systems. Wirel Netw 24(2):647–661. https://doi.org/10.1007/s11276-016-1358-0

    Article  Google Scholar 

  5. Yan X, Zhao J, Hu C, Zeng D (2017) Multimodal optimization problem in contamination source determination of water supply networks. Swarm Evol Comput 2:1–6. https://doi.org/10.1016/j.swevo.2017.05.010

    Article  Google Scholar 

  6. Hu C, Yan X, Gong W, Liu X, Wang L, Gao L (2020) Multi-objective based scheduling algorithm for sudden drinking water contamination incident. Swarm Evol Comput 55:100674. https://doi.org/10.1016/j.swevo.2020.100674

    Article  Google Scholar 

  7. Yau K-LA, Kwong KH, Shen C (2013) Reinforcement learning models for scheduling in wireless networks. Front Comp Sci 7(5):754–766

    Article  MathSciNet  Google Scholar 

  8. Kim B-G, Zhang Y, Van Der Schaar M, Lee J-W (2015) Dynamic pricing and energy consumption scheduling with reinforcement learning. IEEE Trans Smart Grid 7(5):2187–2198

    Article  Google Scholar 

  9. Moghadam MH, Babamir SM (2018) Makespan reduction for dynamic workloads in cluster-based data grids using reinforcement-learning based scheduling. J Comput Sci 24:402–412

    Article  MathSciNet  Google Scholar 

  10. Karimpanal TG, Wilhelm E (2017) Identification and off-policy learning of multiple objectives using adaptive clustering. Neurocomputing 263:39–47

    Article  Google Scholar 

  11. Ferreira PVR, Paffenroth R, Wyglinski AM, Hackett TM, Bilén SG, Reinhart RC, Mortensen DJ (2017) Multi-objective reinforcement learning-based deep neural networks for cognitive space communications, in. Cognit Commun Aerosp Appl Workshop (CCAA) 2017:1–8

    Google Scholar 

  12. Brys T, Harutyunyan A, Vrancx P, Nowe A, Taylor ME (2017) Multi-objectivization and ensembles of shapings in reinforcement learning. Neurocomputing 263:48–59

    Article  Google Scholar 

  13. Rui W, Zhou Z, Ishibuchi H, Liao T, Tao Z (2018) Localized weighted sum method for many-objective optimization. IEEE Trans Evol Comput 22(1):3–18

    Article  Google Scholar 

  14. Pirotta M, Parisi S, Restelli M (2015) Multi-objective reinforcement learning with continuous pareto frontier approximation. Science 2:2928–2934

    MATH  Google Scholar 

  15. Ruizmontiel M, Mandow L, Perezdelacruz J (2017) A temporal difference method for multi-objective reinforcement learning. Neurocomputing 263:15–25

    Article  Google Scholar 

  16. Parisi S, Pirotta M, Peters J (2017) Manifold-based multi-objective policy search with sample reuse. Neurocomputing 263:3–14

    Article  Google Scholar 

  17. Chen X, Ghadirzadeh A, Björkman M, Jensfelt P (2019) Meta-learning for multi-objective reinforcement learning. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 977–983. https://doi.org/10.1109/IROS40897.2019.8968092

  18. Li K, Zhang T, Wang R (2020) Deep reinforcement learning for multiobjective optimization. IEEE Trans Cybern 2:1–12. https://doi.org/10.1109/TCYB.2020.2977661

  19. O’Callaghan D, Mannion P (2021) Tunable behaviours in sequential social dilemmas using multi-objective reinforcement learning. In: Proceedings of the 20th international conference on autonomous agents and multiagent systems, pp 1610–1612

  20. Westbrink F, Elbel A, Schwung A, Ding SX (2021) Optimization of dem parameters using multi-objective reinforcement learning. Powder Technol 379:602–616

    Article  Google Scholar 

  21. Ren T, Niu J, Cui J, Ouyang Z, Liu X (2021) An application of multi-objective reinforcement learning for efficient model-free control of canals deployed with IoT networks. J Netw Comput Appl 182:103049

    Article  Google Scholar 

  22. Taormina R, Galelli S (2018) Deep-learning approach to the detection and localization of cyber-physical attacks on water distribution systems. J Water Resour Plan Manag 144(10):04018065

    Article  Google Scholar 

  23. Sutton RS, Barto AG (1999) Reinforcement learning: an introduction. Neural Inf Process Syst 2:158

  24. Mnih V, Kavukcuoglu K, Silver D et al. (2013) Playing atari with deep reinforcement learning, arXiv:1312.5602

  25. Hu C, Cai J, Zeng D, Yan X, Gong W, Wang L (2019) Deep reinforcement learning based valve scheduling for pollution isolation in water distribution network. Math Biosci Eng 17:105–121

  26. Van Moffaert K, Drugan MM, Nowé A (2013) Hypervolume-based multi-objective reinforcement learning. In: International conference on evolutionary multi-criterion optimization, Springer, pp 352–366

  27. Li K, Zhang T, Wang R (2020) Deep reinforcement learning for multiobjective optimization. IEEE Trans Cybern 5:1669

    Google Scholar 

Download references

Acknowledgements

This research was supported in part by the NSF of China (Grant No. 62073300, U1911205, 62076225). This paper has been subjected to Hubei Key Laboratory of Intelligent Geo-Information Processing, China University of Geosciences, Wuhan 430074, China.

Author information

Authors and Affiliations

  1. Department of Computer Science, China University of Geosciences, Wuhan, China

    Chengyu Hu, Qiuming Wang, Wenyin Gong & Xuesong Yan

Authors
  1. Chengyu Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiuming Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenyin Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuesong Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuesong Yan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, C., Wang, Q., Gong, W. et al. Multi-objective deep reinforcement learning for emergency scheduling in a water distribution network. Memetic Comp. 14, 211–223 (2022). https://doi.org/10.1007/s12293-022-00366-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12293-022-00366-9

Keywords

Search

Navigation