Skip to main content
SpringerLink
Log in

EVADE: Efficient Moving Target Defense for Autonomous Network Topology Shuffling Using Deep Reinforcement Learning

  • Conference paper
  • First Online:
Applied Cryptography and Network Security (ACNS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13905))

Included in the following conference series:

  • 803 Accesses

Abstract

We propose an Efficient moVing tArget DEfense (EVADE) that periodically changes a network topology to thwart potential attackers for protecting a given network. To achieve autonomous network topology adaptations under high dynamics, we leverage deep reinforcement learning (DRL) in a moving target defense (MTD) strategy to defeat epidemic attacks. EVADE has two objectives, minimizing security vulnerability caused by the software monoculture and maximizing network connectivity for seamless communications. We design EVADE to autonomously shuffle a network topology by identifying a pair of network adaptation budgets to add and remove edges for generating a robust and connected network topology. To improve the learning convergence speed: 1) We propose a vulnerability ranking algorithm of edges and nodes (VREN) to effectively direct the DRL agent to select adaptations; 2) We develop a Fractal-based Solution Search (FSS) to build an efficient sampling environment for the agent to quickly converge to an optimal solution; and 3) We design density optimization (DO)-based greedy MTD to further refine the solution search space. This hybrid approach achieves faster training allowing running the DRL agent online. Via our extensive experiments under both real and synthetic networks, we demonstrate the outperformance of EVADE over its counterparts.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Achleitner, S., Porta, T.L., McDaniel, P., Sugrim, S., Krishnamurthy, S.V., Chadha, R.: Deceiving network reconnaissance using SDN-based virtual topologies. IEEE Trans. Netw. Serv. Manage. 14, 1098–1112 (2017)

    Article  Google Scholar 

  2. Anwar, A.H., Leslie, N.O., Kamhoua, C., Kiekintveld, C.: A game theoretic framework for software diversity for network security. In: GameSec 2020. LNCS, vol. 12513, pp. 297–311. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64793-3_16

    Chapter  MATH  Google Scholar 

  3. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34(6), 26–38 (2017)

    Article  Google Scholar 

  4. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: A brief survey of deep reinforcement learning. arXiv preprint: arXiv:1708.05866 (2017)

  5. Chai, X., Wang, Y., Yan, C., Zhao, Y., Chen, W., Wang, X.: DQ-MOTAG: deep reinforcement learning-based moving target defense against DDoS attacks. In: 2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC), pp. 375–379. IEEE (2020)

    Google Scholar 

  6. Cho, J.H., et al.: Toward proactive, adaptive defense: a survey on moving target defense. IEEE Commun. Surv. Tutorials 22(1), 709–745 (2020)

    Article  Google Scholar 

  7. Colbourn, C.: Network resilience. SIAM J. Algebraic Discrete Methods 8(3), 404–409 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  8. CVSS, Common Vulnerability Scoring System (CVSS), National Vulnerability Database (2022). https://www.first.org/cvss/

  9. Darvariu, V.-A., Hailes, S., Musolesi, M.: Improving the robustness of graphs through reinforcement learning and graph neural networks. arXiv preprint: arXiv:2001.11279 (2020)

  10. Das, D.: A fuzzy multiobjective approach for network reconfiguration of distribution systems. IEEE Trans. Power Delivery 21(1), 202–209 (2005)

    Article  Google Scholar 

  11. Desai, A., Milner, S.: Autonomous reconfiguration in free-space optical sensor networks. IEEE J. Sel. Areas Commun. 23(8), 1556–1563 (2005)

    Article  Google Scholar 

  12. Eghtesad, T., Vorobeychik, Y., Laszka, A.: Adversarial deep reinforcement learning based adaptive moving target defense. In: GameSec 2020. LNCS, vol. 12513, pp. 58–79. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64793-3_4

    Chapter  Google Scholar 

  13. Mavoungou, S., et al.: Survey on threats and attacks on mobile networks. IEEE Access 4, 4543–4572 (2016)

    Article  Google Scholar 

  14. Fernández, N., et al.: Virtual topology reconfiguration in optical networks by means of cognition: evaluation and experimental validation. IEEE/OSA J. Opt. Commun. Networking 7(1), A162–A173 (2015)

    Article  Google Scholar 

  15. Ge, M., Cho, J.-H., Kim, D., Dixit, G., Chen, I.-R.: Proactive defense for internet-of-things: moving target defense with cyberdeception. ACM Trans. Internet Technol. (TOIT) 22(1), 1–31 (2021)

    Article  Google Scholar 

  16. Grimmett, G.: Percolation and disordered systems. In: Bernard, P. (ed.) Lectures on Probability Theory and Statistics. LNM, vol. 1665, pp. 153–300. Springer, Heidelberg (1997). https://doi.org/10.1007/BFb0092620

    Chapter  MATH  Google Scholar 

  17. Hole, K.J.: Diversity reduces the impact of malware. IEEE Secur. Privacy 13(3), 48–54 (2015)

    Article  Google Scholar 

  18. Hong, J.B., Kim, D.S.: Assessing the effectiveness of moving target defenses using security models. IEEE Trans. Dependable Secure Comput. 13(2), 163–177 (2016)

    Article  Google Scholar 

  19. Hong, J.B., Yoon, S., Lim, H., Kim, D.S.: Optimal network reconfiguration for software defined networks using shuffle-based online MTD. In: 2017 IEEE 36th Symposium on Reliable Distributed Systems (SRDS), pp. 234–243 (2017)

    Google Scholar 

  20. Huang, C., Zhu, S., Erbacher, R.: Toward software diversity in heterogeneous networked systems. In: Atluri, V., Pernul, G. (eds.) DBSec 2014. LNCS, vol. 8566, pp. 114–129. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-43936-4_8

    Chapter  Google Scholar 

  21. Huang, C., Zhu, S., Guan, Q., He, Y.: A software assignment algorithm for minimizing worm damage in networked systems. J. Inf. Secur. Appl. 35, 55–67 (2017)

    Google Scholar 

  22. Jensen, T.R., Toft, B.: Graph Coloring Problems, vol. 39. John Wiley & Sons, Hoboken (2011)

    MATH  Google Scholar 

  23. Kaur, T., Baek, J.: A strategic deployment and cluster-header selection for wireless sensor networks. IEEE Trans. Consum. Electron. 55(4), 1890–1897 (2009)

    Article  Google Scholar 

  24. Kim, S., et al.: DIVERGENCE: deep reinforcement learning-based adaptive traffic inspection and moving target defense countermeasure framework. IEEE Trans. Netw. Serv. Manag. 19, 4834–4846 (2022)

    Article  Google Scholar 

  25. Kohl, N., Stone, P.: Policy gradient reinforcement learning for fast quadrupedal locomotion. In: Proceedings. ICRA2004, vol. 3, pp. 2619–2624. IEEE (2004)

    Google Scholar 

  26. Kreutz, D., Ramos, F.M.V., Veríssimo, P.E., Rothenberg, C.E., Azodolmolky, S., Uhlig, S.: Software-defined networking: a comprehensive survey. Proc. IEEE 103(1), 14–76 (2015)

    Article  Google Scholar 

  27. Leong, A.S., Quevedo, D.E., Ahlén, A., Johansson, K.H.: On network topology reconfiguration for remote state estimation. IEEE Trans. Autom. Control 61(12), 3842–3856 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  28. Leskovec, J., Kleinberg, J., Faloutsos, C.: Graphs over time: densification laws, shrinking diameters and possible explanations. In: Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, pp. 177–187 (2005)

    Google Scholar 

  29. Leskovec, J., Mcauley, J.: Learning to discover social circles in ego networks. In: Advances in Neural Information Processing Systems, vol. 25 (2012)

    Google Scholar 

  30. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  31. Najjar, W., Gaudiot, J.L.: Network resilience: a measure of network fault tolerance. IEEE Trans. Comput. 39(2), 174–181 (1990)

    Article  Google Scholar 

  32. Newman, M.: Networks: An Introduction. Oxford University Press, Oxford (2010)

    Book  MATH  Google Scholar 

  33. Newman, M., Watts, D.: Scaling and percolation in the small-world network model. Phys. Rev. E 60(6), 7332–7342 (1999)

    Article  Google Scholar 

  34. O’Donnell, A.J., Sethu, H.: On achieving software diversity for improved network security using distributed coloring algorithms. In: Proceedings of the 11th ACM Conference on Computer and Communications Security, pp. 121–131. ACM (2004)

    Google Scholar 

  35. University of Washington, Rocketfuel maps and data, April 2003. http://www.cs.washington.edu/research/networking/rocketfuel/

  36. Rao, R.S., Ravindra, K., Satish, K., Narasimham, S.: Power loss minimization in distribution system using network reconfiguration in the presence of distributed generation. IEEE Trans. Power Syst. 28(1), 317–325 (2012)

    Article  Google Scholar 

  37. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. CoRR, vol. abs/1707.06347 (2017). http://arxiv.org/abs/1707.06347

  38. Singh, S., Litman, D., Kearns, M., Walker, M.: Optimizing dialogue management with reinforcement learning: experiments with the NJFun system. J. Artif. Intell. Res. 16, 105–133 (2002)

    Article  MATH  Google Scholar 

  39. Sterbenz, J.P., et al.: Resilience and survivability in communication networks: strategies, principles, and survey of disciplines. Comput. Netw. 54(8), 1245–1265 (2010)

    Article  MATH  Google Scholar 

  40. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  41. Temizkan, O., Park, S., Saydam, C.: Software diversity for improved network security: optimal distribution of software-based shared vulnerabilities. Inf. Syst. Res. 28(4), 828–849 (2017)

    Article  Google Scholar 

  42. Touhiduzzaman, M., Hahn, A., Srivastava, A.K.: A diversity-based substation cyber defense strategy utilizing coloring games. IEEE Trans. Smart Grid 10, 5405–5415 (2018)

    Article  Google Scholar 

  43. Tozer, B., Mazzuchi, T., Sarkani, S.: Optimizing attack surface and configuration diversity using multi-objective reinforcement learning. In: IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 144–149. IEEE (2015)

    Google Scholar 

  44. Wan, Z., Mahajan, Y., Kang, B.W., Moore, T.J., Cho, J.-H.: A survey on centrality metrics and their implications in network resilience (2020)

    Google Scholar 

  45. Yang, Y., Zhu, S., Cao, G.: Improving sensor network immunity under worm attacks: a software diversity approach. In: Proceedings of the 9th ACM International Symposium on Mobile Ad Hoc Networking and Computing, ser. MobiHoc 2008, pp. 149–158 (2008)

    Google Scholar 

  46. Yang, Y.: Improving sensor network immunity under worm attacks: a software diversity approach. Ad Hoc Networks, vol. 47, no. Supplement C, pp. 26–40 (2016)

    Google Scholar 

  47. Zhang, Q., Cho, J.H., Moore, T.J.: Network resilience under epidemic attacks: deep reinforcement learning network topology adaptations. In: IEEE Global Communications Conference (GLOBECOM), pp. 1–7 (2021)

    Google Scholar 

  48. Zhang, Q., Cho, J.H., Moore, T.J., Chen, R.: Vulnerability-aware resilient networks: Software diversity-based network adaptation. IEEE Trans. Netw. Serv. Manag. (2020)

    Google Scholar 

  49. Zhang, Q., Cho, J.H., Moore, T.J., Nelson, F.F.: DREVAN: deep reinforcement learning-based vulnerability-aware network adaptations for resilient networks. In: IEEE Conference on Communications and Network Security (CNS), pp. 137–145 (2021)

    Google Scholar 

  50. Zhang, Q., Mohammed, A.Z., Wan, Z., Cho, J.H., Moore, T.J.: Diversity-by-design for dependable and secure cyber-physical systems: a survey (2020)

    Google Scholar 

  51. Zhang, T., et al.: DQ-RM: deep reinforcement learning-based route mutation scheme for multimedia services. In: 2020 IEEE International Wireless Communications and Mobile Computing (IWCMC), pp. 291–296 (2020)

    Google Scholar 

  52. Zhang, Y., Murata, M., Takagi, H., Ji, Y.: Traffic-based reconfiguration for logical topologies in large-scale WDM optical networks. J. Lightw. Technol. 23(10), 2854–2867 (2005)

    Article  Google Scholar 

  53. Zhu, M., Hu, Z., Liu, P.: Reinforcement learning algorithms for adaptive cyber defense against heartbleed. In: Proceedings of the First ACM Workshop on Moving Target Defense, pp. 51–58 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Virginia Tech, Falls Church, VA, USA

    Qisheng Zhang & Jin-Hee Cho

  2. US Army Research Laboratory, Adelphi, MD, USA

    Terrence J. Moore & Frederica Nelson

  3. School of ITEE, The University of Queensland, Saint Lucia, Australia

    Dan Dongseong Kim

  4. Korea Institute of Energy Technology, Naju-si, South Korea

    Hyuk Lim

Authors
  1. Qisheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin-Hee Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Terrence J. Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dan Dongseong Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hyuk Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Frederica Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qisheng Zhang .

Editor information

Editors and Affiliations

  1. NTT Social Informatics Laboratories, Tokyo, Japan

    Mehdi Tibouchi

  2. Indiana University at Bloomington, Bloomington, IN, USA

    XiaoFeng Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, Q., Cho, JH., Moore, T.J., Kim, D.D., Lim, H., Nelson, F. (2023). EVADE: Efficient Moving Target Defense for Autonomous Network Topology Shuffling Using Deep Reinforcement Learning. In: Tibouchi, M., Wang, X. (eds) Applied Cryptography and Network Security. ACNS 2023. Lecture Notes in Computer Science, vol 13905. Springer, Cham. https://doi.org/10.1007/978-3-031-33488-7_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-33488-7_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-33487-0

  • Online ISBN: 978-3-031-33488-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation