Abstract
Moving target defense (MTD) provides a systematic framework to achieving proactive defense in the presence of advanced and stealthy attacks. To obtain robust MTD in the face of unknown attack strategies, a promising approach is to model the sequential attacker-defender interactions as a two-player Markov game, and formulate the defender’s problem as finding the Stackelberg equilibrium (or a variant of it) with the defender and the leader and the attacker as the follower. To solve the game, however, existing approaches typically assume that the attacker type (including its physical, cognitive, and computational abilities and constraints) is known or is sampled from a known distribution. The former rarely holds in practice as the initial guess about the attacker type is often inaccurate, while the latter leads to suboptimal solutions even when there is no distribution shift between when the MTD policy is trained and when it is applied. On the other hand, it is often infeasible to collect enough samples covering various attack scenarios on the fly in security-sensitive domains. To address this dilemma, we propose a two-stage meta-reinforcement learning based MTD framework in this work. At the training stage, a meta-MTD policy is learned using experiences sampled from a set of possible attacks. At the test stage, the meta-policy is quickly adapted against a real attack using a small number of samples. We show that our two-stage MTD defense obtains superb performance in the face of uncertain/unknown attacker type and attack behavior.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al-Shaer, E., Duan, Q., Jafarian, J.H.: Random host mutation for moving target defense. In: Keromytis, A.D., Di Pietro, R. (eds.) SecureComm 2012. LNICST, vol. 106, pp. 310–327. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36883-7_19
Allen, L., Heriyanto, T., Ali, S.: Kali Linux-Assuring Security by Penetration Testing. Packt Publishing Ltd. (2014)
Basar, T.: Lecture notes on non-cooperative game theory (2010). https://www.hamilton.ie/ollie/Downloads/Game.pdf
Başar, T., Olsder, G.J.: Dynamic Noncooperative Game Theory. SIAM (1998)
Booth, H., Rike, D., Witte, G.A., et al.: The national vulnerability database (NVD): overview (2013)
Bowers, K.D., Dijk, M.E.V., Juels, A., Oprea, A.M., Rivest, R.L., Triandopoulos, N.: Graph-based approach to deterring persistent security threats. US Patent 8813234 (2014)
Brockman, G., et al.: OpenAI gym. ArXiv abs/1606.01540 (2016)
Cho, J.H., et al.: Toward proactive, adaptive defense: a survey on moving target defense. IEEE Commun. Surv. Tutor. 22(1), 709–745 (2020)
Derman, E., Mannor, S.: Distributional robustness and regularization in reinforcement learning. In: The Theoretical Foundations of Reinforcement Learning Workshop at ICML 2020 (2020)
Duan, Y., Schulman, J., Chen, X., Bartlett, P.L., Sutskever, I., Abbeel, P.: RL\(^2\): fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 (2016)
Eldosouky, A., Saad, W., Niyato, D.: Single controller stochastic games for optimized moving target defense. In: IEEE International Conference on Communications (ICC) (2016)
Fallah, A., Mokhtari, A., Ozdaglar, A.: Generalization of model-agnostic meta-learning algorithms: recurring and unseen tasks. In: NeurIPS (2021)
Feng, X., Zheng, Z., Mohapatra, P., Cansever, D.: A Stackelberg game and Markov modeling of moving target defense. In: Rass, S., An, B., Kiekintveld, C., Fang, F., Schauer, S. (eds.) GameSec 2017. LNCS, vol. 10575, pp. 315–335. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68711-7_17
Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning (ICML), pp. 1126–1135 (2017)
Fujimoto, S., Hoof, H., Meger, D.: Addressing function approximation error in actor-critic methods. In: International Conference on Machine Learning (ICML), pp. 1587–1596 (2018)
Ghavamzadeh, M., Mannor, S., Pineau, J., Tamar, A.: Bayesian reinforcement learning: a survey. Found. Trends Mach. Learn. 8(5–6), 359–492 (2015)
Hu, J., Wellman, M.P.: Nash Q-learning for general-sum stochastic games. J. Mach. Learn. Res. 4, 1039–1069 (2003)
Huang, P., Xu, M., Fang, F., Zhao, D.: Robust reinforcement learning as a Stackelberg game via adaptively-regularized adversarial training. arXiv preprint arXiv:2202.09514 (2022)
Jackson, T., et al.: Compiler-generated software diversity. In: Jajodia, S., Ghosh, A., Swarup, V., Wang, C., Wang, X. (eds.) Moving Target Defense. Advances in Information Security, vol. 54, pp. 77–98. Springer, New York (2011). https://doi.org/10.1007/978-1-4614-0977-9_4
Jafarian, J.H., Al-Shaer, E., Duan, Q.: OpenFlow random host mutation: transparent moving target defense using software defined networking. In: Proceedings of the First Workshop on Hot Topics in Software Defined Networks (HotSDN), pp. 127–132 (2012)
Jajodia, S., Ghosh, A.K., Swarup, V., Wang, C., Wang, X.S.: Moving Target Defense: Creating Asymmetric Uncertainty for Cyber Threats, vol. 54. Springer, Heidelberg (2011). https://doi.org/10.1007/978-1-4614-0977-9
Könönen, V.: Asymmetric multiagent reinforcement learning. Web Intell. Agent Syst. Int. J. (WIAS) 2(2), 105–121 (2004)
Li, H., Shen, W., Zheng, Z.: Spatial-temporal moving target defense: a Markov Stackelberg game model. In: International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) (2020)
Li, H., Zheng, Z.: Optimal timing of moving target defense: a Stackelberg game model. In: IEEE Military Communications Conference (MILCOM). IEEE (2019)
Luo, Y.B., Wang, B.S., Wang, X.F., Hu, X.F., Cai, G.L., Sun, H.: RPAH: random port and address hopping for thwarting internal and external adversaries. In: 2015 IEEE Trustcom/BigDataSE/ISPA, vol. 1, pp. 263–270. IEEE (2015)
Mell, P., Scarfone, K., Romanosky, S.: Common vulnerability scoring system. IEEE Secur. Priv. 4(6), 85–89 (2006)
Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 (2018)
Paruchuri, P., Pearce, J.P., Marecki, J., Tambe, M., Ordonez, F., Kraus, S.: Playing games for security: an efficient exact algorithm for solving Bayesian Stackelberg games. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 895–902 (2008)
Paulin, A.: Secure SQL server-enabling secure access to remote relational data. arXiv preprint arXiv:1201.1081 (2012)
Peng, W., Li, F., Huang, C.T., Zou, X.: A moving-target defense strategy for cloud-based services with heterogeneous and dynamic attack surfaces. In: International Conference on Communications (ICC), pp. 804–809. IEEE (2014)
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-baselines3: Reliable reinforcement learning implementations. J. Mach. Learn. Res. (2021)
Saputro, N., Tonyali, S., Aydeger, A., Akkaya, K., Rahman, M.A., Uluagac, S.: A review of moving target defense mechanisms for internet of things applications. Model. Design Secure Internet Things 563–614 (2020)
Sengupta, S., Kambhampati, S.: Multi-agent reinforcement learning in bayesian Stackelberg Markov games for adaptive moving target defense. arXiv preprint arXiv:2007.10457 (2020)
Sengupta, S., et al.: A game theoretic approach to strategy generation for moving target defense in web applications. In: International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 178–186 (2017)
Sharma, D.P., Kim, D.S., Yoon, S., Lim, H., Cho, J.H., Moore, T.J.: FRVM: flexible random virtual IP multiplexing in software-defined networks. In: IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pp. 579–587. IEEE (2018)
Sinha, A., Nguyen, T.H., Kar, D., Brown, M., Tambe, M., Jiang, A.X.: From physical security to cybersecurity. J. Cybersecur. 1(1), 19–35 (2015)
von Stengel, B., Zamir, S.: Leadership with commitment to mixed strategies. CDAM Research Report LSE-CDAM-2004-01 (2004)
Taguinod, M., Doupé, A., Zhao, Z., Ahn, G.J.: Toward a moving target defense for web applications. In: 2015 IEEE International Conference on Information Reuse and Integration, pp. 510–517. IEEE (2015)
Thomas, S., Williams, L.: Using automated fix generation to secure SQL statements. In: International Workshop on Software Engineering for Secure Systems (SESS). IEEE (2007)
Thompson, M., Evans, N., Kisekka, V.: Multiple OS rotational environment an implemented moving target defense. In: The 7th International Symposium on Resilient Control Systems (ISRCS), pp. 1–6. IEEE (2014)
Vorobeychik, Y., Singh, S.: Computing Stackelberg equilibria in discounted stochastic games (corrected version). In: Twenty-Sixth Conference on Artificial Intelligence (AAAI) (2012)
Vu, Q.L., et al.: Stackelberg policy gradient: evaluating the performance of leaders and followers. In: ICLR 2022 Workshop on Gamification and Multiagent Solutions (2022)
Wang, J.X., et al.: Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 (2016)
Weng, L.: Meta-learning: learning to learn fast. lilianweng.github.io (2018). https://lilianweng.github.io/posts/2018-11-30-meta-learning/
Xie, Q., Chen, Y., Wang, Z., Yang, Z.: Learning zero-sum simultaneous-move Markov games using function approximation and correlated equilibrium. In: COLT (2020)
Zhang, Y., Li, M., Bai, K., Yu, M., Zang, W.: Incentive compatible moving target defense against VM-colocation attacks in clouds. In: Gritzalis, D., Furnell, S., Theoharidou, M. (eds.) SEC 2012. IAICT, vol. 376, pp. 388–399. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30436-1_32
Acknowledgement
This work has been funded in part by NSF grant CNS-1816495. We thank the anonymous reviewers for their valuable and constructive comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, H., Zheng, Z. (2023). Robust Moving Target Defense Against Unknown Attacks: A Meta-reinforcement Learning Approach. In: Fang, F., Xu, H., Hayel, Y. (eds) Decision and Game Theory for Security. GameSec 2022. Lecture Notes in Computer Science, vol 13727. Springer, Cham. https://doi.org/10.1007/978-3-031-26369-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-26369-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-26368-2
Online ISBN: 978-3-031-26369-9
eBook Packages: Computer ScienceComputer Science (R0)