Abstract
This paper presents an evaluation of the red team automation tool Lore in two live-fire cyber defense exercises (CDX). During the CDXs, Lore and manual “red” teams subjected 72 network security analysts (i.e., defenders; the “blue” side) to various threats such as software exploits and shell commands. Ten hypotheses related to how the actions by manual red teams and Lore are perceived and managed by the security analysts are examined. Evaluations were made by studying the subjective judgements of the analysts and by comparing the objective ground truth to their submitted incident reports. The results show that none of the null hypotheses could be rejected. In other words, the security analysts could not tell the difference between the actions made by the manual red team and those made by Lore, and their performance was similar regardless of the source of the threats.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
CCDCOE, “Locked Shields” (2022), https://ccdcoe.org/exercises/locked-shields/.
- 2.
See the National Guard Bureau home page for further information, https://www.nationalguard.mil/.
- 3.
NIST, “NICE Framework Resource Center”, https://www.nist.gov/itl/applied-cybersecurity/nice/nice-framework-resource-center.
References
Abbott, R.G., McClain, J., Anderson, B., Nauer, K., Silva, A., Forsythe, C.: Automated performance assessment in cyber training exercises, p. 7
Andrew, A., Spillard, S., Collyer, J., Dhir, N.: Developing optimal causal cyber-defence agents via cyber security simulation. arXiv preprint arXiv:2207.12355 (2022)
Bashir, M., Lambert, A., Guo, B., Memon, N., Halevi, T.: Cybersecurity competitions: the human angle 13(5), 74–79. https://doi.org/10.1109/MSP.2015.100
Delacre, M., Lakens, D., Leys, C.: Why psychologists should by default use Welch’s t-test instead of student’s t-test. 30(1), 92–101. https://doi.org/10.5334/irsp.82, http://www.rips-irsp.com/articles/10.5334/irsp.82/
Dutta, A., Chatterjee, S., Bhattacharya, A., Halappanavar, M.: Deep reinforcement learning for cyber system defense under dynamic adversarial uncertainties. arXiv preprint arXiv:2302.01595 (2023)
Gustafsson, T., Almroth, J.: Cyber range automation overview with a case study of CRATE. In: Asplund, M., Nadjm-Tehrani, S. (eds.) NordSec 2020. LNCS, vol. 12556, pp. 192–209. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-70852-8_12
Henshel, D.S., et al.: Predicting proficiency in cyber defense team exercises. In: MILCOM 2016 - 2016 IEEE Military Communications Conference, pp. 776–781 (2016). https://doi.org/10.1109/MILCOM.2016.7795423
Holm, H.: Lore A Red Team Emulation Tool, p. 1.https://doi.org/10.1109/TDSC.2022.3160792
Holm, H., Sommestad, T.: SVED: scanning, vulnerabilities, exploits and detection. In: 2016 IEEE Military Communications Conference, pp. 976–981. IEEE (2016)
Li, L., Fayad, R., Taylor, A.: Cygil: a cyber gym for training autonomous agents over emulated network systems. arXiv preprint arXiv:2109.03331 (2021)
Lif, P., Varga, S., Wedlin, M., Lindahl, D., Persson, M.: Evaluation of information elements in a cyber incident report. In: 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS &PW), pp. 17–26. IEEE (2020)
Maennel, K., Ottis, R., Maennel, O.: Improving and measuring learning effectiveness at cyber defense exercises. In: Lipmaa, H., Mitrokotsa, A., Matulevičius, R. (eds.) NordSec 2017. LNCS, vol. 10674, pp. 123–138. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70290-2_8
McIntosh, S.E.: The wingman-philosopher of mig alley: John boyd and the ooda loop. Air Power History 58(4), 24–33 (2011). http://www.jstor.org/stable/26276108
Miehling, E., Rasouli, M., Teneketzis, D.: Optimal defense policies for partially observable spreading processes on Bayesian attack graphs. In: Proceedings of the Second ACM Workshop on Moving Target Defense, pp. 67–76 (2015)
Miller, D., Alford, R., Applebaum, A., Foster, H., Little, C., Strom, B.: Automated adversary emulation: a case for planning and acting with unknowns. Technical report, MITRE CORP MCLEAN VA MCLEAN (2018)
Mäses, S., Hallaq, B., Maennel, O.: Obtaining Better Metrics for Complex Serious Games Within Virtualised Simulation Environments, p. 9
Nhu, N.X., Nghia, T.T., Quyen, N.H., Pham, V.H., Duy, P.T., et al.: Leveraging deep reinforcement learning for automating penetration testing in reconnaissance and exploitation phase. In: 2022 RIVF International Conference on Computing and Communication Technologies, pp. 41–46. IEEE (2022)
Rajivan, P., Moriano, P., Kelley, T., Camp, L.J.: What can johnny do?–Factors in an end-user expertise instrument. In: HAISA, pp. 199–208 (2016)
Sarraute, C., Buffet, O., Hoffmann, J.: Penetration testing== pomdp solving? arXiv preprint arXiv:1306.4714 (2013)
Stupp, C.: Sweden tests cyber defenses as war and nato bid raise security risks. https://www.wsj.com/articles/sweden-tests-cyber-defenses-as-war-and-nato-bid-raise-security-risks-11663925402
Sultana, M., Taylor, A., Li, L.: Autonomous network cyber offence strategy through deep reinforcement learning. In: Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, vol. 11746, pp. 490–502. SPIE (2021)
Zilberman, P., Puzis, R., Bruskin, S., Shwarz, S., Elovici, Y.: Sok: a survey of open-source threat emulators. arXiv preprint arXiv:2003.01518 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 IFIP International Federation for Information Processing
About this paper
Cite this paper
Holm, H., Reuben, J. (2024). Evaluation of a Red Team Automation Tool in Live Cyber Defence Exercises. In: Meyer, N., Grocholewska-Czuryło, A. (eds) ICT Systems Security and Privacy Protection. SEC 2023. IFIP Advances in Information and Communication Technology, vol 679. Springer, Cham. https://doi.org/10.1007/978-3-031-56326-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-56326-3_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56325-6
Online ISBN: 978-3-031-56326-3
eBook Packages: Computer ScienceComputer Science (R0)
