Skip to main content
SpringerLink
Log in

Simulating Deception for Web Applications Using Reinforcement Learning

  • Conference paper
  • First Online:
Computer Security. ESORICS 2023 International Workshops (ESORICS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14399))

Included in the following conference series:

  • 160 Accesses

Abstract

Web applications are constantly under attack as the public-facing components of information systems. One defense mechanism is deception, which introduces deceptive components into the application to detect the attacks with high fidelity, while distracting attackers from the successful attack path.

One important challenge that hinders the widespread adoption of deception is the difficulty to assess its effectiveness. This often requires conducting human experiments, which can be both costly and impractical for every individual web application scenario. A recent solution proposed to address this issue for network-layer deception has been to use a Reinforcement Learning (RL) based framework to simulate an attacker in a network with deceptive elements.

In this paper, we extend this framework to simulate the different components of web applications and related deceptive strategies. We then conduct several experiments to understand how the different quantities and types of deceptive elements impact the time to detect the attacker. Our empirical findings reveal that a larger number of honeytokens impede the agent’s learning, and allows for earlier attack detection. We also demonstrate the impact of each honeytoken on the success rate of attack detection, and how the implementation of deceptive elements can affect the performance of the agent.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Betarte, G., Pardo, A., Martínez, R.: Web application attacks detection using machine learning techniques. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 1065–1072 (Dec 2018). https://doi.org/10.1109/ICMLA.2018.00174

  2. Brockman, G., et al.: Openai gym. CoRR (2016). http://arxiv.org/abs/1606.01540

  3. Caminero Fernández, G., Lopez-Martin, M., Carro, B.: Adversarial environment reinforcement learning algorithm for intrusion detection. Comput. Netw. 159, 96–109 (2019). https://doi.org/10.1016/j.comnet.2019.05.013

    Article  Google Scholar 

  4. Charpentier, A., Boulahia Cuppens, N., Cuppens, F., Yaich, R.: Deep Reinforcement Learning-Based Defense Strategy Selection. In: Proceedings of the 17th International Conference on Availability, Reliability and Security, pp. 1–11. ACM, Vienna Austria (Aug 2022). https://doi.org/10.1145/3538969.3543789

  5. El-Kosairy, A., Azer, M.A.: A New web deception system framework. In: 2018 1st International Conference on Computer Applications & Information Security (ICCAIS), pp. 1–10 (Apr 2018). https://doi.org/10.1109/CAIS.2018.8442027

  6. Elderman, R., Pater, J.J., L., S. Thie, A., M. Drugan, M., M. Wiering, M.: Adversarial Reinforcement Learning in a Cyber Security Simulation. In: Proceedings of the 9th International Conference on Agents and Artificial Intelligence, pp. 559–566. SCITEPRESS - Science and Technology Publications, Porto, Portugal (2017). https://doi.org/10.5220/0006197105590566, http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0006197105590566

  7. Erdődi, L., Sommervoll, A.A., Zennaro, F.M.: Simulating SQL injection vulnerability exploitation using Q-learning reinforcement learning agents. Journal of Information Security and Applications 61(C) (Sep 2021). https://doi.org/10.1016/j.jisa.2021.102903

  8. Even-Dar, E., Mansour, Y.: Learning Rates for Q-Learning. In: Goos, G., Hartmanis, J., Van Leeuwen, J., Helmbold, D., Williamson, B. (eds.) Computational Learning Theory, vol. 2111, pp. 589–604. Springer, Berlin Heidelberg, Berlin, Heidelberg (2001). https://doi.org/10.1007/3-540-44581-1_39, http://link.springer.com/10.1007/3-540-44581-1_39, series Title: Lecture Notes in Computer Science

  9. Gan, Y., et al.: An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems. In: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 3–18. ACM, Providence RI USA (Apr 2019). https://doi.org/10.1145/3297858.3304013, https://dl.acm.org/doi/10.1145/3297858.3304013

  10. Han, X., Kheir, N., Balzarotti, D.: Evaluation of Deception-Based Web Attacks Detection. In: Proceedings of the 2017 Workshop on Moving Target Defense, pp. 65–73. ACM, Dallas Texas USA (Oct 2017). https://doi.org/10.1145/3140549.3140555, https://dl.acm.org/doi/10.1145/3140549.3140555

  11. Han, X., Kheir, N., Balzarotti, D.: Deception techniques in computer security: a research perspective. ACM Comput. Surv. 51(4), 80 (2018). https://doi.org/10.1145/3214305

  12. van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. CoRR (2015). http://arxiv.org/abs/1509.06461

  13. Kunz, T., Fisher, C., La Novara-Gsell, J., Nguyen, C., Li, L.: A Multiagent CyberBattleSim for RL Cyber Operation Agents (Apr 2023). 10.48550/arXiv. 2304.11052, http://arxiv.org/abs/2304.11052, arXiv:2304.11052 [cs]

  14. Li, H., Guo, Y., Huo, S., Hu, H., Sun, P.: Defensive deception framework against reconnaissance attacks in the cloud with deep reinforcement learning. Sci. China Inf. Sci. 65(7), 170305 (Jul 2022). https://doi.org/10.1007/s11432-021-3462-4, https://link.springer.com/10.1007/s11432-021-3462-4

  15. Li, L., Fayad, R., Taylor, A.: CyGIL: A Cyber Gym for Training Autonomous Agents over Emulated Network Systems (Sep 2021). https://doi.org/10.48550/arXiv.2109.03331

  16. Li, Q., et al.: A hierarchical deep reinforcement learning model with expert prior knowledge for intelligent penetration testing. Computers & Security 132, 103358 (Sep 2023). https://doi.org/10.1016/j.cose.2023.103358, https://www.sciencedirect.com/science/article/pii/S0167404823002687

  17. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., et al.: Playing Atari with Deep Reinforcement Learning. NIPS Deep Learning Workshop 2013 (Dec 2013), http://arxiv.org/abs/1312.5602,arXiv: 1312.5602

  18. Reti, D., Elzer, K., Schotten, H.D.: SCANTRAP: Protecting Content Management Systems from Vulnerability Scanners with Cyber Deception and Obfuscation (Jan 2023). http://arxiv.org/abs/2301.10502arXiv:2301.10502 [cs]

  19. Sahin, M., Hebert, C., De Oliveira, A.S.: Lessons Learned from SunDEW: A Self Defense Environment for Web Applications. In: Proceedings 2020 Workshop on Measurements, Attacks, and Defenses for the Web. Internet Society, San Diego, CA (2020). https://doi.org/10.14722/madweb.2020.23005, https://www.ndss-symposium.org/wp-content/uploads/2020/02/23005.pdf

  20. Sahin, M., Hébert, C., Cabrera Lozoya, R.: An Approach to Generate Realistic HTTP Parameters for Application Layer Deception. In: Ateniese, G., Venturi, D. (eds.) Applied Cryptography and Network Security. vol. 13269, pp. 337–355. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-031-09234-3-17, https://link.springer.com/10.1007/978-3-031-09234-3_17, series Title: Lecture Notes in Computer Science

  21. Shashkov, A., Hemberg, E., Tulla, M., O’Reilly, U.M.: Adversarial agent-learning for cybersecurity: a comparison of algorithms. The Knowledge Engineering Review 38, e3 (Jan 2023). https://doi.org/10.1017/S0269888923000012, publisher: Cambridge University Press

  22. Standen, M., Lucas, M., Bowman, D., Richer, T.J., Kim, J., Marriott, D.: CybORG: A Gym for the Development of Autonomous Cyber Agents (Aug 2021). https://doi.org/10.48550/arXiv.2108.09118

  23. van der Stock, A., Glas, B., Smithline, N., Gigler, T.: Owasp Web Security Testing Guide v4.2. https://github.com/OWASP/wstg/releases/download/v4.2/wstg-v4.2.pdf (2014)

  24. van der Stock, A., Glas, B., Smithline, N., Gigler, T.: Owasp Appsensor project guide v2. https://owasp.org/www-pdf-archive/Owasp-appsensor-guide-v2.pdf (2015)

  25. van der Stock, A., Glas, B., Smithline, N., Gigler, T.: OWASP Top 10 project (2021). https://owasp.org/Top10/

  26. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (2018)

    Google Scholar 

  27. Team., M.D.R.: Cyberbattlesim. https://github.com/microsoft/cyberbattlesim (2021)

  28. Walter, E., Ferguson-Walter, K., Ridley, A.: Incorporating Deception into CyberBattleSim for Autonomous Defense. IJCAI-21 1st International Workshop on Adaptive Cyber Defense (Aug 2021), http://arxiv.org/abs/2108.13980arXiv:2108.13980 [cs]

  29. Wang, S., Pei, Q., Wang, J., Tang, G., Zhang, Y., Liu, X.: An Intelligent Deployment Policy for Deception Resources Based on Reinforcement Learning. IEEE Access 8, 35792–35804 (2020). https://doi.org/10.1109/ACCESS.2020.2974786, conference Name: IEEE Access

  30. Xin, W., Gengyu, W., Yixian, Y.: Web application vulnerability detection based on reinforcement learning. Int. J. Digital Content Technol. Appl. 6, 12–20 (2012). https://doi.org/10.4156/jdcta.vol6.issue10.2

    Article  Google Scholar 

  31. Yao, Q., Wang, Y., Xiong, X., Wang, P., Li, Y.: Adversarial decision-making for moving target defense: a multi-agent markov game and reinforcement learning approach. Entropy 25(4), 605 (Apr 2023). https://doi.org/10.3390/e25040605, https://www.mdpi.com/1099-4300/25/4/605, number: 4 Publisher: Multidisciplinary Digital Publishing Institute

  32. Zhang, L., Thing, V.L.L.: Three Decades of Deception Techniques in Active Cyber Defense - Retrospect and Outlook. Computers & Security 106, 102288 (Jul 2021). https://doi.org/10.1016/j.cose.2021.102288, http://arxiv.org/abs/2104.03594,arXiv:2104.03594 [cs]

  33. Zhu, M., Anwar, A.H., Wan, Z., Cho, J.H., Kamhoua, C., Singh, M.P.: Game-theoretic and machine learning-based approaches for defensive deception: a survey (May 2021). http://arxiv.org/abs/2101.10121arXiv:2101.10121 [cs]

Download references

Author information

Authors and Affiliations

  1. Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong

    Andrei Kvasov

  2. SAP Labs France, 805 Avenue du Dr Donat, 06259, Mougins, France

    Merve Sahin, Cedric Hebert & Anderson Santana De Oliveira

Authors
  1. Andrei Kvasov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Merve Sahin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cedric Hebert
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anderson Santana De Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrei Kvasov .

Editor information

Editors and Affiliations

  1. Norwegian University of Science and Technology, Gjøvik, Norway

    Sokratis Katsikas

  2. Norwegian Computing Center, Oslo, Norway

    Habtamu Abie

  3. University of Trento, Trento, Italy

    Silvio Ranise

  4. University of Genoa, Genoa, Italy

    Luca Verderame

  5. Consiglio Nazionale delle Ricerche (CNR), Genoa, Italy

    Enrico Cambiaso

  6. SINTEF A.S., Oslo, Norway

    Rita Ugarelli

  7. Instituto Superior de Engenharia do Porto, Porto, Portugal

    Isabel Praça

  8. Hong Kong Polytechnic University, Hong Kong, China

    Wenjuan Li

  9. Technical University of Denmark, Kongens Lyngby, Denmark

    Weizhi Meng

  10. University of Nottingham, Nottingham, UK

    Steven Furnell

  11. Norwegian University of Science and Technology, Gjøvik, Norway

    Basel Katt

  12. Norwegian Computing Center, Oslo, Norway

    Sandeep Pirbhulal

  13. Institute for Energy Technology (IFE), Halden, Norway

    Ankur Shukla

  14. University of Calabria, Rende, Italy

    Michele Ianni

  15. University of Verona, Verona, Italy

    Mila Dalla Preda

  16. The University of Texas at San Antonio, San Antonio, TX, USA

    Kim-Kwang Raymond Choo

  17. University of Lisbon, Lisbon, Portugal

    Miguel Pupo Correia

  18. University of Twente, Enschede, The Netherlands

    Abhishta Abhishta

  19. University of Amsterdam, Amsterdam, The Netherlands

    Giovanni Sileno

  20. Open University in the Netherlands, Heerlen, The Netherlands

    Mina Alishahi

  21. Robert Gordon University, Aberdeen, UK

    Harsha Kalutarage

  22. Osaka University, Osaka, Japan

    Naoto Yanai

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kvasov, A., Sahin, M., Hebert, C., De Oliveira, A.S. (2024). Simulating Deception for Web Applications Using Reinforcement Learning. In: Katsikas, S., et al. Computer Security. ESORICS 2023 International Workshops. ESORICS 2023. Lecture Notes in Computer Science, vol 14399. Springer, Cham. https://doi.org/10.1007/978-3-031-54129-2_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-54129-2_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-54128-5

  • Online ISBN: 978-3-031-54129-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation