Skip to main content
Springer Nature Link
Log in

Attention Mechanism Based Adversarial Attack Against Deep Reinforcement Learning

  • Conference paper
  • First Online:
Security, Privacy, and Anonymity in Computation, Communication, and Storage (SpaCCS 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12382))

  • 1362 Accesses

Abstract

Deep reinforcement learning (DRL) aims to maximize long-term future rewards to achieve specific goals by learning polices based on deep learning models. However, existing research has found that machine learning models are vulnerable to maliciously craft adversarial examples, so does the DRL since it uses deep model to learn policies. Usually gradient information is adopted to generate adversarial perturbation on the clean observation states to fail DRL. In order to develop a novel attack method for further defect detection of DRL, we propose a novel attention mechanism based adversarial attack. Instead of gradient information, we make full use of hidden features extracted in the DRL by attention operations to generate more effective adversarial examples. Both channel attention and pixel attention are applied to extract feature to modify the clean state to an adversarial one. Deep Q-Learing Network (DQN), one of the state-of-the-art DRL models, is utilized as the target model to train Flappybird game environment to guarantee continuous running and high success rate. Comprehensive attack experiments are carried out on DQN to testify the attack performance in aspects of reward and loss convergence.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    MATH  Google Scholar 

  2. Peters, M., Goltz, J., Wiedenmann, S., Mundt, T.: Using machine learning to find anomalies in field bus network traffic. In: Wang, G., Feng, J., Bhuiyan, M.Z.A., Lu, R. (eds.) SpaCCS 2019. LNCS, vol. 11611, pp. 336–353. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24907-6_26

    Chapter  Google Scholar 

  3. Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)

  4. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)

  5. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937 (2016)

    Google Scholar 

  6. Huang, S., Papernot, N., Goodfellow, I., Duan, Y., Abbeel, P.: Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284 (2017)

  7. Bansal, T., Pachocki, J., Sidor, S., Sutskever, I., Mordatch, I.: Emergent complexity via multi-agent competition. arXiv preprint arXiv:1710.03748 (2017)

  8. Bai, X., Niu, W., Liu, J., Gao, X., Xiang, Y., Liu, J.: Adversarial examples construction towards white-box Q table variation in DQN pathfinding training. In: 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC), pp. 781–787. IEEE (2018)

    Google Scholar 

  9. Chen, T., et al.: Gradient band-based adversarial training for generalized attack immunity of A3C path finding. arXiv preprint arXiv:1807.06752 (2018)

  10. Inkawhich, M., Chen, Y., Li, H.: Snooping attacks on deep reinforcement learning. arXiv preprint arXiv:1905.11832 (2019)

  11. Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: International Conference on Machine Learning, pp. 1889–1897 (2015)

    Google Scholar 

  12. Lin, Y.-C., Liu, M.-Y., Sun, M., Huang, J.-B.: Detecting adversarial attacks on neural network policies with visual foresight. arXiv preprint arXiv:1710.00814 (2017)

  13. Behzadan, V., Munir, A.: Vulnerability of deep reinforcement learning to policy induction attacks. In: Perner, P. (ed.) MLDM 2017. LNCS (LNAI), vol. 10358, pp. 262–275. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-62416-7_19

    Chapter  Google Scholar 

  14. Liu, J., et al.: A method to effectively detect vulnerabilities on path planning of VIN. In: Qing, S., Mitchell, C., Chen, L., Liu, D. (eds.) ICICS 2017. LNCS, vol. 10631, pp. 374–384. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89500-0_33

    Chapter  Google Scholar 

  15. Zhao, Y., Shumailov, I., Cui, H., Gao, X., Mullins, R., Anderson, R.: Blackbox attacks on reinforcement learning agents using approximated temporal information. arXiv preprint arXiv:1909.02918 (2019)

  16. Ebeling-Rump, M., Kao, M., Hervieux-Moore, Z.: Applying q-learning to flappy bird. Department of Mathematics and Statistics, Queens University (2016)

    Google Scholar 

  17. Ma, K., Jiang, R., Dong, M., Jia, Y., Li, A.: Neural network based web log analysis for web intrusion detection. In: Wang, G., Atiquzzaman, M., Yan, Z., Choo, K.-K.R. (eds.) SpaCCS 2017. LNCS, vol. 10658, pp. 194–204. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-72395-2_19

    Chapter  Google Scholar 

  18. Yan, X., Cui, B., Li, J.: Malicious domain name recognition based on deep neural networks. In: Wang, G., Chen, J., Yang, L.T. (eds.) SpaCCS 2018. LNCS, vol. 11342, pp. 497–505. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-05345-1_43

    Chapter  Google Scholar 

  19. Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

  20. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  21. Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57. IEEE (2017)

    Google Scholar 

  22. Kos, J., Song, D.: Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452 (2017)

  23. Lin, Y.-C., Hong, Z.-W., Liao, Y.-H., Shih, M.-L., Liu, M.-Y., Sun, M.: Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06748 (2017)

  24. Xiang, Y., Niu, W., Liu, J., Chen, T., Han, Z.: A PCA-based model to predict adversarial examples on q-learning of path finding. In: 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC), pp. 773–780. IEEE (2018)

    Google Scholar 

  25. Chatterjee, P., Yalchin, A., Shelton, J., Roy, K., Yuan, X., Edoh, K.D.: Presentation attack detection using wavelet transform and deep residual neural net. In: Wang, G., Feng, J., Bhuiyan, M.Z.A., Lu, R. (eds.) SpaCCS 2019. LNCS, vol. 11637, pp. 86–94. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24900-7_7

    Chapter  Google Scholar 

  26. Tretschk, E., Oh, S.J., Fritz, M.: Sequential attacks on agents for long-term adversarial goals. arXiv preprint arXiv:1805.12487 (2018)

  27. Russo, A., Proutiere, A.: Optimal attacks on reinforcement learning policies. arXiv preprint arXiv:1907.13548 (2019)

  28. Gu, Z., Jia, Z., Choset, H.: Adversary A3C for robust reinforcement learning. arXiv preprint arXiv:1912.00330 (2019)

  29. Fischer, M., Mirman, M., Vechev, M.: Online robustness training for deep reinforcement learning. arXiv preprint arXiv:1911.00887 (2019)

  30. Behzadan, V., Munir, A.: Mitigation of policy manipulation attacks on deep q-networks with parameter-space noise. In: Gallina, B., Skavhaug, A., Schoitsch, E., Bitsch, F. (eds.) SAFECOMP 2018. LNCS, vol. 11094, pp. 406–417. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99229-7_34

    Chapter  Google Scholar 

  31. Havens, A., Jiang, Z., Sarkar, S.: Online robust policy learning in the presence of unknown adversaries. In: Advances in Neural Information Processing Systems, pp. 9916–9926 (2018)

    Google Scholar 

  32. Lee, X.Y., Havens, A., Chowdhary, G., Sarkar, S.: Learning to cope with adversarial attacks. arXiv preprint arXiv:1906.12061 (2019)

  33. Lütjens, B., Everett, M., How, J.P.: Certified adversarial robustness for deep reinforcement learning. arXiv preprint arXiv:1910.12908 (2019)

  34. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  35. Dong, Y., et al.: Boosting adversarial attacks with momentum. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9185–9193 (2018)

    Google Scholar 

  36. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)

Download references

Acknowledgment

This research was supported by the National Natural Science Foundation of China under Grant No. 62072406, the Natural Science Foundation of Zhejiang Provincial under Grant No. LY19F020025, the Major Special Funding for “Science and Technology Innovation 2025” in Ningbo under Grant No. 2018B10063.

Author information

Authors and Affiliations

  1. Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou, 310023, China

    Jinyin Chen

  2. The College of Information Engineering, Zhejiang University of Technology, Hangzhou, 310023, China

    Jinyin Chen, Xueke Wang, Yan Zhang & Haibin Zheng

  3. The College of Computer Science and Technology, Zhejiang University, Hangzhou, 310058, China

    Shouling Ji

Authors
  1. Jinyin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xueke Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haibin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shouling Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinyin Chen .

Editor information

Editors and Affiliations

  1. Guangzhou University, Guangzhou, China

    Guojun Wang

  2. Aeronautics and Astronautics, Nanjing University, Nanjing, China

    Bing Chen

  3. Computer Science, Georgia State University, Atlanta, GA, USA

    Wei Li

  4. College of Science and Engineering, Qatar Foundation Education City, Doha, Qatar

    Roberto Di Pietro

  5. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Xuefeng Yan

  6. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Hao Han

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, J., Wang, X., Zhang, Y., Zheng, H., Ji, S. (2021). Attention Mechanism Based Adversarial Attack Against Deep Reinforcement Learning. In: Wang, G., Chen, B., Li, W., Di Pietro, R., Yan, X., Han, H. (eds) Security, Privacy, and Anonymity in Computation, Communication, and Storage. SpaCCS 2020. Lecture Notes in Computer Science(), vol 12382. Springer, Cham. https://doi.org/10.1007/978-3-030-68851-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68851-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68850-9

  • Online ISBN: 978-3-030-68851-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics