Skip to main content

Advertisement

Springer Nature Link
Log in

A reinforcement learning malware detection model based on heterogeneous information network path representation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

With the significant increase of Android malware, the APP privacy data leakage incidents occur frequently, which poses a great threat to user property and information security. Specifically, the new malware has the characteristics of high evolution rate and diverse variants, leading to the fact that the current malware detection methods still have three key problems: (1) Difficulty in acquiring Android sample structural features; (2) Weakly in representing malware behavior structure; (3) Poor robustness of the detection model. To address the above limitations, we propose a new malware detection framework MPRLDroid with reinforcement learning. First of all, the MPRLDroid model extracts the Android APP structural features and constructs the heterogeneous information network data based on the semantic call structure between APP, API and permission. Subsequently, the model utilizes reinforcement learning to adaptively generate a meta-path for each sample and combines it with a graph attention network to effectively represent the graph of nodes. Finally, the low-dimensional graph node vector data is brought into the downstream detection task for classification, where the performance change of the classification result is used as a reward function for reinforcement learning. The experimental results demonstrate that the MPRLDroid model, when integrated with reinforcement learning, outperforms the baseline models in terms of performance, and its detection model exhibits greater robustness compared to other models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Data Availability

The data supporting the findings of this study are available from the authors upon request.

Notes

  1. https://androguard.readthedocs.io/en/latest/

  2. https://virusshare.com/about

  3. https://androzoo.uni.lu/

  4. https://www.virustotal.com/gui/home

  5. https://github.com/ClaudiuGeorgiu/Obfusc

References

  1. Statcounter (2023) Mobile operating system market share worldwide. In: https://gs.statcounter.com/os-market-share/mobile/worldwide, pp. 1–2

  2. DAVID C (2023) Android mobile market statistics. In: https://www.Businessofapps.com/data/android-statistics/, pp. 1–2

  3. Isohara T, Takemori K (2011) Kubota A (2011) Kernel-based behavior analysis for android malware detection. In: Wang Y, Cheung Y, Guo P, Wei Y (eds) Seventh International Conference on Computational Intelligence and Security. CIS 2011, Sanya, Hainan, China, December 3–4, pp 1011–1015

    Google Scholar 

  4. Kramer S, Bradfield JC (2010) A general definition of malware. J Comput Virol 6(2):105–114

    Article  Google Scholar 

  5. Kim AC, Kim S, Park WH, Lee DH (2014) Fraud and financial crime detection model using malware forensics. Multim. Tools Appl. 68(2):479–496

    Article  Google Scholar 

  6. Sami A, Yadegari B, Rahimi H, Peiravian N, Hashemi S, Hamzeh A (2010) Malware detection based on mining API calls. In: Shin, S.Y., Ossowski, S., Schumacher, M., Palakal, M.J., Hung, C. (eds.) Proceedings of the 2010 ACM Symposium on Applied Computing (SAC), Sierre, Switzerland, March 22-26, 2010, pp. 1020–1025

  7. Seraj S, Khodambashi S, Pavlidis M, Polatidis N (2022) Hamdroid: permission-based harmful android anti-malware detection using neural networks. Neural Comput Appl 34(18):15165–15174

    Article  Google Scholar 

  8. Ilham S, Ghadi A, Abdelhakim AB (2018) Permission based malware detection in android devices. In: Proceedings of the 3rd International Conference on Smart City Applications, SCA 2018, Tetouan, Morocco, October 10-11, 2018, pp. 1–6

  9. Uppal D, Sinha R, Mehra V, Jain V (2014) Malware detection and classification based on extraction of API sequences. In: 2014 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2014, Delhi, India, September 24-27, 2014, pp. 2337–2342

  10. Shankarapani MK, Ramamoorthy S, Movva RS, Mukkamala S (2011) Malware detection using assembly and API call sequences. J Comput Virol 7(2):107–119

    Article  Google Scholar 

  11. Pektas A, Acarman T (2020) Deep learning for effective android malware detection using API call graph embeddings. Soft Comput 24(2):1027–1043

    Article  Google Scholar 

  12. Zhou H, Zhang W, Wei F, Chen Y (2017) Analysis of android malware family characteristic based on isomorphism of sensitive API call graph. In: Second IEEE International Conference on Data Science in Cyberspace, DSC 2017, Shenzhen, China, June 26-29, 2017, pp. 319–327

  13. Li C, Cheng Z, Zhu H, Wang L, Lv Q, Wang Y, Li N, Sun D (2022) Dmalnet: Dynamic malware analysis based on API feature engineering and graph learning. Comput Secur 122:102872

    Article  Google Scholar 

  14. Hou S, Ye Y, Song Y, Abdulhayoglu M (2017) Hindroid: An intelligent android malware detection system based on structured heterogeneous information network. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017, pp. 1507–1515

  15. Hei Y, Yang R, Peng H, Wang L, Xu X, Liu J, Liu H, Xu J, Sun L (2021) HAWK: rapid android malware detection through heterogeneous graph attention networks. arXiv:2108.07548

  16. Wang X, Bo D, Shi C, Fan S, Ye Y, Yu PS (2023) A survey on heterogeneous graph embedding: Methods, techniques, applications and sources. IEEE Trans. Big Data 9(2):415–436

    Article  Google Scholar 

  17. Yang CJ, Xiao Y, Zhang Y, Sun Y, Han J (2022) Heterogeneous network representation learning: A unified framework with survey and benchmark. IEEE Trans Knowl Data Eng 34(10):4854–4873

    Article  Google Scholar 

  18. Amin M, Tanveer TA, Tehseen M, Khan M, Khan FA, Anwar S (2020) Static malware detection and attribution in android byte-code through an end-to-end deep system. Future Gener. Comput. Syst. 102:112–126

    Article  Google Scholar 

  19. Jeon S, Moon J (2020) Malware-detection method with a convolutional recurrent neural network using opcode sequences. Inf Sci 535:1–15

    Article  MathSciNet  Google Scholar 

  20. Ye Y, Chen L, Wang D, Li T, Jiang Q, Zhao M (2009) SBMDS: an interpretable string based malware detection system using SVM ensemble with bagging. J Comput Virol 5(4):283–293

    Article  Google Scholar 

  21. Arora A, Peddoju SK, Conti M (2020) Permpair: Android malware detection using permission pairs. IEEE Trans Inf Forensics Secur 15:1968–1982

    Article  Google Scholar 

  22. Li J, Sun L, Yan Q, Li Z, Srisa-an W, Ye H (2018) Significant permission identification for machine-learning-based android malware detection. IEEE Trans. Ind. Informatics 14(7):3216–3225

    Article  Google Scholar 

  23. You I, Yim K (2010) Malware obfuscation techniques: A brief survey. In: Proceedings of the Fifth International Conference on Broadband and Wireless Computing, Communication and Applications, BWCCA 2010, November 4-6, 2010, Fukuoka Institute of Technology, Fukuoka, Japan (In Conjunction with the 3PGCIC-2010 International Conference), pp. 297–300

  24. Ki Y, Kim E, Kim HK (2015) A novel approach to detect malware based on API call sequence analysis. Int. J. Distributed Sens. Networks 11:659101–16591019

    Article  Google Scholar 

  25. Gascon H, Yamaguchi F, Arp D, Rieck K (2013) Structural detection of android malware using embedded call graphs. In: Sadeghi, A., Nelson, B., Dimitrakakis, C., Shi, E. (eds.) AISec’13, Proceedings of the 2013 ACM Workshop on Artificial Intelligence and Security, Co-located with CCS 2013, Berlin, Germany, November 4, 2013, pp. 45–54

  26. Hassen M, Chan PK (2017) Scalable function call graph-based malware classification. In: Ahn, G., Pretschner, A., Ghinita, G. (eds.) Proceedings of the Seventh ACM Conference on Data and Application Security and Privacy, CODASPY 2017, Scottsdale, AZ, USA, March 22-24, 2017, pp. 239–248

  27. Hu X, Chiueh T, Shin KG (2009) Large-scale malware indexing using function-call graphs. In: Al-Shaer, E., Jha, S., Keromytis, A.D. (eds.) Proceedings of the 2009 ACM Conference on Computer and Communications Security, CCS 2009, Chicago, Illinois, USA, November 9-13, 2009, pp. 611–620

  28. Chen Z, Kang H, Yin S, Kim S (2017) Automatic ransomware detection and analysis based on dynamic API calls flow graph. In: Proceedings of the International Conference on Research in Adaptive and Convergent Systems, RACS 2017, Krakow, Poland, September 20-23, 2017, pp. 196–201

  29. Pektas A, Acarman T (2020) Deep learning for effective android malware detection using API call graph embeddings. Soft Comput 24(2):1027–1043

    Article  Google Scholar 

  30. Bruschi D, Martignoni L, Monga M (2006) Detecting self-mutating malware using control-flow graph matching. In: Büschkes, R., Laskov, P. (eds.) Detection of Intrusions and Malware & Vulnerability Assessment, Third International Conference, DIMVA 2006, Berlin, Germany, July 13-14, 2006, Proceedings. Lecture Notes in Computer Science, vol. 4064, pp. 129–143

  31. Yan J, Yan G, Jin D (2019) Classifying malware represented as control flow graphs using deep graph convolutional neural network. In: 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019, Portland, OR, USA, June 24-27, 2019, pp. 52–63

  32. Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep reinforcement learning: A brief survey. IEEE Signal Process Mag 34(6):26–38

    Article  Google Scholar 

  33. Mousavi SS, Schukat M, Howley E (2018) Deep reinforcement learning: An overview. arXiv:1806.08894

  34. Wu C, Shi J, Yang Y, Li W (2018) Enhancing machine learning based malware detection model by reinforcement learning. In: Proceedings of the 8th International Conference on Communication and Network Security, ICCNS 2018, Qingdao, China, November 02-04, 2018, pp. 74–78

  35. Fang Z, Wang J, Geng J, Kan X (2019) Feature selection for malware detection based on reinforcement learning. IEEE Access 7:176177–176187

    Article  Google Scholar 

  36. Wang Y, Stokes JW, Marinescu M (2020) Actor critic deep reinforcement learning for neural malware control. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 1005–1012

  37. Ebrahimi R, Pacheco J, Hu J, Chen H (2024) Learning contextualized action representations in sequential decision making for adversarial malware optimization. IEEE Transactions on Dependable and Secure Computing

  38. Al-Fawa’reh M, Abu-Khalaf J, Szewczyk P, Kang JJ (2023) Malbot-drl: Malware botnet detection using deep reinforcement learning in iot networks. IEEE Internet of Things Journal

  39. Zhong F, Hu P, Zhang G, Li H, Cheng X (2022) Reinforcement learning based adversarial malware example generation against black-box detectors. Comput Secur 121:102869

    Article  Google Scholar 

  40. Tian B, Jiang J, He Z, Yuan X, Dong L, Sun C (2024) Functionality-verification attack framework based on reinforcement learning against static malware detectors. IEEE Trans Inf Forensics Secur 19:8500–8514

    Article  Google Scholar 

  41. Anderson HS, Kharkar A, Filar B, Evans D, Roth P (2018) Learning to evade static PE machine learning malware models via reinforcement learning. arXiv:1801.08917

  42. Castro RL, Franz S, Rodosek GD (2021) AIMED-RL: exploring adversarial malware examples with reinforcement learning. In: Dong, Y., Kourtellis, N., Hammer, B., Lozano, J.A. (eds.) Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 12978, pp. 37–52

  43. Zhong F, Hu P, Zhang G, Li H, Cheng X (2022) Reinforcement learning based adversarial malware example generation against black-box detectors. Comput Secur 121:102869

    Article  Google Scholar 

  44. Wu Y, Li M, Zeng Q, Yang T, Wang J, Fang Z, Cheng L (2023) Droidrl: Feature selection for android malware detection with reinforcement learning. Comput Secur 128:103126

    Article  Google Scholar 

  45. Zhang L, Liu P, Choi Y, Chen P (2023) Semantics-preserving reinforcement learning attack against graph neural networks for malware detection. IEEE Trans. Dependable Secur. Comput. 20(2):1390–1402

    Article  Google Scholar 

  46. Zapzalka D, Salem S, Mohaisen D (2024) Semantics-preserving node injection attacks against gnn-based acfg malware classifiers. IEEE Transactions on Dependable and Secure Computing

  47. Onwuzurike L, Mariconti E, Andriotis P, Cristofaro ED, Ross GJ, Stringhini G (2019) Mamadroid: Detecting android malware by building markov chains of behavioral models (extended version). ACM Trans. Priv. Secur. 22(2):14–11434

    Article  Google Scholar 

  48. Gong L, Li Z, Qian F, Zhang Z, Chen QA, Qian Z, Lin H (2020) Liu Y (2020) Experiences of landing machine learning onto market-scale mobile malware detection. In: Bilas A, Magoutis K, Markatos EP, Kostic D, Seltzer MI (eds) EuroSys ’20: Fifteenth EuroSys Conference 2020. Heraklion, Greece, April 27–30, pp 2–1214

  49. Xie N, Zeng F, Qin X, Zhang Y, Zhou M (2018) Lv C (2018) Repassdroid: Automatic detection of android malware based on essential permissions and semantic features of sensitive apis. In: Pang J, Zhang C, He J, Weng J (eds) 2018 International Symposium on Theoretical Aspects of Software Engineering. TASE 2018, Guangzhou, China, August 29–31, pp 52–59

    Chapter  Google Scholar 

  50. Otterlo M, Wiering MA (2012) Reinforcement learning and markov decision processes. In: Wiering, M.A., Otterlo, M. (eds.) Reinforcement Learning. Adaptation, Learning, and Optimization, vol. 12, pp. 3–42

  51. Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Schuurmans, D., Wellman, M.P. (eds.) Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp. 2094–2100

  52. Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q (2015) LINE: large-scale information network embedding. In: Gangemi, A., Leonardi, S., Panconesi, A. (eds.) Proceedings of the 24th International Conference on World Wide Web, WWW 2015, Florence, Italy, May 18-22, 2015, pp. 1067–1077

  53. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon, I., Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998–6008

  54. Hou Y, Liu L, Wei Q, Xu X, Chen C (2017) A novel DDPG method with prioritized experience replay. In: 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017, Banff, AB, Canada, October 5-8, 2017, pp. 316–321

  55. Lashkari AH, Kadir AFA, Taheri L, Ghorbani AA (2018) Toward developing a systematic approach to generate benchmark android malware datasets and classification. In: 2018 International Carnahan Conference on Security Technology, ICCST 2018, Montreal, QC, Canada, October 22-25, 2018, pp. 1–7

  56. Imtiaz SI, Rehman S, Javed AR, Jalil Z, Liu X, Alnumay WS (2021) Deepamd: Detection and identification of android malware using high-efficient deep artificial neural network. Future Gener. Comput. Syst. 115:844–856

    Article  Google Scholar 

  57. Taheri L, Kadir AFA (2019) Lashkari AH (2019) Extensible android malware detection and family classification using network-flows and api-calls. In: Thomas GL, John M (eds) 2019 International Carnahan Conference on Security Technology. ICCST 2019, Chennai, India, October 1–3, pp 1–8

    Google Scholar 

  58. Sahin DÖ, Akleylek S, Kiliç E (2021) On the effect of k values and distance metrics in KNN algorithm for android malware detection. Adv. Data Sci. Adapt. Anal. 13(3 &4):2141001–1214100120

    Article  Google Scholar 

  59. Zulkifli A, Hamid IRA, Shah WM, Abdullah Z (2018) Android malware detection based on network traffic using decision tree algorithm. In: Ghazali, R., Deris, M.M., Nawi, N.M., Abawajy, J.H. (eds.) Recent Advances on Soft Computing and Data Mining - Proceedings of the Third International Conference on Soft Computing and Data Mining (SCDM 2018), Johor, Malaysia, February 06-07, 2018. Advances in Intelligent Systems and Computing, vol. 700, pp. 485–494

  60. Alam MS, Vuong ST (2013) Random forest classification for detecting android malware. In: 2013 IEEE International Conference on Green Computing and Communications (GreenCom) and IEEE Internet of Things (iThings) and IEEE Cyber, Physical and Social Computing (CPSCom), Beijing, China, August 20-23, 2013, pp. 663–669

  61. Grover A, Leskovec J (2016) node2vec: Scalable feature learning for networks. In: Krishnapuram B, Shah M, Smola AJ, Aggarwal CC, Shen D, Rastogi R. (eds.) Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 855–864

  62. Perozzi B, Al-Rfou R (2014) Skiena S (2014) Deepwalk: online learning of social representations. In: Macskassy SA, Perlich C, Leskovec J, Wang W, Ghani R (eds) The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’14, New York, NY, USA - August 24–27, pp 701–710

    Chapter  Google Scholar 

  63. Velickovic P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2017) Graph attention networks. arXiv:1710.10903

  64. Gao H, Cheng S, Zhang W (2021) Gdroid: Android malware detection and classification with graph convolutional network. Comput Secur 106:102264

  65. Wang X, Ji H, Shi C, Wang B, Ye Y, Cui P (2019) Yu PS (2019) Heterogeneous graph attention network. In: Liu L, White RW, Mantrach A, Silvestri F, McAuley JJ, Baeza-Yates R, Zia L (eds) The World Wide Web Conference. WWW 2019, San Francisco, CA, USA, May 13–17, pp 2022–2032

    Chapter  Google Scholar 

  66. Yang Z, Yang D, Dyer C, He X, Smola AJ, Hovy EH (2016) Hierarchical attention networks for document classification. In: Knight K, Nenkova A, Rambow O. (eds.) NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pp. 1480–1489

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their valuable comments on this paper, which helped us identify problems and improve the quality of the paper.

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory of Computer Software Testing & Evaluating, 201112, Shanghai, China

    Kang Yang, Lizhi Cai, Jianhua Wu, Zhenyu Liu & Meng Zhang

  2. Shanghai Development Center of Computer Software Technology, 201112, Shanghai, China

    Kang Yang, Lizhi Cai, Jianhua Wu, Zhenyu Liu & Meng Zhang

Authors
  1. Kang Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Lizhi Cai
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jianhua Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zhenyu Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Meng Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Kang Yang: Conceptualization, Methodology, Writing original draft. Lizhi Cai: Conceptualization, Methodology, Validation, Software. Jianhua Wu: Software, Validation, Writing. Zhenyu Liu: Supervision, Investigation, Writing. Meng Zhang: Resources, Writing.

Corresponding author

Correspondence to Lizhi Cai.

Ethics declarations

Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical and informed consent for data used

This study did not involve human or animal subjects, and thus, no ethical approval was required. The study protocol adhered to the guidelines established by the journal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, K., Cai, L., Wu, J. et al. A reinforcement learning malware detection model based on heterogeneous information network path representation. Appl Intell 55, 538 (2025). https://doi.org/10.1007/s10489-025-06417-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-025-06417-1

Keywords