Abstract
API-call sequence, a significant dynamic feature of the software, is widely applied to malware detection. Unfortunately, native approaches to API-call analysis are time-consuming and cause heavy performance penalties. To improve the efficiency of API-call analysis, this paper proposes a novel dynamic analysis approach named SeqTrace based on Intel Process Trace (PT) and Virtual Machine Introspection (VMI) technologies. First, we propose an API-call Tracing approach based on the Intel PT feature of the CPU. It leverages Intel PT to trace the execution of analyzed samples and logs relative information of their API calls with slight overhead. Then, to efficiently translate the semantics of API calls from logged information, we design Semantic Decoder based on VMI technology. Moreover, we implement a prototype API called SeqTrace on the QEMU/KVM platform and evaluate it through a set of experiments. Compared with previous approaches, the experimental results show that SeqTrace achieves API-call sequence tracing with fine-grained semantic information and reduces the tracing overhead by more than 80%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Steven, H., Anil, S.: Intrusion detection using sequences of system calls. J. Comput. Secur. 6(3), (1999)
Amin, K., Sajjad A., Collin, M., William, R., Engin, K.: UNVEIL: a large-scale, automated approach to detecting ransomware. In: 25th USENIX Security Symposium (USENIX Security 2016), pp. 757–772 (2016)
Thomas, N.-K., Max W.: Semi-supervised classification with graph convolutional networks. arXiv preprint, arXiv:1609.02907 (2016)
Rosenberg, I., Shabtai, A., Rokach, L., Elovici, Y.: Generic black-box end-to-end attack against state of the art API call based malware classifiers. In: Bailey, M., Holz, T., Stamatogiannakis, M., Ioannidis, S. (eds.) RAID 2018. LNCS, vol. 11050, pp. 490–510. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00470-5_23
Fadadu, F., Handa, A., Kumar, N., Shukla, S.K.: Evading API call sequence based malware classifiers. In: Zhou, J., Luo, X., Shen, Q., Xu, Z. (eds.) ICICS 2019. LNCS, vol. 11999, pp. 18–33. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-41579-2_2
Bergenholtz, E., Casalicchio, E., Ilie, D., Moss, A.: Detection of metamorphic malware packers using multilayered LSTM networks. In: Meng, W., Gollmann, D., Jensen, C.D., Zhou, J. (eds.) ICICS 2020. LNCS, vol. 12282, pp. 36–53. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61078-4_3
Binghui, W., Zhenqiang, G.: Attacking graph-based classification via manipulating the graph structure. In: 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 2023–2040 (2019)
Pfoh, J., Schneider, C., Eckert, C., et al.: Nitro: hardware-based system call tracing for virtual machines. In: International Workshop on Security, pp. 96–112 (2011)
Holz, T., Freiling, F., Willems, C.: Toward automated dynamic malware analysis using CWSandbox. IEEE Secur. Priv. 5(2), 32–39 (2007)
Bojan, J.: A Not-So-Common Cold: Malware Statistics in 2021. March 2021. https://dataprot.net/statistics/malware-statistics
Garfinkel, T., Rosenblum, M.: A virtual machine introspection based architecture for intrusion detection. In: 10th Network and Distributed System Symposium (NDSS 2003), San Diego, CA, USA (2003)
Bauman, E., Ayoade, G., Lin, Z.: A survey on hypervisor-based monitoring: approaches, applications, and evolutions. ACM Comput. Surv. 48(1), 1001–1033 (2015)
Intel 64 and IA-32 architectures software developer’s manual. (2016)
Payne, B.-D.: Simplifying virtual machine introspection using LibVMI. In: Technical Reports SAND2012-7818, Sandia National Laboratories (2012)
Sergej, S., Cornelius, A., Robert, G., Sebastian, S., Thorsten, H.: kAFL: hardware-assisted feedback fuzzing for OS kernels. In: 26th USENIX Conference on Security Symposium, Vancouver, BC, pp. 167–182. USENIX Association (2017)
Ding, Z., Cui, L., Fei, H., et al.: A high-efficiency and comprehensive dynamic behavior analysis system for malware based on hardware virtualization. In: 22nd International Conference on High Performance Computing and Communications; 18th International Conference on Smart City; 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), IEEE (2020)
HD Tune Pro, www.hdtune.com
Fritz Chess, www.jens-hartmann.at/Fritzmarks/
Bohme, M., Pham, V.-T., Roychoudhury, A.: Coverage-based greybox fuzzing as markov chain. IEEE Trans. Software Eng. 45(5), 489–506 (2019)
Cha, S.-K., Woo, M., Brumley, D.: Program-adaptive mutational fuzzing. In: IEEE Symposium on Security and Privacy, pp. 725–741. IEEE (2015)
Sanjay, R., Vivek, J., Ashish, K., Lucian, C., Cristiano, G., Herbert, B.: VUzzer: application-aware evolutionary fuzzing. In: 24th Network and Distributed System Symposium (NDSS 2017), San Diego, CA, USA (2017)
Ge, X., Talele, N., Payer, M., et al.: IEEE European Symposium on Security and Privacy. Fine-grained control-flow integrity for kernel software, IEEE (2016)
Vishwath, M., Per, L., Stefan, B., Kevin, W.-H., Michael, F.: Opaque control-flow integrity. In: 22th Network and Distributed System Symposium (NDSS 2015), San Diego, CA, USA (2015)
Carlini, N., Barresi, A., Payer, M., Wagner, D., Gross, T.-R.: Control-flow bending: on the effectiveness of control-flow integrity. In: 24th USENIX Conference on Security Symposium, Washington, D.C., pp. 161–176. USENIX Association (2015)
Payer, M., Barresi, A., Gross, T.R.: Fine-grained control-flow integrity through binary hardening. In: Almgren, M., Gulisano, V., Maggi, F. (eds.) DIMVA 2015. LNCS, vol. 9148, pp. 144–164. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20550-2_8
Wang, M., Yin, H., Bhaskar, A.-V., Su, P., Feng, D.: Binary code continent: finer-grained control flow integrity for stripped binaries. In: 31st Annual Computer Security Applications Conference (ACSAC 2015), Los Angeles, CA, USA, pp. 331–340 (2015)
Gu, Y., Zhao, Q., Zhang, Y., Lin, Z.: PT-CFI: transparent backward-edge control flow violation detection using intel processor trace. In: 7th ACM on Conference on Data and Application Security and Privacy, Scottsdale, Arizona, USA, pp. 173–184 (2017)
Ge, X., Cui, W., Jaeger, T.: GRIFFIN guarding control flows using Intel Processor Trace. In: 22nd International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2017), Xi’an, China, pp. 585–598 (2017)
Wang, X.-R., Liu, Y.-T., Chen, H.-B.: Transparent protection of kernel module against ROP with Intel processor trace. J. Software 29(5), 1333–1347 (2018)
Alazab, M., Layton, R., Venkataraman, S., Watters, P.: Malware detection based on structure detection based on structural and behaal and behavioural features of API calls. In: proceedings of the 2010 International Cyber Resilience Conference, pp. 1–10 (2010)
Sami, A., Yadegari, B., Rahimi, H., Peiravian, N., Hashemi, S., Hamze, A.: Malware detection based on mining API calls. In: proceedings of the 2010 ACM Symposium on Applied Computing (SAC 2010), New York, USA, pp. 1020–1025 (2010)
Wang, C., Pang, J., Zhao, R., Liu, X.: Using API sequence and Bayes algorithm to detect suspicious behavior. In: 2009 International Conference on Communication Software and Networks, pp. 544–548 (2009)
Oktavianto, D., Muhardianto, I.: Cuckoo Malware Analysis. Packt Publishing Ltd (2013)
Shi, B., Cui, L., Li, B., Liu, X., Hao, Z., Shen, H.: ShadowMonitor: an effective In-VM monitoring framework with hardware-enforced isolation. In: Research in Attacks, Intrusions, and Defenses, pp. 670–690 (2018)
Bryan, D.-P., Carbone, M., Lee, W., et al.: Secure and Flexible Monitoring of Virtual Machines. In: 23th Annual Computer Security Applications Conference, pp. 385–397, ACM (2007)
Dinaburg, A., Royal, P., Sharif, M., et al.: Ether: malware analysis via hardware virtualization extensions. In: 15th ACM conference on Computer and Communications Security, pp. 51–62 (2008)
Wang, C., Hao, Z., Yun, X.: NOR: towards non-intrusive, real-time and OS-agnostic introspection for virtual machines in cloud environment. In: Chen, X., Lin, D., Yung, M. (eds.) Inscrypt 2017. LNCS, vol. 10726, pp. 500–517. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75160-3_29
Tamas, K.-L., Steve, M., Bryan, D.-P., et al.: Scalability, fidelity and stealth in the DRAKVUF dynamic malware analysis system. In: 30th Annual Computer Security Applications Conference, pp. 386–395 (2014)
VirusSign, www.virussign.com/index.html
Acknowledgments
This work is supported by the National Natural Science Foundation of China (grant no. 62072453, 61972392 ), Youth Innovation Promotion Association of the Chinese Academy of Sciences (no. 2020164).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Ding, Z. et al. (2023). SeqTrace: API Call Tracing Based on Intel PT and VMI for Malware Detection. In: Meng, W., Lu, R., Min, G., Vaidya, J. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2022. Lecture Notes in Computer Science, vol 13777. Springer, Cham. https://doi.org/10.1007/978-3-031-22677-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-22677-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22676-2
Online ISBN: 978-3-031-22677-9
eBook Packages: Computer ScienceComputer Science (R0)