Abstract
Kernel control-flow integrity (CFI) of virtual machines is very important to cloud security. VMI-based dynamic tracing and analyzing methods are promising options for checking kernel CFI in cloud. However, the CFI monitors based on tracing always work at instruction or branch level and result in serious virtual machine performance degradation. To meet the performance requirements in the cloud, we present a page-level dynamic VMI-based kernel CFI checking solution. We trace VM kernel execution at page level, which means that the in-page instruction execution cannot trigger our monitor. As a result, the tracing overhead can be greatly reduced. Based on page-level execution information, we propose two policies to describe the kernel control-flow so as to build the secure kernel control-flow database in the learning stage. In the monitoring stage, we compare runtime execution information with the secure database to check kernel CFI. To further reduce the monitoring overhead, we propose two performance optimization strategies. We implement the prototype on Xen and leverage hardware events to trace VM memory page execution. Then, we evaluate the effectiveness and performance of the prototype. The experimental results prove that our system has enough detection capability and the overhead is acceptable.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Brindha T, Shaji RS (2016) A secure transaction of cloud data using conditional source trust attributes encryption mechanism. Soft Computing, pp 1–10. doi:10.1007/s00500-016-2405-6
Brown A, Chase JS (2011) Trusted platform-as-a-service: a foundation for trustworthy cloud-hosted applications. In: Proceedings of the 3rd ACM workshop on Cloud computing security workshop, pp 15–20
Butt S, Lagar-Cavilla HA, Srivastava A, Ganapathy V (2012) Self-service cloud computing. In: Proceedings of the 2012 ACM conference on Computer and communications security, pp 253–264
Carbone M, Conover M, Montague B, Lee W (2012) Secure and robust monitoring of virtual machines through guest-assisted introspection. International workshop on recent advances in intrusion detection. Springer, Berlin, pp 22–41
Danger JL, Guilley S, Porteboeuf T, Praden F, Timbert M (2014) Hcode: hardware-enhanced real-time cfi. In: Proceedings of the 4th ACM program protection and reverse engineering workshop, p 6
Distorm. http://github.com/gdabah/distorm Accessed 03 May 2017
Du X, Xiao Y, Guizani M, Chen HH (2007) An effective key management scheme for heterogeneous sensor networks. Ad Hoc Netw 5(1):24–34
Du X, Guizani M, Xiao Y, Chen HH (2009) A routingDriven elliptic curve cryptography based key management scheme for heterogeneous sensor networks. IEEE Trans Wireless Commun 8(3):1223–1229
Du X, Rozenblit M, Shayman M (2011) Implementation and performance analysis of SNMP on a TLS/TCP base. In: Proceedings of the seventh IFIP/IEEE international symposium on integrated network management, pp 453–466
Garfinkel T, Rosenblum M et al (2003) A virtual machine introspection based architecture for intrusion detection. NDSS 3:191–206
Guide P (2010) Intel 64 and ia-32 architectures software developers manual
Hizver J, Chiueh Tc (2013) Cloud-based application whitelisting. In: 2013 IEEE 6th international conference on cloud computing, pp 636–643
Hofmann OS, Dunn AM, Kim S, Roy I, Witchel E (2011) Ensuring operating system kernel integrity with osck. ACM SIGARCH Comput Archit News 39:279–290
Huang HD, Lee CS, Wang MH, Kao HY (2014) It2fs-based ontology with soft-computing mechanism for malware behavior analysis. Soft Comput 18(2):267–284
Intel virtualization technology. http://www.intel.com/content/www/us/en/virtuali-zation/virtualization-technology/intel-virtualization-technology.html Accessed 03 May 2017
Li J, Wang Z, Bletsch T, Srinivasan D, Grace M, Jiang X (2011) Comprehensive and efficient protection of kernel control data. IEEE Trans Inf Forens Secur 6(4):1404–1417
Liang S, Du X (2014) Permission-Combination-based scheme for android mobile malware detection. In: Proceedings of IEEE international conference on communications (ICC), pp 2301–2306
Liao Z, Luo Y (2015) A stack-based lightweight approach to detect kernel-level rookits. In: 2015 IEEE international conference on progress in informatics and computing (PIC), pp 602–607
Malone C, Zahran M, Karri R (2011) Are hardware performance counters a cost effective way for integrity checking of programs. In: Proceedings of the 6th ACM workshop on Scalable trusted computing, pp 71–76
Martín A, Menéndez HD, Camacho D (2016) Mocdroid: multi-objective evolutionary classifier for android malware detection. Soft Comput. pp 1–11. doi:10.1007/s00500-016-2283-y
Petroni Jr NL, Hicks M (2007) Automated detection of persistent kernel control-flow attacks. In: Proceedings of the 14th ACM conference on computer and communications security, pp 103–115
Prakash A, Yin H, Liang Z (2013) Enforcing system-wide control flow integrity for exploit detection and diagnosis. In: Proceedings of the 8th ACM SIGSAC symposium on Information, computer and communications security, pp 311–322
Rhee J, Riley R, Xu D, Jiang X (2010) Kernel malware analysis with un-tampered and temporal views of dynamic kernel memory. International workshop on recent advances in intrusion detection. Springer, Berlin, pp 178–197
Shi W, Zhou H, Yuan J, Liang B (2014) Dcfi-checker: checking kernel dynamic control flow integrity with performance monitoring counter. China Commun 11(9):31–46
Srivastava A, Raj H, Giffin J, England P (2012) Trusted vm snapshots in untrusted cloud infrastructures. International workshop on recent advances in intrusion detection. Springer, Berlin, pp 1–21
Sysenter. http://wiki.osdev.org/SYSENTER Accessed 03 May 2017
The xen project. http://www.xenproject.org/ Accessed 03 May 2017
Vogl S, Eckert C (2012) Using hardware performance events for instruction-level monitoring on the x86 architecture. In: Proceedings of the 2012 European workshop on system security EuroSec, 12
Wang X, Karri R (2013) Numchecker: detecting kernel control-flow modifying rootkits by using hardware performance counters. In: 50th ACM/EDAC/IEEE conference on design automation (DAC), pp 1–7
Wei J, Payne BD, Giffin J, Pu C (2008) Soft-timer driven transient kernel control flow attacks and defense. In: Annual IEEE conference on computer security applications, pp 97–107
Willems C, Hund R, Fobian A, Felsch D, Holz T, Vasudevan A (2012) Down to the bare metal: using processor features for binary analysis. In: Proceedings of the 28th ACM annual computer security applications conference, pp 189–198
wook Baek H, Srivastava A, Van der Merwe J (2014) Cloudvmi: Virtual machine introspection as a cloud service. In: IEEE International Conference on Cloud Engineering (IC2E), pp 153–158
Xiao Y, Chen HH, Du X, Guizani M (2009) Streambased cipher feedback mode in wireless error channel. IEEE Trans Wireless Commun 8(2):622–626
Xiao Y, Rayi V, Sun B, Du X, Hu F, Galloway M (2007) A survey of key management schemes in wireless sensor networks. Comput Commun 30(11):2314–2341
Zawawi N, Hamdy M, Ghary R, Tolba M (2016) Realization of a data traceability and recovery service for a trusted authority service co-ordination within a cloud environment. Soft Comput 20(12):5039–5050
Zeng J, Fu Y, Lin Z (2015) Pemu: a pin highly compatible out-of-vm dynamic binary instrumentation framework. ACM SIGPLAN Not 50:147–160
Funding
This study was funded by Enterprise-University-Research Institute Cooperation Project of Guangdong Province, China (Grant No. 2016B090921001), and National Natural Science Foundation of China (Grants No. 61601146).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by V. Loia.
Rights and permissions
About this article
Cite this article
Zhan, D., Ye, L., Fang, B. et al. Checking virtual machine kernel control-flow integrity using a page-level dynamic tracing approach. Soft Comput 22, 7977–7987 (2018). https://doi.org/10.1007/s00500-017-2745-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-017-2745-x