Skip to main content
Log in

The self modifying code (SMC)-aware processor (SAP): a security look on architectural impact and support

  • Original Paper
  • Published:
Journal of Computer Virology and Hacking Techniques Aims and scope Submit manuscript

Abstract

Self modifying code (SMC) are code snippets that modify themselves at runtime. Malware use SMC to hide payloads and achieve persistence. Software-based SMC detection solutions impose performance penalties for real-time monitoring and do not benefit from runtime architectural information (cache invalidation or pipeline flush, for instance). We revisit SMC impact on hardware internals and discuss the implementation of an SMC detector at distinct architectural points. We consider three detection approaches: (i) existing hardware counters; (ii) block invalidation by the cache coherence protocol; (iii) the use of Memory Management Unit (MMU) information to control SMC execution. We compare the identified instrumentation points to highlight their strong and weak points. We also compare them to previous SMC detectors’ implementations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. We are hereafter referring to overhead to denote the runtime monitoring overhead, as the overhead of running detection routines is unavoidable to any AV solution.

  2. https://upx.github.io/.

References

  1. AMD: AMD Secure Virtual Machine Architecture Reference Manual (2008). https://www.mimuw.edu.pl/~vincent/lecture6/sources/amd-pacifica-specification.pdf

  2. ARM: Smc (2013). https://community.arm.com/processors/b/blog/posts/caches-and-self-modifying-code

  3. Babar, K., Khalid, F.: Generic unpacking techniques. In: International Conference on Computer, Control and Communication, pp. 1–6 (2009)

  4. Ballapuram, C.S., Sharif, A., Lee, H.H.S.: Exploiting access semantics and program behavior to reduce snoop power in chip multiprocessors. SIGARCH Comput. Archit. News 36(1), 60–69 (2008)

    Article  Google Scholar 

  5. Bonfante, G., Fernandez, J., Marion, J.Y., Rouxel, B., Sabatier, F., Thierry, A.: Codisasm: Medium scale concatic disassembly of self-modifying binaries with overlapping instructions. In: ACM SIGSAC Conference on Computer and Communications Security, pp. 745–756 (2015)

  6. Borello, J.M., Mé, L.: Code obfuscation techniques for metamorphic viruses. JICVHT 3, 211–220 (2008)

    Google Scholar 

  7. Botacin, M., de Geus, P., Grégio, A.: Enhancing branch monitoring for security purposes: from control flow integrity to malware analysis and debugging. ACM Trans. Priv. Secur. 21(1), 1–30 (2018)

    Article  Google Scholar 

  8. Cai, H., Shao, Z., Vaynberg, A.: Certified self-modifying code. SIGPLAN Not. 42(6), 66–77 (2007)

    Article  Google Scholar 

  9. Caserta, P., Zendra, O.: A tracing technique using dynamic bytecode instrumentation of java applications and libraries at basic block level. In: Proceedings of the 6th Workshop on Implementation, Compilation, Optimization of Object-Oriented Languages, Programs and Systems, ICOOOLPS’11, pp. 6:1–6:4. ACM, New York, NY, USA (2011). https://doi.org/10.1145/2069172.2069178

  10. Censier, F.: A new solution to coherence problems in multicache systems. IEEE Trans. Comput. C–27(12), 1112–1118 (1978). https://doi.org/10.1109/TC.1978.1675013

    Article  MATH  Google Scholar 

  11. Coke, J., Baliga, H., Cooray, N., Gamsaragan, E., Smith, P., Yoon, K., Abel, J., Valles., A.: Improvements in the Intel’s Core2 penryn processor family architecture and microarchitecture (2008)

  12. Debray, S., Patel, J.: Reverse engineering self-modifying code: unpacker extraction. In: Working Conference on Reverse Engineering, pp. 131–140 (2010)

  13. Dehnert, J., Grant, B., Banning, J., Johnson, R., Kistler, T., Klaiber, A., Mattson, J.: The transmeta code morphing trade; software: using speculation, recovery, and adaptive retranslation to address real-life challenges. In: International Symposium on Code Generation and Optimization: Feedback-Directed and Runtime Optimization, pp. 15–24. IEEE Computer Society (2003)

  14. Dinaburg, A., Royal, P., Sharif, M., Lee, W.: Ether: malware analysis via hardware virtualization extensions. In: ACM Conference on Computer and Communications Security, pp. 51–62 (2008)

  15. Gebai, M., Dagenais, M.R.: Survey and analysis of kernel and userspace tracers on linux: design, implementation, and overhead. ACM Comput. Surv. 51(2), 26:1–26:33 (2018). https://doi.org/10.1145/3158644

    Article  Google Scholar 

  16. Gruss, D., Maurice, C., Wagner, K., Mangard, S.: Flush+flush: a fast and stealthy cache attack. In: Interenational Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pp. 279–299 (2016)

  17. Gutierrez, A., Pusdesris, J., Dreslinski, R.G., Mudge, T.: Lazy cache invalidation for self-modifying codes. In: International Conference on Compilers, Architectures and Synthesis for Embedded Systems, pp. 151–160 (2012)

  18. Intel: Intel 64 and IA-32 Architectures Software Developer’s Manual. Intel (2013). https://www.intel.com.br/content/www/br/pt/architecture-and-technology/64-ia-32-architectures-software-developersystem-programming-manual-325384.html

  19. Inci, M.S., Eisenbarth, T., Sunar, B.: Wait a Minute! A Fast, Cross-VM Attack on AES, pp. 299–319. Springer, Springer (2014)

    Google Scholar 

  20. Korczynski, D.: Repeconstruct: reconstructing binaries with self-modifying code and import address table destruction. In: International Conference on Malicious and Unwanted Software, pp. 1–8 (2016)

  21. Liu, A., Wang, W.: Ascms: an accurate self-modifying code cache management strategy in binary translation. In: International Conference on Information Science and Cloud Computing Companion, pp. 405–410 (2013)

  22. Luk, C.K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI’05, pp. 190–200. ACM, New York, NY, USA (2005). https://doi.org/10.1145/1065010.1065034

  23. Maebe, J., De Bosschere, K.: Self-modifying Code (2003). arXiv:cs/0309029

  24. Martignoni, L., Christodorescu, M., Jha, S.: Omniunpack: fast, generic, and safe unpacking of malware. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 431–441 (2007). https://doi.org/10.1109/ACSAC.2007.15

  25. Microsoft: x64 intrinsics. https://msdn.microsoft.com/en-us/library/hh977022.aspx

  26. Mody, R.P.: Functional programming is not self-modifying code. SIGPLAN Not. 27(11), 13–14 (1992)

    Article  Google Scholar 

  27. Moret, P., Binder, W., Tanter, E.: Polymorphic bytecode instrumentation. In: Proceedings of the Tenth International Conference on Aspect-oriented Software Development, AOSD’11, pp. 129–140. ACM, New York, NY, USA (2011). https://doi.org/10.1145/1960275.1960292

  28. Neiger, G.,, Santoni, A., Leung, F., Rodgers, D., Uhlig, R.: Intel virtualization technology: hardware support for efficient processor virtualization (2006). https://www.ece.cmu.edu/~ece845/docs/vt-overview-itj06.pdf. Accessed Jan 2019

  29. Ray, K., Kramer, M., England, P., Field, S.: On-access scan of memory for malware, US Patent 7,836,504 (2010)

  30. Roemer, R., Buchanan, E., Shacham, H., Savage, S.: Return-oriented programming: systems, languages, and applications. ACM Trans. Inf. Syst. Secur. 15(1), 1–34 (2012)

    Article  Google Scholar 

  31. Shar, L.E., Lawton, K.P.: Trace cache for efficient self-modifying code processing, US Patent 7,606,975 (2009)

  32. Uluski, D., Moffie, M., Kaeli, D.: Characterizing antivirus workload execution. SIGARCH Comput. Archit. News 33(1), 90–98 (2005). https://doi.org/10.1145/1055626.1055639

    Article  Google Scholar 

  33. Willems, C., Hund, R., Fobian, A., Felsch, D., Holz, T., Vasudevan, A.: Down to the bare metal: using processor features for binary analysis. In: ACSAC, pp. 189–198 (2012)

  34. Wu, M., Zhang, Y., Mi, X.: Binary protection using dynamic fine-grained code hiding and obfuscation. In: International Conference on Information and Network Security, pp. 1–8 (2016)

  35. Xianya, M., Yi, Z., Baosheng, W., Yong, T.: A survey of software protection methods based on self-modifying code. In: International Conference on Computational Intelligence and Communication Networks, pp. 589–593 (2015)

  36. Yarom, Y., Falkner, K.: Flush+reload: a high resolution, low noise, l3 cache side-channel attack. In: USENIX Security, pp. 719–732 (2014)

  37. Zaidi, N.: System and method for tracking in-flight instructions in a pipeline, US Patent 6,237,088 (2001)

Download references

Acknowledgements

This project was partially financed by the Serrapilheira Institute (Grant Number Serra-1709-16621), the Brazilian National Counsel of Technological and Scientific Development (CNPq, Ph.D. Scholarship, process 164745/2017-3) and the Coordination for the Improvement of Higher Education Personnel (CAPES, Project FORTE, Forensics Sciences Program 24/2014, process 23038.007604/2014-69).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Marcus Botacin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Botacin, M., Zanata, M. & Grégio, A. The self modifying code (SMC)-aware processor (SAP): a security look on architectural impact and support. J Comput Virol Hack Tech 16, 185–196 (2020). https://doi.org/10.1007/s11416-020-00348-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-020-00348-w

Navigation