Abstract
Malicious software, better known as malware, is a major threat to society. Malware today typically employ a technique called obfuscation. Obfuscation detection in malware is a well-documented problem and has been analyzed using dynamic analysis. However, many tools that detect obfuscation in malware make no attempts to use the presence of obfuscation as a method of detecting malware because their schemes would also detect benign applications. We present three main contributions. First, we conduct a unique study into the prevalence of obfuscation in benign applications. Second, we create discriminating features that can distinguish obfuscation in benign applications versus malware. Third, we prove that using the presence of obfuscation can detect previously hard-to-detect malware. Our results show that for our set of programs, we are able to reduce the number of malware missed by five market-leading AV tools by 25% while only falsely detecting 2.45% of tested benign applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A CTI is called direct where the CTI’s target is a constant; otherwise it is called indirect.
References
Anckaert, B., Madou, M., Bosschere, K.: A model for self-modifying code. In: Camenisch, J.L., Collberg, C.S., Johnson, N.F., Sallee, P. (eds.) IH 2006. LNCS, vol. 4437, pp. 232–248. Springer, Heidelberg (2007). doi:10.1007/978-3-540-74124-4_16
Bania, P.: Generic unpacking of self-modifying, aggressive, packed binary programs. arXiv preprint (2009). arXiv:0905.4581
Bayer, U., Moser, A., Kruegel, C., Kirda, E.: Dynamic analysis of malicious code. J. Comput. Virol. 2(1), 67–77 (2006)
Blackburn, S.M., Garner, R., Hoffmann, C., Khang, A.M., McKinley, K.S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S.Z., et al.: The dacapo benchmarks: java benchmarking development and analysis. In: ACM Sigplan Notices, vol. 41, pp. 169–190. ACM (2006)
Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscating transformations. Department of Computer Science, The University of Auckland, New Zealand, Technical report (1997)
Microsoft Corporation: What is data execution prevention? May 2016
Microsoft Corporation: Internet security threat report 21, 5, April 2016
Debray, S., Patel, J.: Reverse engineering self-modifying code: unpacker extraction. In: 2010 17th Working Conference on Reverse Engineering (WCRE), pp. 131–140. IEEE (2010)
Dinaburg, A., Royal, P., Sharif, M., Lee, W.: Ether: malware analysis via hardware virtualization extensions. In: Proceedings of the 15th ACM Conference on Computer and Communications Security, pp. 51–62. ACM (2008)
Ghosh, S., Hiser, J., Davidson, J.W.: Software protection for dynamically-generated code. In: Proceedings of the 2nd ACM SIGPLAN Program Protection and Reverse Engineering Workshop, p. 1. ACM (2013)
Guarnieri, C., Tanasi, A., Bremer, J., Schloesser, M.: The cuckoo sandbox (2012)
Hungenberg, T., Eckert, M.: Inetsim: internet services simulation suite (2013)
Kang, M.G., Poosankam, P., Yin, H.: Renovo: a hidden code extractor for packed executables. In: Proceedings of the 2007 ACM Workshop on Recurring Malcode, pp. 46–53. ACM (2007)
Kapoor, A.: An approach towards disassembly of malicious binary executables. Ph.D. thesis, University of Louisiana at Lafayette (2004)
Kivity, A., Kamay, Y., Laor, D., Lublin, U., Liguori, A.: KVM: the Linux virtual machine monitor. In: Proceedings of the Linux Symposium, vol. 1, pp. 225–230 (2007)
Kwon, B.J., Mondal, J., Jang, J., Bilge, L., Dumitras, T.: The dropper effect: insights into malware distribution with downloader graph analytics. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1118–1129. ACM (2015)
Luk, C.K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V.J., Hazelwood, K.: Pin: building customized program analysis tools with dynamic instrumentation. In: ACM Sigplan Notices, vol. 40, pp. 190–200. ACM (2005)
Madou, M., Van Put, L., De Bosschere, K.: Understanding obfuscated code. In: 14th IEEE International Conference on Program Comprehension (ICPC 2006), pp. 268–274. IEEE (2006)
Maebe, J., De Bosschere, K.: Instrumenting self-modifying code. arXiv preprint cs/0309029 (2003)
Martignoni, L., Christodorescu, M., Jha, S.: OmniUnpack: fast, generic, and safe unpacking of malware. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 431–441. IEEE (2007)
Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: Twenty-third Annual Computer Security Applications Conference (ACSAC 2007), pp. 421–430. IEEE (2007)
O’Kane, P., Sezer, S., McLaughlin, K.: Obfuscation: the hidden malware. IEEE Secur. Priv. 9(5), 41–47 (2011)
Pike, R., Locanthi, B., Reiser, J.: Hardware/software trade-offs for bitmap graphics on the blit. Softw. Pract. Exper. 15(2), 131–151 (1985)
Popov, I.V., Debray, S.K., Andrews, G.R.: Binary obfuscation using signals. In: Usenix Security (2007)
Prakash, C.: Design of x86 emulator for generic unpacking. In: Assocation of Anti-Virus Asia Researchers International Conference, Seoul, South Korea (2007)
Quist, D.A., Liebrock, L.M.: Visualizing compiled executables for malware analysis. In: 6th International Workshop on Visualization for Cyber Security (VizSec 2009), pp. 27–32. IEEE (2009)
Quist, D., Smith, V.: Covert debugging circumventing software armoring techniques. Black hat briefings USA (2007)
Rabek, J.C., Khazan, R.I., Lewandowski, S.M., Cunningham, R.K.: Detection of injected, dynamically generated, and obfuscated malicious code. In: Proceedings of the 2003 ACM Workshop on Rapid Malcode, pp. 76–82. ACM (2003)
Rodrigues, T.: Extracting known bad hash set from nsrl. https://digital-forensics.sans.org/blog/2010/02/22/extracting-known-bad-hashset-from-nsrl/
Roundy, K.A., Miller, B.P.: Binary-code obfuscations in prevalent packer tools. ACM J. Name 1, 21 (2012)
Royal, P., Halpin, M., Dagon, D., Edmonds, R., Lee, W.: PolyUnpack: automating the hidden-code extraction of unpack-executing malware. In: Null, pp. 289–300. IEEE (2006)
Santos, I., Devesa, J., Brezo, F., Nieves, J., Bringas, P.G.: Opem: a static-dynamic approach for machine-learning-based malware detection. In: Herrero, A., et al. (eds.) International Joint Conference CISIS ’12-ICEUTE ’12-SOCO ’12 Special Sessions. AISC, vol. 189, pp. 271–280. Springer, Heidelberg (2013). doi:10.1007/978-3-642-33018-6
Sharif, M., Lanzi, A., Giffin, J., Lee, W.: Automatic reverse engineering of malware emulators. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 94–109. IEEE (2009)
Song, D., et al.: BitBlaze: a new approach to computer security via binary analysis. In: Sekar, R., Pujari, A.K. (eds.) ICISS 2008. LNCS, vol. 5352, pp. 1–25. Springer, Heidelberg (2008). doi:10.1007/978-3-540-89862-7_1
National Institute of Standards and Technology: National software reference library. Online
Stewart, J.: Ollybone: semi-automatic unpacking on IA-32. In: Proceedings of the 14th DEF CON Hacking Conference (2006)
Wang, X., Jhi, Y.C., Zhu, S., Liu, P.: Still: exploit code detection via static taint and initialization analyses. In: Annual Computer Security Applications Conference (ACSAC 2008), pp. 289–298. IEEE (2008)
Wang, Y.M., Vo, B., Roussev, R., Verbowski, C., Johnson, A.: Strider ghostbuster: why it’s a bad idea for stealth software to hide files. Technical report, Technical Report MSR-TR-2004-71, Microsoft Research (2004)
Willems, C., Holz, T., Freiling, F.: Toward automated dynamic malware analysis using CWSandbox. IEEE Secur. Priv. 2, 32–39 (2007)
Ying, L., Su, P., Feng, D., Wang, X., Yang, Y., Liu, Y.: Reconbin: reconstructing binary file from execution for software analysis. In: Third IEEE International Conference on Secure Software Integration and Reliability Improvement (SSIRI 2009), pp. 222–229. IEEE (2009)
Yuschuk, O.: Ollydbg (2007)
Acknowledgments
The research was sponsored in part by contracts H9823013D00560016 and H9823013D00560037 from the US Department of Defense via the Maryland Procurement Office. The ARCS Foundation has also played a role in funding the research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Kim, D. et al. (2017). DynODet: Detecting Dynamic Obfuscation in Malware. In: Polychronakis, M., Meier, M. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2017. Lecture Notes in Computer Science(), vol 10327. Springer, Cham. https://doi.org/10.1007/978-3-319-60876-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-60876-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-60875-4
Online ISBN: 978-3-319-60876-1
eBook Packages: Computer ScienceComputer Science (R0)