Skip to main content
SpringerLink
Log in

Misleading Malware Similarities Analysis by Automatic Data Structure Obfuscation

  • Conference paper
Information Security (ISC 2010)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 6531))

Included in the following conference series:

Abstract

Program obfuscation techniques have been widely used by malware to dodge the scanning from anti-virus detectors. However, signature based on the data structures appearing in the runtime memory makes traditional code obfuscation useless. Laika [2] implements this signature using Bayesian unsupervised learning, which clusters similar vectors of bytes in memory into the same class. We present a novel malware obfuscation technique that automatically obfuscate the data structure layout so that memory similarities between malware programs are blurred and hardly recognized. We design and implement the automatic data structure obfuscation technique as a GNU GCC compiler extension that can automatically distinguish the obfuscability of the data structures and convert part of the unobfuscable data structures into obfuscable. After evaluated by fourteen real-world malware programs, we present that our tool maintains a high proportion of obfuscated data structures as 60.19% for type and 60.49% for variable.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Lin, Z., Zhang, X., Xu, D.: Automatic Reverse Engineering of Data Structures from Binary Execution. In: Proceedings of the 17th Annual Network and Distributed System Security Symposium (2010)

    Google Scholar 

  2. Cozzie, A., Stratton, F., Xue, H., King, S.T.: Digging for Data Structures. In: The 8th USENIX Symposium on Operating Systems Design and Implementation (2008)

    Google Scholar 

  3. Anubis: Analyzing Unknown Binaries (2009), http://anubis.seclab.tuwien.ac.at

  4. CWSandbox (2009), http://www.cwsandbox.org/

  5. Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM Conference on Computer and Communications Security (2003)

    Google Scholar 

  6. Christodorescu, M., Jha, S., Seshia, S.A., Songand, D., Bryant, R.E.: Semantics-Aware Malware Detection. In: Proceedings of the 2005 IEEE Symposium on Security and Privacy (2005)

    Google Scholar 

  7. Popov, I.V., Debray, S.K., Andrews, G.R.: Binary obfuscation using signals. In: Proceedings of the 16th USENIX Security Symposium (2007)

    Google Scholar 

  8. Moser, A., Kruegel, C., Kirda, E.: Limits of Static Analysis for Malware Detection. In: 23rd Annual Computer Security Applications Conference (2007)

    Google Scholar 

  9. Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D.X.: Automatically Identifying Trigger-based Behavior in Malware. In: Lee, W., et al. (eds.) Book chapter in Botnet Analysis and Defense (2007)

    Google Scholar 

  10. Moser, A., Kruegel, C., Kirda, E.: Exploring Multiple Execution Paths for Malware Analysis. In: Proceedings of the 28th IEEE Symposium on Security and Privacy (2007)

    Google Scholar 

  11. Coogan, K., Debray, S.K., Kaochar, T., Townsend, G.M.: Automatic Static Unpacking of Malware Binaries. In: The 16th Working Conference on Reverse Engineering (2009)

    Google Scholar 

  12. Balakrishnan, G., Reps, T.: Analyzing Memory Accesses in x86 Executables. In: Duesterwald, E. (ed.) CC 2004. LNCS, vol. 2985, pp. 5–23. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  13. Balakrishnan, G., Reps, T.W.: DIVINE: Discovering Variables IN Executables. In: Proceeding of Verification Model Checking and Abstract Interpretation (2007)

    Google Scholar 

  14. Szor, P.: The Art of Computer Virus Research and Defense. Addison Wesley, Reading (2005)

    Google Scholar 

  15. Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: Proceedings of the 16th USENIX Security Symposium (2003)

    Google Scholar 

  16. Sharif, M.I., Lanzi, A., Giffin, J.T., Lee, W.: Impeding Malware Analysis Using Conditional Code Obfuscation. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (2008)

    Google Scholar 

  17. Pearce, S.: Viral polymorphism. VX Heavens (2003)

    Google Scholar 

  18. The Mental Drille Metamorphism in practice or How I made MetaPHOR and what I’ve learnt. VX Heavens (February 2002)

    Google Scholar 

  19. Kirda, E., Kruegel, C., Banks, G., Vigna, G., Kemmerer, R.A.: Behavior-based spyware detection. In: Proceedings of the 15th Conference on USENIX Security Symposium (2006)

    Google Scholar 

  20. Stallman, R.: Using GCC: the GNU compiler collection reference manual. GNU Press (2009)

    Google Scholar 

  21. TESO. Burneye ELF encryption program (January 2004), http://teso.scene.at

  22. Detristan, T., Ulenspiegel, T., Malcom, Y., von Underduk, M.S.: Polymorphic Shellcode Engine Using Spectrum Analysis. Phrack 61 (2003)

    Google Scholar 

  23. Julus, L.: Metamorphism. VX heaven (March 2000), http://vx.netlux.org/lib/vlj00.html

  24. Kruegel, C., Robertson, W., Vigna, G.: Detecting Kernel-Level Rootkits Through Binary Analysis. In: Proceedings of the 20th Annual Computer Security Applications Conference (2004)

    Google Scholar 

  25. Lin, Z., Riley, R.D., Xu, D.: Polymorphing Software by Randomizing Data Structure Layout. In: Proceedings of the 6th SIDAR Conference on Detection of Intrusions and Malware and Vulnerability Assessment (2009)

    Google Scholar 

  26. Balakrishnan, A., Schulze, C.: Code Obfuscation Literature Survey (2005), http://pages.cs.wisc.edu/~arinib/projects.htm

  27. Colberg, Thomborson: Watermarking, Tamper-Proofing, and Obfuscation–Tools for Software Protection. IEEE Transactions on Software Engineering 28(8) (2002)

    Google Scholar 

  28. Bhatkar, S., Sekar, R., DuVarney, D.C.: Efficient techniques for comprehensive protection from memory error exploits. In: Proceedings of the 14th Conference on USENIX Security Symposium (2005)

    Google Scholar 

  29. Bhatkar, S., DuVarney, D.C., Sekar, R.: Address obfuscation: an efficient approach to combat a board range of memory error exploits. In: Proceedings of the 12th Conference on USENIX Security Symposium (2003)

    Google Scholar 

  30. Cifuentes, C., Gough, K.J.: Decompilation of Binary Programs. Software Practice & Experience (July 1995)

    Google Scholar 

  31. Ramalingam, G., Field, J., Tip, F.: Aggregate structure identification and its application to program analysis. In: Proceedings of the 26th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (1999)

    Google Scholar 

  32. Status of C99 features in GCC, GNU (1999), http://gcc.gnu.org/c99status.html

  33. Richard Stevens, W.: Advanced Programming in the UNIX Environment. Addison-Wesley, Reading (1992)

    MATH  Google Scholar 

  34. Shapiro, M., Horwitz, S.: The Effects of the Precision of Pointer Analysis. Lecture Notes in Computer Science (1997)

    Google Scholar 

  35. Collberg, C., Thomborson, C., Low, D.: A Taxonomy of Obfuscating Transformations. Technical Report 148, University of Auckland (1997)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Department of Computer Science and Technology, Nanjing University, Nanjing, 210093, China

    Zhi Xin, Huiyu Chen, Hao Han, Bing Mao & Li Xie

Authors
  1. Zhi Xin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huiyu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bing Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Li Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Center for Security and Assurance in IT (C-SAIT), Department of Computer Science, Florida State University, 32306-4530, Tallahassee, FL, USA

    Mike Burmester

  2. Department of Computer Science, University of California, Irvine, 92697-3425, CA, USA

    Gene Tsudik

  3. Center for Cryptology and Information Security (CCIS), Department of Mathematical Sciences, Florida Atlantic University, 33431-0991, Boca Raton, FL, USA

    Spyros Magliveras

  4. Mathematical Sciences Department, Florida Atlantic University, 33431-0991, Boca Raton, FL, USA

    Ivana Ilić

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Xin, Z., Chen, H., Han, H., Mao, B., Xie, L. (2011). Misleading Malware Similarities Analysis by Automatic Data Structure Obfuscation. In: Burmester, M., Tsudik, G., Magliveras, S., Ilić, I. (eds) Information Security. ISC 2010. Lecture Notes in Computer Science, vol 6531. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-18178-8_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-18178-8_16

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-18177-1

  • Online ISBN: 978-3-642-18178-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation