Skip to main content

Polymorphing Software by Randomizing Data Structure Layout

  • Conference paper
Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA 2009)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 5587))

Abstract

This paper introduces a new software polymorphism technique that randomizes program data structure layout. This technique will generate different data structure layouts for a program and thus diversify the binary code compiled from the same program source code. This technique can mitigate attacks (e.g., kernel rootkit attacks) that require knowledge about data structure definitions. It is also able to disrupt the generation of data structure-based program signatures. We have implemented our data structure layout randomization technique in the open source compiler collection gcc-4.2.4 and applied it to a number of programs. Our evaluation results show that our technique is able to achieve software binary diversity. We also apply the technique to one operating system data structure in order to foil a number of kernel rootkit attacks. Meanwhile, programs produced by the technique were analyzed by a state-of-the-art data structure inference system and it was demonstrated that reliance on data structure signatures alone may lead to false negatives in malware detection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Gnu compiler collection (gcc) internals, http://gcc.gnu.org/onlinedocs/gccint/

  2. Offensive computing, http://www.offensivecomputing.net/

  3. Using the gnu compiler collection (gcc), http://gcc.gnu.org/onlinedocs/gcc-4.2.4/gcc/

  4. Vx heavens, http://vx.netlux.org/

  5. Aho, A.V., Lam, M.S., Sethi, R., Ullman, J.D.: Compilers: Principles, Techniques and Tools, 2nd edn. Addison-Wesley, Reading (2006)

    MATH  Google Scholar 

  6. Barrantes, E.G., Ackley, D.H., Palmer, T.S., Stefanovic, D., Zovi, D.D.: Randomized instruction set emulation to disrupt binary code injection attacks. In: Proceedings of the 10th ACM conference on Computer and communications security (CCS 2003), pp. 281–289. ACM, New York (2003)

    Chapter  Google Scholar 

  7. Berger, E.D., Zorn, B.G.: Diehard: probabilistic memory safety for unsafe languages. In: Proceedings of the 2006 ACM SIGPLAN conference on Programming language design and implementation (PLDI 2006), pp. 158–168. ACM, New York (2006)

    Chapter  Google Scholar 

  8. Bhatkar, E., Duvarney, D.C., Sekar, R.: Address obfuscation: an efficient approach to combat a broad range of memory error exploits. In: Proceedings of the 12th USENIX Security Symposium, pp. 105–120 (2003)

    Google Scholar 

  9. Bhatkar, S., Sekar, R.: Data space randomization. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 1–22. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  10. Bhatkar, S., Sekar, R., DuVarney, D.C.: Efficient techniques for comprehensive protection from memory error exploits. In: Proceedings of the 14th conference on USENIX Security Symposium, Berkeley, CA, USA (2005), USENIX Association

    Google Scholar 

  11. Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: Automatic extraction of protocol format using dynamic binary analysis. In: Proceedings of the 14th ACM Conference on Computer and and Communications Security (CCS 2007) (2007)

    Google Scholar 

  12. Cadar, C., Akritidis, P., Costa, M., Martin, J.-P., Castro, M.: Data randomization. Technical Report MSR-TR-2008-120, Microsoft Research (2008)

    Google Scholar 

  13. Chew, M., Song, D.: Mitigating buffer overflows by operating system randomization. Technical Report CMU-CS-02-197, Carnegie Mellon University (2002)

    Google Scholar 

  14. Cho, S., Chang, H., Cho, Y.: Implementation of an obfuscation tool for c/c++ source code protection on the xscale architecture. In: Brinkschulte, U., Givargis, T., Russo, S. (eds.) SEUS 2008. LNCS, vol. 5287, pp. 406–416. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  15. Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscating transformations (1997)

    Google Scholar 

  16. Comparetti, P.M., Wondracek, G., Kruegel, C., Kirda, E.: Prospex: Protocol specification extraction. In: Proceedings of 2009 IEEE Symposium on Security and Privacy, Oakland, CA (May 2009)

    Google Scholar 

  17. Cowan, C., Beattie, S., Johansen, J., Wagle, P.: Pointguard: protecting pointers from buffer overflow vulnerabilities. In: Proceedings of the 12th conference on USENIX Security Symposium, Berkeley, CA, USA (2003), USENIX Association

    Google Scholar 

  18. Cox, B., Evans, D., Filipi, A., Rowanhill, J., Hu, W., Davidson, J., Knight, J., Nguyen-Tuong, A., Hiser, J.: N-variant systems: a secretless framework for security through diversity. In: Proceedings of the 15th conference on USENIX Security Symposium, Berkeley, CA, USA (2006), USENIX Association

    Google Scholar 

  19. Cozzie, A., Stratton, F., Xue, H., King, S.T.: Digging for data structures. In: Proceeding of 8th Symposium on Operating System Design and Implementation (OSDI 2008) (December 2008)

    Google Scholar 

  20. Cui, W., Peinado, M., Chen, K., Wang, H.J., Irun-Briz, L.: Tupni: Automatic reverse engineering of input formats. In: Proceedings of the 15th ACM Conference on Computer and Communications Security (CCS 2008), Alexandria, VA (October 2008)

    Google Scholar 

  21. Cui, W., Peinado, M., Wang, H.J., Locasto, M.: Shieldgen: Automatic data patch generation for unknown vulnerabilities with informed probing. In: Proceedings of 2007 IEEE Symposium on Security and Privacy, Oakland, CA (May 2007)

    Google Scholar 

  22. Etoh, H.: GCC extension for protecting applications from stack-smashing attacks, ProPolice (2003), http://www.trl.ibm.com/projects/security/ssp/

  23. Forrest, S., Somayaji, A., Ackley, D.: Building diverse computer systems. In: Proceedings of the 6th Workshop on Hot Topics in Operating Systems (HotOS-VI), Washington, DC, USA, p. 67. IEEE Computer Society, Los Alamitos (1997)

    Google Scholar 

  24. Garfinkel, T., Rosenblum, M.: A Virtual Machine Introspection Based Architecture for Intrusion Detection. In: Proc. Network and Distributed Systems Security Symposium (NDSS 2003) (February 2003)

    Google Scholar 

  25. Golovanevsky, O., Zaks, A.: Struct-reorg: current status and future perspectives. In: Proceedings of the GCC Developers’ Summit (2007)

    Google Scholar 

  26. Hagog, M., Tice, C.: Cache aware data layout reorganization optimization in gcc. In: Proceedings of the GCC Developers’ Summit (2005)

    Google Scholar 

  27. Jiang, X., Wang, H.J., Xu, D., Wang, Y.-M.: Randsys: Thwarting code injection attacks with system service interface randomization. In: Proceedings of the 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007), Washington, DC, USA, pp. 209–218. IEEE Computer Society, Los Alamitos (2007)

    Google Scholar 

  28. Kc, G.S., Keromytis, A.D., Prevelakis, V.: Countering code-injection attacks with instruction-set randomization. In: Proceedings of the 10th ACM conference on Computer and communications security (CCS 2003), Washington D.C., USA, pp. 272–280. ACM, New York (2003)

    Chapter  Google Scholar 

  29. Lin, Z., Jiang, X., Xu, D., Zhang, X.: Automatic protocol format reverse engineering through context-aware monitored execution. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS 2008), San Diego, CA (February 2008)

    Google Scholar 

  30. Lin, Z., Zhang, X.: Deriving input syntactic structure from execution. In: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2008), Atlanta, GA, USA (November 2008)

    Google Scholar 

  31. Low, D.: Protecting java code via code obfuscation. Crossroads 4(3), 21–23 (1998)

    Article  Google Scholar 

  32. Novark, G., Berger, E.D., Zorn, B.G.: Exterminator: automatically correcting memory errors with high probability. In: Proceedings of ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 007), San Diego, California, USA. ACM Press, New York (2007)

    Google Scholar 

  33. Percival, C.: Naive differences of executable code (2003), http://www.daemonology.net/bsdiff/

  34. Raman, E., Hundt, R., Mannarswamy, S.: Structure layout optimization for multithreaded programs. In: Proceedings of the International Symposium on Code Generation and Optimization (CGO 2007), Washington, DC, USA, pp. 271–282. IEEE Computer Society Press, Los Alamitos (2007)

    Chapter  Google Scholar 

  35. Riley, R., Jiang, X., Xu, D.: Guest-Transparent Prevention of Kernel Rootkits with VMM-Based Memory Shadowing. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 1–20. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  36. Salamat, B., Gal, A., Yermolovich, A., Manivannan, K., Franz, M.: Reverse stack execution. Technical Report No. 07-07, University of California, Irvine (2007)

    Google Scholar 

  37. Seshadri, A., Luk, M., Qu, N., Perrig, A.: SecVisor: A Tiny Hypervisor to Guarantee Lifetime Kernel Code Integrity for Commodity OSes. In: Proceedings of the ACM Symposium on Operating Systems Principles (SOSP 2007) (October 2007)

    Google Scholar 

  38. PaX Team. Pax address space layout randomization (aslr), http://pax.grsecurity.net/docs/aslr.txt

  39. Wang, X., Li, Z., Xu, J., Reiter, M.K., Kil, C., Choi, J.Y.: Packet vaccine: Black-box exploit detection and signature generation. In: Proceedings of the 13th ACM Conference on Computer and Communication Security (CCS 2006), pp. 37–46. ACM Press, New York (2006)

    Google Scholar 

  40. Wondracek, G., Milani, P., Kruegel, C., Kirda, E.: Automatic network protocol analysis. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS 2008), San Diego, CA (February 2008)

    Google Scholar 

  41. Xu, J., Kalbarczyk, Z., Iyer, R.K.: Transparent runtime randomization for security. In: Proceedings of the 22nd International Symposium on Reliable Distributed Systems (SRDS 2003), pp. 260–269. IEEE Computer Society, Los Alamitos (2003)

    Google Scholar 

  42. Zhong, Y., Orlovich, M., Shen, X., Ding, C.: Array regrouping and structure splitting using whole-program reference affinity. In: Proceedings of the ACM SIGPLAN 2004 conference on Programming language design and implementation, PLDI 2004 (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lin, Z., Riley, R.D., Xu, D. (2009). Polymorphing Software by Randomizing Data Structure Layout. In: Flegel, U., Bruschi, D. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2009. Lecture Notes in Computer Science, vol 5587. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02918-9_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-02918-9_7

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-02917-2

  • Online ISBN: 978-3-642-02918-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics