Skip to main content
SpringerLink
Log in

Network-level polymorphic shellcode detection using emulation

  • Original Paper
  • Published:
Journal in Computer Virology Aims and scope Submit manuscript

Abstract

Significant progress has been made in recent years towards preventing code injection attacks at the network level. However, as state-of-the-art attack detection technology becomes more prevalent, attackers are likely to evolve, employing techniques such as polymorphism and metamorphism to defeat these defenses. A major outstanding question in security research and engineering is thus whether we can proactively develop the tools needed to contain advanced polymorphic and metamorphic attacks. While recent results have been promising, most of the existing proposals can be defeated using only minor enhancements to the attack vector. In fact, some publicly-available polymorphic shellcode engines are currently one step ahead of the most advanced publicly-documented network-level detectors. In this paper, we present a heuristic detection method that scans network traffic streams for the presence of previously unknown polymorphic shellcode. In contrast to previous work, our approach relies on a NIDS- embedded CPU emulator that executes every potential instruction sequence in the inspected traffic, aiming to identify the execution behavior of polymorphic shellcode. Our analysis demonstrates that the proposed approach is more robust to obfuscation techniques like self-modifications compared to previous proposals, but also highlights advanced evasion techniques that need to be more closely examined towards a satisfactory solution to the polymorphic shellcode detection problem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. sk, History and advances in windows shellcode. Phrack 11(62), (2004)

  2. Kim, H.-A., Karp, B.: Autograph: toward automated, distributed worm signature detection. In: Proceedings of the 13th USENIX Security Symposium, pp. 271–286, (2004)

  3. Singh, S., Estan, C., Varghese, G., Savage, S.: Automated worm fingerprinting. In: Proceedings of the 6th Symposium on Operating Systems Design & Implementation (OSDI), (2004)

  4. Newsome, J., Karp, B., Song, D.: Polygraph: automatically Generating signatures for polymorphic worms. In: Proceedings of the IEEE Security & Privacy Symposium, pp. 226–241, (2005)

  5. Tang, Y., Chen, S.: Defending against internet worms: a signature-based approach. In: Proceedings of the 24th Annual Joint Conference of IEEE Computer and Communication societies (INFOCOM), (2005)

  6. Wang, K., Stolfo, S.J.: Anomalous payload-based network intrusion detection. In: Proceedings of the 7th International Symposium on Recent Advanced in Intrusion Detection (RAID), pp. 201–222, (2004)

  7. Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Polymorphic worm detection using structural information of executables. In: Proceedings of the International Symposium on Recent Advances in Intrusion Detection (RAID), (2005)

  8. Chinchani, R., Berg, E.V.D.: A fast static analysis approach to detect exploit code inside network flows. In: Proceedings of the International Symposium on Recent Advances in Intrusion Detection (RAID), (2005)

  9. Wang, X., Pan, C.-C., Liu, P., Zhu, S.: Sigfree: a signature-free buffer overflow attack blocker. In: Proceedings of the USENIX Security Symposium (2006)

  10. Li, Z., Sanghi, M., Chen, Y., Kao, M.-Y., Chavez, B.: Hamsa: fast signature generation for zero-day polymorphic worms with provable attack resilience. In: Proceedings of the 2006 IEEE Symposium on Security and Privacy, pp. 32–47, 2006

  11. Ször, P.: The art of computer virus research and defense. Addison-Wesley Professional, (2005)

  12. Ször, P., Ferrie, P.: Hunting for metamorphic. In: Proceedings of the virus bulletin conference. pp. 123–144, (2001)

  13. Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th USENIX Security Symposium (Security’03), (2003)

  14. Roesch, M.: Snort: lightweight intrusion detection for networks. In: Proceedings of USENIX LISA ’99, November 1999, (software available from http://www.snort.org/)

  15. Paxson, V.: Bro: a system for detecting network intruders in real-time. In: Proceedings of the 7th USENIX Security Symposium, (1998)

  16. Jordan C. (2005). Writing detection signatures. USENIX login 30(6): 55–61

    Google Scholar 

  17. K2, ADMmutate, http://www.ktwo.ca/ADMmutate-0.8.4.tar. gz, (2001)

  18. Detristan, T., Ulenspiegel, T., Malcom, Y., Underduk, M.: Polymorphic shellcode engine using spectrum analysis. Phrack 11(61), (2003)

  19. Rix, Writing IA32 alphanumeric shellcodes. Phrack 11(57), (2001)

  20. Bania, P.: TAPiON, http://pb.specialised.info/all/tapion/, (2005)

  21. Toth, T., Kruegel, C.: Accurate buffer overflow detection via abstract payload execution. In: Proceedings of the 5th Symposium on Recent Advances in Intrusion Detection (RAID), (2002)

  22. Akritidis, P., Markatos, E.P., Polychronakis, M., K.: STRIDE: Polymorphic sled detection through instruction sequence analysis. In: Proceedings of the 20th IFIP International Information Security Conference (IFIP/SEC), (2005)

  23. Crandall, J.R., Wu, S.F., Chong, F.T.: Experiences using minos as a tool for capturing and analyzing novel worms for unknown vulnerabilities. In: Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA), (2005)

  24. Pasupulati, A., Coit, J., Levitt, K., Wu, S., Li, S., Kuo, J., Fan, K.: Buttercup: on network-based detection of polymorphic buffer overflow vulnerabilities. In: Proceedings of the Network Operations and Management Symposium (NOMS), pp. 235–248, (2004)

  25. Pincus J. and Baker B. (2004). Beyond stack smashing: recent advances in exploiting buffer overflows. IEEE Security Privacy 2(4): 20–27

    Article  Google Scholar 

  26. Kreibich, C., Crowcroft, J.: Honeycomb–creating intrusion detection signatures using honeypots. In: Proceedings of the Second Workshop on Hot Topics in Networks (HotNets-II), (2003)

  27. Kolesnikov, O., Dagon, D., Lee, W.: Advanced polymorphic worms: evading IDS by blending in with traffic. In: College of Computing, Georgia of Technology, Atlanta, GA 30332, http://www.cc.gatech.edu/~ok/w/ok_pw.pdf, (2004)

  28. Newsome, J., Karp, B., Song, D.: Paragraph: thwarting signature learning by training maliciously. In: Proceedings of the 9th International Symposium on Recent Advances in Intrusion Detection (RAID), (2006)

  29. Payer, U., Teufl, P., Lamberger, M.: Hybrid engine for polymorphic shellcode detection. In: Proceedings of the conference on detection of intrusions and malware and vulnerability assessment (DIMVA), pp. 19–31, (2005)

  30. Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM conference on Computer and communications security (CCS), pp. 290–299, (2003)

  31. Aycock, J., deGraaf, R., Jacobson, M.: Anti-disassembly using cryptographic hash functions. Department of Computer Science, University of Calgary, Technical Report, pp. 793–824, (2005)

  32. Venable, M., Chouchane, M.R., Karim, M.E., Lakhotia, A.: Analyzing memory accesses in obfuscated x86 executables. In: Proceedings of the conference on detection of intrusions and malware and vulnerability assessment (DIMVA), (2005)

  33. Collberg C.S. and Thomborson C. (2002). Watermarking, tamper-proffing and obfuscation: tools for software protection. IEEE Trans. Softw. Eng. 28(8): 735–746

    Article  Google Scholar 

  34. Wang, C., Hill, J., Knight, J., Davidson, J.: Software tamper resistance: Obstructing static analysis of programs. University of Virginia, Technical Report CS-2000–12, (2000)

  35. Madou, M., Anckaert, B., Moseley, P., Debray, S., Sutter, B.D., Bosschere, K.D.: Software protection through dynamic code mutation. In: Proceedings of the 6th International Workshop on Information Security Applications (WISA), pp. 194–206, (2005)

  36. Schwarz, B., Debray, S., Andrews, G.: Disassembly of executable code revisited. In: Proceedings of the ninth working conference on reverse engineering (WCRE), (2002)

  37. Prasad, M., cker Chiueh, T.: A binary rewriting defense against stack based overflow attacks. In: Proceedings of the USENIX annual technical conference, (2003)

  38. Kruegel, C., Robertson, W., Valeur, F., Vigna, G.: Static disassembly of obfuscated binaries. In: Proceedings of the USENIX security symposium, pp. 255–270, (2004)

  39. Cohen F.B. (1993). Operating system protection through program evolution. Comput. Sec. 12(6): 565–584

    Article  Google Scholar 

  40. Metasploit project, http://www.metasploit.com/, (2006)

  41. Cifuentes C. and Gough K.J. (1995). Decompilation of binary programs. Softw. Prac. Exp. 25(7): 811–829

    Article  Google Scholar 

  42. Balakrishnan, G., Reps, T.: Analyzing memory accesses in x86 executables. In: Proceedings of the International Conference on Compiler Construction (CC), (2004)

  43. Noir, GetPC code (was: Shellcode from ASCII), http://www. securityfocus.com/ archive/82/327100/2006-01-03/1, June 2003

  44. Ionescu, C.: GetPC code (was: Shellcode from ASCII), http:// www.securityfocus.com/archive/82/327348/2006-01-03/1, July 2003

  45. Wever, B.-J.: Alpha 2, (2004), http://www.edup.tudelft.nl/bjwever/src/alpha2.c

  46. Perriot, F., Ferrie, P., Ször, P.: Striking similarities. Virus Bull., pp. 4–6, (2002)

  47. Obscou, Building IA32 ‘unicode-proof’ shellcodes. Phrack 11(61), (2003)

  48. Tubella, J., González, A.: Control speculation in multithreaded processors through dynamic loop detection. In: Proceedings of the 4th International Symposium on High- Performance Computer Architecture (HPCA), (1998)

  49. McCanne, S., Leres, C., Jacobson, V.: Libpcap. http://www.tcpdump.org/, (2006)

  50. Wojtczuk, R.: Libnids. http://libnids.sourceforge.net/, (2006)

  51. jt, Libdasm. http://www.klake.org/~jt/misc/libdasm-1.4.tar. gz, (2006)

  52. Apache Chunked Encoding Overflow. http://www.osvdb.org/838, (2002)

  53. Microsoft Windows RPC DCOM Interface Overflow, http://www.osvdb.org/2100, (2003)

  54. Microsoft Windows LSASS Remote Overflow, http://www. osvdb.org/5248, (2004)

  55. Bell J.R. (1973). Threaded code. Comm. of the ACM. 16(6): 370–372

    Article  Google Scholar 

  56. Bellard, F.: QEMU, a fast and portable dynamic translator. In: Proceedings of the USENIX Annual Technical Conference, FREENIX Track, pp. 41–46, (2005)

  57. Bhatkar, S., DuVarney, D.C., Sekar, R.: Address obfuscation: an efficient approach to combat a broad range of memory error exploits. In: Proceedings of the 12th USENIX Security Symposium, (2003)

  58. Anagnostakis, K., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., Keromytis, A.D.: Detecting targeted attacks using shadow honeypots. In: Proceedings of the 14th USENIX Security Symposium, pp. 129–144, (2005)

  59. Hsu, F.-H., Chiueh, T.-C.: CTCP: a transparent centralized tcp/ip architecture for network security. In: Proceedings of the 20th Annual Computer Security Applications Conference (ACSAC), pp. 335–344, (2004)

  60. Liang, Z. Sekar, R.: Fast and automated generation of attack signatures: a basis for building self-protecting servers. In: Proceedings of the 12th ACM conference on Computer and communications security (CCS), pp. 213–222, (2005)

  61. Dreger, H., Kreibich, C., Paxson, V., Sommer, R.: Enhancing the accuracy of network-based intrusion detection with host-based context. In: Proceedings of the Conference on Detection of Intrusions and Malware and Vulnerability Assessment (DIMVA), (2005)

Download references

Author information

Authors and Affiliations

  1. Institute of Computer Science, Foundation for Research & Technology – Hellas, Heraklion, Crete, Greece

    Michalis Polychronakis & Evangelos P. Markatos

  2. Institute for Infocomm Research, Singapore, Singapore

    Kostas G. Anagnostakis

Authors
  1. Michalis Polychronakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kostas G. Anagnostakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Evangelos P. Markatos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michalis Polychronakis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Polychronakis, M., Anagnostakis, K.G. & Markatos, E.P. Network-level polymorphic shellcode detection using emulation. J Comput Virol 2, 257–274 (2007). https://doi.org/10.1007/s11416-006-0031-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-006-0031-z

Keywords

Search

Navigation