Abstract
As state–of–the–art attack detection technology becomes more prevalent, attackers are likely to evolve, employing techniques such as polymorphism and metamorphism to evade detection. Although recent results have been promising, most existing proposals can be defeated using only minor enhancements to the attack vector. We present a heuristic detection method that scans network traffic streams for the presence of polymorphic shellcode. Our approach relies on a NIDS–embedded CPU emulator that executes every potential instruction sequence, aiming to identify the execution behavior of polymorphic shellcodes. Our analysis demonstrates that the proposed approach is more robust to obfuscation techniques like self-modifications compared to previous proposals, but also highlights advanced evasion techniques that need to be more closely examined towards a satisfactory solution to the polymorphic shellcode detection problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
sk: History and advances in windows shellcode. Phrack 11(62) (July 2004)
Kim, H.-A., Karp, B.: Autograph: Toward automated, distributed worm signature detection. In: Proceedings of the 13th USENIX Security Symposium, pp. 271–286 (2004)
Singh, S., Estan, C., Varghese, G., Savage, S.: Automated worm fingerprinting. In: Proceedings of the 6th Symposium on Operating Systems Design & Implementation (OSDI) (December 2004)
Newsome, J., Karp, B., Song, D.: Polygraph: Automatically Generating Signatures for Polymorphic Worms. In: Proceedings of the IEEE Security & Privacy Symposium, May 2005, pp. 226–241 (2005)
Tang, Y., Chen, S.: Defending against internet worms: a signature-based approach. In: Proceedings of the 24th Annual Joint Conference of IEEE Computer and Communication societies (INFOCOM) (2005)
Wang, K., Stolfo, S.J.: Anomalous Payload-Based Network Intrusion Detection. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 203–222. Springer, Heidelberg (2004)
Krügel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Polymorphic Worm Detection Using Structural Information of Executables. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 207–226. Springer, Heidelberg (2006)
Chinchani, R., van den Berg, E.: A fast static analysis approach to detect exploit code inside network flows. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 284–308. Springer, Heidelberg (2006)
Ször, P., Ferrie, P.: Hunting for metamorphic. In: Proceedings of the Virus Bulletin Conference, September 2001, pp. 123–144 (2001)
Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th USENIX Security Symposium (Security 2003) (August 2003)
Bania, P.: TAPiON (2005), http://pb.specialised.info/all/tapion/
Roesch, M.: Snort: Lightweight intrusion detection for networks. In: Proceedings of USENIX LISA 1999 (November 1999), software available from: http://www.snort.org/
Jordan, C.: Writing detection signatures. USENIX; login: 30(6), 55–61 (2005)
K2, ADMmutate (2001), http://www.ktwo.ca/ADMmutate-0.8.4.tar.gz
Detristan, T., Ulenspiegel, T., Malcom, Y., Underduk, M.: Polymorphic shellcode engine using spectrum analysis. Phrack 11(61) (August 2003)
Rix: Writing ia32 alphanumeric shellcodes. Phrack 11(57) (August 2001)
Tóth, T., Krügel, C.: Accurate Buffer Overflow Detection via Abstract Payload Execution. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, p. 274. Springer, Heidelberg (2002)
Akritidis, P., Markatos, E.P., Polychronakis, M., Anagnostakis, K.: STRIDE: Polymorphic Sled Detection through Instruction Sequence Analysis. In: Proceedings of the 20th IFIP International Information Security Conference (IFIP/SEC) (June 2005)
Crandall, J.R., Wu, S.F., Chong, F.T.: Experiences Using Minos as a Tool for Capturing and Analyzing Novel Worms for Unknown Vulnerabilities. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 32–50. Springer, Heidelberg (2005)
Pasupulati, A., Coit, J., Levitt, K., Wu, S., Li, S., Kuo, J., Fan, K.: Buttercup: On Network-based Detection of Polymorphic Buffer Overflow Vulnerabilities. In: Proceedings of the Network Operations and Management Symposium (NOMS), April 2004, pp. 235–248 (2004)
Kreibich, C., Crowcroft, J.: Honeycomb – creating intrusion detection signatures using honeypots. In: Proceedings of the Second Workshop on Hot Topics in Networks (HotNets-II) (November 2003)
Kolesnikov, O., Dagon, D., Lee, W.: Advanced polymorphic worms: Evading IDS by blending in with normal traffic, College of Computing, Georgia Institute of Technology, Atlanta, GA 30332 (2004), http://www.cc.gatech.edu/~ok/w/ok_pw.pdf
Payer, U., Teufl, P., Lamberger, M.: Hybrid Engine for Polymorphic Shellcode Detection. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 19–31. Springer, Heidelberg (2005)
Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM conference on Computer and communications security (CCS), pp. 290–299 (2003)
Aycock, J., de Graaf, R., Jacobson, M.: Anti-disassembly using cryptographic hash functions. Department of Computer Science, University of Calgary, Tech. Rep. 2005-793-24
Venable, M., Chouchane, M.R., Karim, M.E., Lakhotia, A.: Analyzing Memory Accesses in Obfuscated x86 Executables. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 1–18. Springer, Heidelberg (2005)
Collberg, C.S., Thomborson, C.: Watermarking, tamper-proffing, and obfuscation: tools for software protection. IEEE Transactions on Software Engineering 28(8), 735–746 (2002)
Wang, C., Hill, J., Knight, J., Davidson, J.: Software tamper resistance: Obstructing static analysis of programs. University of Virginia, Tech. Rep. CS-2000-12 (2000)
Madou, M., Anckaert, B., Moseley, P., Debray, S., De Sutter, B., De Bosschere, K.: Software protection through dynamic code mutation. In: Song, J.-S., Kwon, T., Yung, M. (eds.) WISA 2005. LNCS, vol. 3786, pp. 194–206. Springer, Heidelberg (2006)
Schwarz, B., Debray, S., Andrews, G.: Disassembly of executable code revisited. In: Proceedings of the Ninth Working Conference on Reverse Engineering (WCRE) (2002)
Prasad, M., cker Chiueh, T.: A binary rewriting defense against stack based overflow attacks. In: Proceedings of the USENIX Annual Technical Conference (June 2003)
Kruegel, C., Robertson, W., Valeur, F., Vigna, G.: Static disassembly of obfuscated binaries. In: Proceedings of the USENIX Security Symposium, August 2004, pp. 255–270 (2004)
Cohen, F.B.: Operating system protection through program evolution. Computer and Security 12(6), 565–584 (1993)
Metasploit Project (2006), http://www.metasploit.com/
Cifuentes, C., Gough, K.J.: Decompilation of binary programs. Software—Practice and Experience 25(7), 811–829 (1995)
Balakrishnan, G., Reps, T.: Analyzing Memory Accesses in x86 Executables. In: Duesterwald, E. (ed.) CC 2004. LNCS, vol. 2985, pp. 5–23. Springer, Heidelberg (2004)
Noir, GetPC code (was: Shellcode from ASCII) (June 2003), http://www.securityfocus.com/archive/82/327100/2006-01-03/1
Ionescu, C.: GetPC code (was: Shellcode from ASCII) (July 2003), http://www.securityfocus.com/archive/82/327348/2006-01-03/1
Wever, B.-J.: Alpha 2 (2004), http://www.edup.tudelft.nl/~bjwever/src/alpha2.c
Perriot, F., Ferrie, P., Ször, P.: Striking similarities. Virus Bulletin, 4–6 (May 2002)
Obscou: Building ia32 ’unicode-proof’ shellcodes. Phrack 11(61) (August 2003)
Tubella, J., González, A.: Control speculation in multithreaded processors through dynamic loop detection. In: Proceedings of the 4th International Symposium on High-Performance Computer Architecture (HPCA) (1998)
McCanne, S., Leres, C., Jacobson, V.: Libpcap (2006), http://www.tcpdump.org/
Wojtczuk, R.: Libnids (2006), http://libnids.sourceforge.net/
jt: Libdasm (2006), http://www.klake.org/~jt/misc/libdasm-1.4.tar.gz
Apache Chunked Encoding Overflow (2002), http://www.osvdb.org/838
Microsoft Windows RPC DCOM Interface Overflow (2003), http://www.osvdb.org/2100
Microsoft Windows LSASS Remote Overflow (2004), http://www.osvdb.org/5248
Bell, J.R.: Threaded code. Comm. of the ACM 16(6), 370–372 (1973)
Bellard, F.: QEMU, a Fast and Portable Dynamic Translator. In: Proceedings of the USENIX Annual Technical Conference, FREENIX Track, pp. 41–46 (2005)
Bhatkar, S., DuVarney, D.C., Sekar, R.: Address Obfuscation: An Efficient Approach to Combat a Broad Range of Memory Error Exploits. In: Proceedings of the 12th USENIX Security Symposium (2003)
Anagnostakis, K., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., Keromytis, A.D.: Detecting Targeted Attacks Using Shadow Honeypots. In: Proceedings of the 14th USENIX Security Symposium, August 2005, pp. 129–144 (2005)
Dreger, H., Kreibich, C., Paxson, V., Sommer, R.: Enhancing the Accuracy of Network-Based Intrusion Detection with Host-Based Context. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 206–221. Springer, Heidelberg (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Polychronakis, M., Anagnostakis, K.G., Markatos, E.P. (2006). Network–Level Polymorphic Shellcode Detection Using Emulation. In: Büschkes, R., Laskov, P. (eds) Detection of Intrusions and Malware & Vulnerability Assessment. DIMVA 2006. Lecture Notes in Computer Science, vol 4064. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11790754_4
Download citation
DOI: https://doi.org/10.1007/11790754_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-36014-8
Online ISBN: 978-3-540-36017-9
eBook Packages: Computer ScienceComputer Science (R0)