Abstract
It is a promising way to detect polymorphic shellcode using emulation method. However, previous emulation-based approaches are limited in their performance and resilience against evasions. A new enhanced emulation-based detection approach is proposed, including an automaton-based model of the dynamic behavior of polymorphic shellcode and a detection algorithm, the detection criterion of which is derived from that model and ensures high detection accuracy. The algorithm also contains several optimization techniques, highly improving the running performance and the resilience against detection evasion shellcode. We have implemented a prototype system for our approach. The advantages of our algorithm are validated by the experiments with real network data, polymorphic shellcode samples generated by available polymorphic engines and hand-crafted detection evasion shellcode.
Similar content being viewed by others
References
Szor P, Ferrie P. Hunting for metamorphic. In: Proceedings of the Virus Bulletin Conference. Oxfordshire: Virus Bulletin Ltd, 2001. 123–144
Payer U, Lamberger M, Teufl P. Hybrid engine for polymorphic shellcode detection. In: Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA’05). Berlin: Springer-Verlag, 2005. 19–31
Chinchani R, Berg E. A fast static analysis approach to detect exploit code inside network flows. In: Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID’05). Berlin: Springer-Verlag, 2005. 284–308
Kruegel C, Kirda E, Mutz D, et al. Polymorphic worm detection using structural information of executables. In: Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID’05). Berlin: Springer-Verlag, 2005
Polychronakis, M, Anagnostakis K G, Markatos E P. Network-level polymorphic shellcode detection using emulation. In: Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment. Berlin: Springer-Verlag, 2006
Zhang Q, Reeves D S, Ning P, et al. Analyzing network traffic to detect self-decrypting exploit code. In: Proceedings of the 2nd ACM Symposium on InformAtion, Computer and Communications Security, New York: ACM, 2007. 4–12
Akritidis P, Markatos E, Polychronakis M, et al. Stride: Polymorphic sled detection through instruction sequence analysis. In: Proceedings of the 20th IFIP International Information Security Conference (SEC’05). Boston: Springer, 2005. 375–392
Toth T, Kruegel C. Accurate buffer overflow detection via abstract payload execution. In: Proceedings of the 5th International Symposium on Recent Advances in Intrusion Detection (RAID’02). Berlin: Springer-Verlag, 2002. 274–291
Payer U, Teufl P, Kraxberger S, et al. Massive data mining for polymorphic code detection. In: Mathematical Methods, Models and Architectures for Computer Network Security Workshop. Berlin: Springer-Verlag, 2005. 25–27
Pasupulati A, Coit J, Levitt K, et al. Buttercup: On network-based detection of polymorphic buffer overflow vulnerabilities. In: Proceedings of Network Operations and Management Symposium 2004. Washington: IEEE Computer Society, 2004
Li Z, Sanghi M, Chen Y, et al. Hamsa: Fast signature generation for zero-day polymorphic worms with provable attack resilience. In: Proceedings of 2006 IEEE Symposium on Security and Privacy (S&P’06). Washington: IEEE Computer Society, 2006. 32–47
Newsome J, Karp B, Song D. Polygraph: automatically generating signatures for polymorphic worms. In: Proceedings of 2005 IEEE Symposium on Security and Privacy (S&P’05). Washington: IEEE Computer Society, 2005. 226–241
Christodorescu M, Jha S, Seshia S A, et al. Bryant. Semantics-aware malware detection. In: Proceedings of 2005 IEEE Symposium on Security and Privacy (S&P’05). Washington: IEEE Computer Society, 2005. 32–46
Christodorescu M, Jha S. Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th USENIX Security Symposium. Berkeley: USENIX Association, 2003. 169–186
Intel Corporation. IA-32 Intel Architecture Software Developer’s Manual, 2006, Volume 2
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, L., Duan, H. & Li, X. Dynamic emulation based modeling and detection of polymorphic shellcode at the network level. Sci. China Ser. F-Inf. Sci. 51, 1883–1897 (2008). https://doi.org/10.1007/s11432-008-0150-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11432-008-0150-x