Skip to main content
SpringerLink
Log in

Dynamic emulation based modeling and detection of polymorphic shellcode at the network level

  • Published:
Science in China Series F: Information Sciences Aims and scope Submit manuscript

Abstract

It is a promising way to detect polymorphic shellcode using emulation method. However, previous emulation-based approaches are limited in their performance and resilience against evasions. A new enhanced emulation-based detection approach is proposed, including an automaton-based model of the dynamic behavior of polymorphic shellcode and a detection algorithm, the detection criterion of which is derived from that model and ensures high detection accuracy. The algorithm also contains several optimization techniques, highly improving the running performance and the resilience against detection evasion shellcode. We have implemented a prototype system for our approach. The advantages of our algorithm are validated by the experiments with real network data, polymorphic shellcode samples generated by available polymorphic engines and hand-crafted detection evasion shellcode.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Szor P, Ferrie P. Hunting for metamorphic. In: Proceedings of the Virus Bulletin Conference. Oxfordshire: Virus Bulletin Ltd, 2001. 123–144

    Google Scholar 

  2. Payer U, Lamberger M, Teufl P. Hybrid engine for polymorphic shellcode detection. In: Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA’05). Berlin: Springer-Verlag, 2005. 19–31

    Google Scholar 

  3. Chinchani R, Berg E. A fast static analysis approach to detect exploit code inside network flows. In: Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID’05). Berlin: Springer-Verlag, 2005. 284–308

    Google Scholar 

  4. Kruegel C, Kirda E, Mutz D, et al. Polymorphic worm detection using structural information of executables. In: Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID’05). Berlin: Springer-Verlag, 2005

    Google Scholar 

  5. Polychronakis, M, Anagnostakis K G, Markatos E P. Network-level polymorphic shellcode detection using emulation. In: Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment. Berlin: Springer-Verlag, 2006

    Google Scholar 

  6. Zhang Q, Reeves D S, Ning P, et al. Analyzing network traffic to detect self-decrypting exploit code. In: Proceedings of the 2nd ACM Symposium on InformAtion, Computer and Communications Security, New York: ACM, 2007. 4–12

    Chapter  Google Scholar 

  7. Akritidis P, Markatos E, Polychronakis M, et al. Stride: Polymorphic sled detection through instruction sequence analysis. In: Proceedings of the 20th IFIP International Information Security Conference (SEC’05). Boston: Springer, 2005. 375–392

    Google Scholar 

  8. Toth T, Kruegel C. Accurate buffer overflow detection via abstract payload execution. In: Proceedings of the 5th International Symposium on Recent Advances in Intrusion Detection (RAID’02). Berlin: Springer-Verlag, 2002. 274–291

    Chapter  Google Scholar 

  9. Payer U, Teufl P, Kraxberger S, et al. Massive data mining for polymorphic code detection. In: Mathematical Methods, Models and Architectures for Computer Network Security Workshop. Berlin: Springer-Verlag, 2005. 25–27

    Google Scholar 

  10. Pasupulati A, Coit J, Levitt K, et al. Buttercup: On network-based detection of polymorphic buffer overflow vulnerabilities. In: Proceedings of Network Operations and Management Symposium 2004. Washington: IEEE Computer Society, 2004

    Google Scholar 

  11. Li Z, Sanghi M, Chen Y, et al. Hamsa: Fast signature generation for zero-day polymorphic worms with provable attack resilience. In: Proceedings of 2006 IEEE Symposium on Security and Privacy (S&P’06). Washington: IEEE Computer Society, 2006. 32–47

    Google Scholar 

  12. Newsome J, Karp B, Song D. Polygraph: automatically generating signatures for polymorphic worms. In: Proceedings of 2005 IEEE Symposium on Security and Privacy (S&P’05). Washington: IEEE Computer Society, 2005. 226–241

    Google Scholar 

  13. Christodorescu M, Jha S, Seshia S A, et al. Bryant. Semantics-aware malware detection. In: Proceedings of 2005 IEEE Symposium on Security and Privacy (S&P’05). Washington: IEEE Computer Society, 2005. 32–46

    Google Scholar 

  14. Christodorescu M, Jha S. Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th USENIX Security Symposium. Berkeley: USENIX Association, 2003. 169–186

    Google Scholar 

  15. Intel Corporation. IA-32 Intel Architecture Software Developer’s Manual, 2006, Volume 2

Download references

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China

    LanJia Wang & Xing Li

  2. Network Research Center, Tsinghua University, Beijing, 100084, China

    HaiXin Duan

Authors
  1. LanJia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. HaiXin Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to LanJia Wang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, L., Duan, H. & Li, X. Dynamic emulation based modeling and detection of polymorphic shellcode at the network level. Sci. China Ser. F-Inf. Sci. 51, 1883–1897 (2008). https://doi.org/10.1007/s11432-008-0150-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11432-008-0150-x

Keywords

Search

Navigation