Skip to main content
SpringerLink
Log in

Specification and evaluation of polymorphic shellcode properties using a new temporal logic

  • Original Paper
  • Published:
Journal in Computer Virology Aims and scope Submit manuscript

Abstract

It is a well-known fact that polymorphism is one of the greatest find of malicious code authors. Applied in the context of Buffer Overflow attacks, the detection of such codes becomes very difficult. In view of this problematic, which constitutes a real challenge for all the international community, we propose in this paper a new formal language (based on temporal logics such as CTL) allowing to specify polymorphic codes, to detect them and to better understand their nature. The efficiency and the expressiveness of this language are shown via the specification of a variety of properties characterizing polymorphic shellcodes. Finally, to make the verification process automatic, this language is supported by a new IDS (Intrusion Detection System) that will also be presented in this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. CAN-2002-0392 - apache chunked-encoding memory corruption vulnerability. http://www.securityfocus.com/bid/5033/discuss

  2. Flawfinder. http://www.dwheeler.com/flawfinder

  3. IA-32 intel architecture software developer’s manual-instruction set reference. http://www.intel.com/design/pentium4/manuals/index_new.htm

  4. The lex & yacc page. http://dinosaur.compilertools.net/

  5. Metasploit. http://www.metasploit.com/

  6. MIT lincoln laboratory. http://www.ll.mit.edu/

  7. National vulnerability database. http://nvd.nist.gov/statistics.cfm

  8. Nessus. http://www.nessus.org

  9. Openwall. http://www.openwall.com/

  10. PAX. http://pax.grsecurity.net/docs/index.html

  11. Rats. http://www.securesoftware.com

  12. Retina. http://www.eeye.com

  13. Stack Shield. http://www.angelfire.com/sk/stackshield/

  14. US-CERT. http://www.us-cert.gov/

  15. Adi K., Debbabi M., Mejri M.: A new logic for electronic commerce protocols. Theor. Comput. Sci. 291(3), 223–283 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  16. Akritidis, P., Evangelos, P., Markatos, Polychronakis, M., Kostas, G., Anagnostakis: STRIDE: Polymorphic sled detection through instruction sequence analysis. In: SEC, pp. 375–392 (2005)

  17. Aleph1. Smashing the stack for fun and profit. http://www.phrack.org/issues.html?issue=49&id=14

  18. Bailleux, C., Grenie, C.: Protections contre l’exploitation des débordements de buffer - bibliothèques et compilateurs. http://www.miscmag.com/

  19. Baratloo, A., Singh, N., Tsai, T.: Libsafe: Protecting critical elements of stacks

  20. Beaucamps P., Filiol E.: On the possibility of practically obfuscating programs towards a unified perspective of code protection. J. Comput. Virol. 3(1), 3–21 (2007)

    Article  Google Scholar 

  21. Bulba and Kil3r. Bypassing Stackguard and Stackshield. http://www.phrack.org/issues.html?issue=56&id=5

  22. Cowan, C., Pu, C., Maier, D., Hintony, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., Zhang, Q.: Stackguard: automatic adaptive detection and prevention of buffer-overflow attacks. In: SSYM. USENIX Association (1998)

  23. Solar Designer. Getting around non-executable stack (and fix). http://www.securityfocus.com/archive/1/7480

  24. Detristan, T., Ulenspiegel, T., Malcom, Y., Superbus M., Von Underduk.: Polymorphic shellcode engine using spectrum analysis. http://www.phrack.org/issues.html?issue=61&id=9

  25. Filiol E.: Metamorphism, formal grammars and undecidable code mutation. Int. J. Comput. Sci. 2(1), 70–75 (2007)

    Google Scholar 

  26. Ben Ghorbel, M., Talbi M., Mejri, M.: Specification and detection of TCP/IP based attacks using the ADM-logic. In: ARES, pp. 206–212. IEEE Computer Society (2007)

  27. Gushin, Y.: Nids polymorphic evasion - the end? http://www.ecl-labs.org/papers.html

  28. K2. Admmutate. http://www.ktwo.ca/

  29. Kolesnikov, O., Lee, W.: Advanced polymorphic worms: Evading IDS by blending in with normal traffic (2004)

  30. Kripke S.A.: Semantical considerations in modal logic. Acta Philosophica Fenica 16, 83–94 (1963)

    MATH  MathSciNet  Google Scholar 

  31. Krügel, C., Kirda, E., Mutz, D., Robertson, W.K., Vigna, G.: Polymorphic worm detection using structural information of executables. In: RAID, pp. 207–226 (2005)

  32. Lespérance, P.L.: Detecting variants of known attacks using temporal logic. In: WPTACT (2005)

  33. Lions, J.L.: ARIANE 5: Flight 501 failure. http://sunnyday.mit.edu/accidents/Ariane5accidentreport.html

  34. McHugh J.: Testing intrusion detection systems: a critique of the 1998 and 1999 darpa intrusion detection system evaluations as performed by lincoln laboratory. ACM Trans. Inform. Syst. Security 3(4), 262–294 (2000)

    Article  Google Scholar 

  35. Newsome, J., Karp, B., Xiaodong Song, D.: Polygraph: automatically generating signatures for polymorphic worms. In: IEEE Symposium on Security and Privacy, pp. 226–241 (2005)

  36. Payer, U., Kraxberger, S.: Polymorphic code detection with GA optimized markov models. In: Communications and Multimedia Security, pp. 210–219 (2005)

  37. Payer, U., Teufl, P., Lamberger, M.: Hybrid engine for polymorphic shellcode detection. In: Julisch, K., Krügel, C. (eds.) DIMVA, vol. 3548. Lecture Notes in Computer Science, pp. 19–31. Springer, Berlin (2005)

  38. Plotkin, G.D.: A structural approach to operational semantics. Technical Report DAIMI FN-19, University of Aarhus (1981)

  39. Writing, R.: IA32 alphanumeric shellcodes. http://www.phrack.org/issues.html?issue=57&id=15

  40. Ruiu, D.: Snort preprocessor—multi-architecture mutated NOP sled detector

  41. Sedalo, M.: JempiScode. http://goodfellas.shellcode.com.ar/proyectos.html

  42. Stirling, C.: Modal and temporal logics for processes. In: Proceedings of the VIII Banff Higher order workshop conference on Logics for concurrency : structure versus automata, pp. 149–237. Springer, Berlin (1996)

  43. Talbi, M.: IDS-logic. http://www.rennes.supelec.fr/ren/perso/mtalbi/outils/IDS-Logic.tar.gz

  44. Toth, T., Krügel, C.: Accurate buffer overflow detection via abstract payload execution. In: RAID, pp. 274–291 (2002)

  45. Wojtczuk, R.: The advanced return-into-lib(c) exploits: PAX case study. http://www.phrack.org/issues.html?issue=58&id=4

Download references

Author information

Authors and Affiliations

  1. Digital Security Unit, Higher School of Communications of Tunis, Tunis, Tunisia

    Mehdi Talbi & Adel Bouhoula

  2. LSFM Research Group, Computer Security Department, Laval University, Quebec, QC, Canada

    Mohamed Mejri

Authors
  1. Mehdi Talbi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Mejri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adel Bouhoula
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehdi Talbi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Talbi, M., Mejri, M. & Bouhoula, A. Specification and evaluation of polymorphic shellcode properties using a new temporal logic. J Comput Virol 5, 171–186 (2009). https://doi.org/10.1007/s11416-008-0089-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-008-0089-x

Keywords

Search

Navigation