research-article

Comprehensive shellcode detection using runtime heuristics

Authors:

Michalis Polychronakis,

Kostas G. Anagnostakis,

Evangelos P. MarkatosAuthors Info & Claims

ACSAC '10: Proceedings of the 26th Annual Computer Security Applications Conference

Pages 287 - 296

https://doi.org/10.1145/1920261.1920305

Published: 06 December 2010 Publication History

Abstract

A promising method for the detection of previously unknown code injection attacks is the identification of the shellcode that is part of the attack vector using payload execution. Existing systems based on this approach rely on the self-decrypting behavior of polymorphic code and can identify only that particular class of shellcode. Plain, and more importantly, metamorphic shellcode do not carry a decryption routine nor exhibit any self-modifications and thus both evade existing detection systems. In this paper, we present a comprehensive shellcode detection technique that uses a set of runtime heuristics to identify the presence of shellcode in arbitrary data streams. We have identified fundamental machine-level operations that are inescapably performed by different shellcode types, based on which we have designed heuristics that enable the detection of plain and metamorphic shellcode regardless of the use of self-decryption. We have implemented our technique in Gene, a code injection attack detection system based on passive network monitoring. Our experimental evaluation and real-world deployment show that Gene can effectively detect a large and diverse set of shellcode samples that are currently missed by existing detectors, while so far it has not generated any false positives.

References

[1]

Goodfellas security research team. http://goodfellas.shellcode.com.ar/.

[2]

The metasploit project. http://www.metasploit.com/.

[3]

milw0rm. http://milw0rm.com/shellcode/win32/.

[4]

Packet storm. http://www.packetstormsecurity.org/.

[5]

Win32 assembly components, Dec. 2002. http://lsd-pl.net.

[6]

Common shellcode naming initiative, 2009. http://nepenthes.carnivore.it/csni.

[7]

Retrieving kernel32's base address, June 2009. http://www.harmonysecurity.com/blog/2009/06/retrieving-kernel32s-base-address.html.

[8]

S. Andersson, A. Clark, and G. Mohay. Network-based buffer overflow detection by exploit code analysis. In Proceedings of the Asia Pacific Information Technology Security Conference (AusCERT), 2004.

[9]

P. Baecher and M. Koetter. libemu, 2009. http://libemu.carnivore.it/.

[10]

P. Bania. Evading network-level emulation, 2009. http://piotrbania.com/all/articles/pbania-evading-nemu2009.pdf.

[11]

K. Borders, A. Prakash, and M. Zielinski. Spector: Automatically analyzing shell code. In Proceedings of the Annual Computer Security Applications Conference (ACSAC), 2007.

[12]

R. Chinchani and E. V. D. Berg. A fast static analysis approach to detect exploit code inside network flows. In Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID), Sept. 2005.

Digital Library

[13]

S. P. Chung and A. K. Mok. Swarm attacks against network-level emulation/analysis. In Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection (RAID), September 2008.

Digital Library

[14]

M. Egele, P. Wurzinger, C. Kruegel, and E. Kirda. Defending browsers against drive-by downloads: Mitigating heap-spraying code injection attacks. In Proceedings of the 6th international conference on Detection of Intrusions and Malware, & Vulnerability Assessment (DIMVA), 2009.

Digital Library

[15]

S. Ford, M. Cova, C. Kruegel, and G. Vigna. Wepawet, 2009. http://wepawet.cs.ucsb.edu/.

[16]

I)ruid. Context-keyed payload encoding. Uninformed, 9, Oct. 2007.

[17]

C. Kruegel, E. Kirda, D. Mutz, W. Robertson, and G. Vigna. Polymorphic worm detection using structural information of executables. In Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID), Sept. 2005.

Digital Library

[18]

J. Ma, J. Dunagan, H. J. Wang, S. Savage, and G. M. Voelker. Finding diversity in remote code injection exploits. In Proceedings of the 6th Internet Measurement Conference (IMC), 2006.

Digital Library

[19]

J. Mason, S. Small, F. Monrose, and G. MacManus. English shellcode. In Proceedings of the 16th ACM conference on Computer and communications security (CCS), 2009.

Digital Library

[20]

U. Payer, P. Teufl, and M. Lamberger. Hybrid engine for polymorphic shellcode detection. In Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA), pages 19--31, July 2005.

Digital Library

[21]

M. Pietrek. A crash course on the depths of Win32#8482;structured exception handling, 1997. http://www.microsoft.com/msj/0197/exception/exception.aspx.

[22]

M. Polychronakis, K. G. Anagnostakis, and E. P. Markatos. An empirical study of real-world polymorphic code injection attacks. In Proceedings of the 2nd USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET), April 2009.

Digital Library

[23]

M. Polychronakis, E. P. Markatos, and K. G. Anagnostakis. Network-level polymorphic shellcode detection using emulation. In Proceedings of the Third Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA), July 2006.

Digital Library

[24]

M. Polychronakis, E. P. Markatos, and K. G. Anagnostakis. Emulation-based detection of non-self-contained polymorphic shellcode. In Proceedings of the 10th International Symposium on Recent Advances in Intrusion Detection (RAID), September 2007.

Digital Library

[25]

M. Shimamura and K. Kono. Yataglass: Network-level code emulation for analyzing memory-scanning attacks. In Proceedings of the 6th international conference on Detection of Intrusions and Malware, & Vulnerability Assessment (DIMVA), 2009.

Digital Library

[26]

sk. History and advances in windows shellcode. Phrack, 11(62), July 2004.

[27]

Skape. Understanding windows shellcode, 2003. http://www.hick.org/code/skape/papers/win32-shellcode.pdf.

[28]

Skape. Safely searching process virtual address space, 2004. http://www.hick.org/code/skape/papers/egghunt-shellcode.pdf.

[29]

SkyLined. Finding the base address of kernel32 in Windows 7. http://skypher.com/index.php/2009/07/22/shellcode-finding-kernel32-in-windows-7/.

[30]

SkyLined. SEH GetPC (XP SP3), July 2009. http://skypher.com/wiki/index.php/Hacking/Shellcode/Alphanumeric/ALPHA3/x86/ASCII/Mixedcase/SEH_GetPC_(XP_sp3).

[31]

Y. Song, M. E. Locasto, A. Stavrou, A. D. Keromytis, and S. J. Stolfo. On the infeasibility of modeling polymorphic shellcode. In Proceedings of the 14th ACM conference on Computer and communications security (CCS), 2007.

Digital Library

[32]

P. Ször. The Art of Computer Virus Research and Defense. Addison-Wesley Professional, February 2005.

Digital Library

[33]

T. Toth and C. Kruegel. Accurate buffer overflow detection via abstract payload execution. In Proceedings of the 5th Symposium on Recent Advances in Intrusion Detection (RAID), Oct. 2002.

Digital Library

[34]

X. Wang, Y.-C. Jhi, S. Zhu, and P. Liu. Still: Exploit code detection via static taint and initialization analyses. In Proceedings of the Annual Computer Security Applications Conference (ACSAC), 2008.

Digital Library

[35]

X. Wang, C.-C. Pan, P. Liu, and S. Zhu. Sigfree: A signature-free buffer overflow attack blocker. In Proceedings of the USENIX Security Symposium, Aug. 2006.

Digital Library

[36]

B.-J. Wever. SEH Omelet Shellcode, 2009. http://code.google.com/p/w32-seh-omelet-shellcode/.

[37]

G. Wicherski. Win32 egg search shellcode, 33 bytes. http://blog.oxff.net/2009/02/win32-egg-search-shellcode-33-bytes.html.

[38]

Q. Zhang, D. S. Reeves, P. Ning, and S. P. Lyer. Analyzing network traffic to detect self-decrypting exploit code. In Proceedings of the 2nd ACM Symposium on Information, Computer and Communications Security (ASIACCS), 2007.

Digital Library

Cited By

Liu TZhang YZhao ZDong YMeng GChen KBalzarotti DXu W(2024)Making them ask and answerProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699164(4711-4728)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699164
Sayadi HHe Z(2024)Adapt and Defend: Reinforcement Learning for Hardware-Assisted SecurityAI-Enabled Electronic Circuit and System Design10.1007/978-3-031-71436-8_12(427-466)Online publication date: 17-Oct-2024
https://doi.org/10.1007/978-3-031-71436-8_12
Zhou Y(2024)Least Information Redundancy Algorithm of Printable Shellcode Encoding for X86Computer Security. ESORICS 2023 International Workshops10.1007/978-3-031-54129-2_21(361-376)Online publication date: 12-Mar-2024
https://doi.org/10.1007/978-3-031-54129-2_21
Show More Cited By

Index Terms

Comprehensive shellcode detection using runtime heuristics
1. Security and privacy
2. Social and professional topics
  1. Computing / technology policy
    1. Computer crime

Recommendations

English shellcode
CCS '09: Proceedings of the 16th ACM conference on Computer and communications security

History indicates that the security community commonly takes a divide-and-conquer approach to battling malware threats: identify the essential and inalienable components of an attack, then develop detection and prevention techniques that directly target ...
Network–Level polymorphic shellcode detection using emulation
DIMVA'06: Proceedings of the Third international conference on Detection of Intrusions and Malware & Vulnerability Assessment

As state–of–the–art attack detection technology becomes more prevalent, attackers are likely to evolve, employing techniques such as polymorphism and metamorphism to evade detection. Although recent results have been promising, most existing proposals ...
Detection of malicious web pages based on hybrid analysis

Malicious web pages have become an increasingly serious threat to web security in recent years. In this paper, we propose a new detection method that consists of static and dynamic analyses for detecting malicious web pages. Static analysis utilizes ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACSAC '10: Proceedings of the 26th Annual Computer Security Applications Conference

December 2010

419 pages

ISBN:9781450301336

DOI:10.1145/1920261

Conference Chair:
Carrie Gates
CA Labs
,
Program Chairs:
Michael Franz
University of California, Irvine
,
John McDermott
Naval Research Lab

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACSA: Applied Computing Security Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 December 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ACSAC '10

Sponsor:

ACSA

ACSAC '10: 2010 Annual Computer Security Applications Conference

December 6 - 10, 2010

Texas, Austin, USA

Acceptance Rates

Overall Acceptance Rate 104 of 497 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
544
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu TZhang YZhao ZDong YMeng GChen KBalzarotti DXu W(2024)Making them ask and answerProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699164(4711-4728)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699164
Sayadi HHe Z(2024)Adapt and Defend: Reinforcement Learning for Hardware-Assisted SecurityAI-Enabled Electronic Circuit and System Design10.1007/978-3-031-71436-8_12(427-466)Online publication date: 17-Oct-2024
https://doi.org/10.1007/978-3-031-71436-8_12
Zhou Y(2024)Least Information Redundancy Algorithm of Printable Shellcode Encoding for X86Computer Security. ESORICS 2023 International Workshops10.1007/978-3-031-54129-2_21(361-376)Online publication date: 12-Mar-2024
https://doi.org/10.1007/978-3-031-54129-2_21
Zhao SYu XLuo JXie G(2023)Speculation-Free Function Table Construction in LLVM IR for Fine-Grained Control Flow IntegrityJournal of Circuits, Systems and Computers10.1142/S021812662350281X32:16Online publication date: 29-May-2023
https://doi.org/10.1142/S021812662350281X
El-Zoghby ASaid Elsayed MJurcut AAzer M(2023)NG-MVEE: A New Proposed Hybrid Technique for Enhanced Mitigation of Code Re-Use AttackIEEE Access10.1109/ACCESS.2023.326988111(48169-48191)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3269881
Yang GChen XZhou YYu C(2022)DualSC: Automatic Generation and Summarization of Shellcode via Transformer and Dual Learning2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00052(361-372)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00052
Karmous NAoueileyine MAbdelkader MYoussef N(2022)IoT Real-Time Attacks Classification Framework Using Machine Learning2022 IEEE Ninth International Conference on Communications and Networking (ComNet)10.1109/ComNet55492.2022.9998441(1-5)Online publication date: 1-Nov-2022
https://doi.org/10.1109/ComNet55492.2022.9998441
Liu QBao KHagenmeyer V(2022)Binary Exploitation in Industrial Control Systems: Past, Present and FutureIEEE Access10.1109/ACCESS.2022.317192210(48242-48273)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3171922
Lu XWang FJiang CLio P(2021)A Universal Malicious Documents Static Detection Framework Based on Feature GeneralizationApplied Sciences10.3390/app11241213411:24(12134)Online publication date: 20-Dec-2021
https://doi.org/10.3390/app112412134
USUI TIKUSE TOTSUKI YKAWAKOYA YIWAMURA MMIYOSHI JMATSUURA K(2020)ROPminer: Learning-Based Static Detection of ROP Chain Considering Linkability of ROP GadgetsIEICE Transactions on Information and Systems10.1587/transinf.2019ICP0016E103.D:7(1476-1492)Online publication date: 1-Jul-2020
https://doi.org/10.1587/transinf.2019ICP0016
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten