skip to main content
10.1145/2046707.2046739acmconferencesArticle/Chapter ViewAbstractPublication PagesccsConference Proceedingsconference-collections
research-article

Deobfuscation of virtualization-obfuscated software: a semantics-based approach

Published: 17 October 2011 Publication History

Abstract

When new malware are discovered, it is important for researchers to analyze and understand them as quickly as possible. This task has been made more difficult in recent years as researchers have seen an increasing use of virtualization-obfuscated malware code. These programs are difficult to comprehend and reverse engineer, since they are resistant to both static and dynamic analysis techniques. Current approaches to dealing with such code first reverse-engineer the byte code interpreter, then use this to work out the logic of the byte code program. This outside-in approach produces good results when the structure of the interpreter is known, but cannot be applied to all cases. This paper proposes a different approach to the problem that focuses on identifying instructions that affect the observable behavior of the obfuscated code. This inside-out approach requires fewer assumptions, and aims to complement existing techniques by broadening the domain of obfuscated programs eligible for automated analysis. Results from a prototype tool on real-world malicious code are encouraging.

References

[1]
VX Heavens, 2011. http://vx.netlux.org/.
[2]
A. V. Aho, R. Sethi, and J. D. Ullman. phCompilers -- Principles, Techniques, and Tools. Addison-Wesley, Reading, Mass., 1985.
[3]
F. Bellard. QEMU, a fast and portable dynamic translator. In phUSENIX Annual Technical Conference, FREENIX Track, pages 41--46. USENIX, 2005.
[4]
K. Coogan and S. Debray. Equational reasoning on x86 assembly code. Source Code Analysis and Manipulation, IEEE International Workshop on, 2011.
[5]
A. Dinaburg, P. Royal, M. I. Sharif, and W. Lee. Ether: malware analysis via hardware virtualization extensions. In Proceedings of the 2008 ACM Conference on Computer and Communications Security, CCS 2008, Alexandria, Virginia, USA, October 27--31, 2008, pages 51--62, 2008.
[6]
N. Falliere. Inside the jaws of Trojan.Clampi. Technical report, Symantec Corp., Nov. 2009.
[7]
N. D. Jones, C. K. Gomard, and P. Sestoft. phPartial Evaluation and Automatic Program Generation. Prentice Hall, 1993.
[8]
A. Lakhotia, E. U. Kumar, and M. Venable. A method for detecting obfuscated calls in malicious binaries. IEEE Transactions on Software Engineering, 31 (11): 955--968, 2005.
[9]
B. Lau. Dealing with virtualization packer. In Second CARO Workshop on Packers, Decryptors, and Obfuscators, May 2008.
[10]
A. Moser, C. Kruegel, and E. Kirda. Exploring multiple execution paths for malware analysis. In SP '07: Proceedings of the 2007 IEEE Symposium on Security and Privacy, pages 231--245, 2007.
[11]
Oreans Technologies. Code virtualizer: Total obfuscation against reverse engineering, Dec. 2008. http://www.oreans.com/codevirtualizer.php.
[12]
R. Rolles. Unpacking virtualization obfuscators. In Proc. 3rd USENIX Workshop on Offensive Technologies (WOOT '09), Aug. 2009.
[13]
M. Sharif, A. Lanzi, J. Giffin, and W. Lee. Automatic reverse engineering of malware emulators. In Proc. 2009 IEEE Symposium on Security and Privacy, May 2009.
[14]
F. Tip. A survey of program slicing techniques. Journal of Programming Languages, 3: 121--189, 1995.
[15]
S. K. Udupa, S. K. Debray, and M. Madou. Deobfuscation: Reverse engineering obfuscated code. In Proc. 12th IEEE Working Conference on Reverse Engineering, pages 45--54, Nov. 2005.
[16]
VMProtect Software. Vmprotect software protection, 2008. http://vmpsoft.com/.
[17]
C. Wang, J. Davidson, J. Hill, and J. Knight. Protection of software-based survivability mechanisms. In Proc. International Conference of Dependable Systems and Networks, July 2001.

Cited By

View all
  • (2024)Comparing malware evasion theory with practiceProceedings of the Twentieth USENIX Conference on Usable Privacy and Security10.5555/3696899.3696903(61-80)Online publication date: 12-Aug-2024
  • (2024)Control-Flow Deobfuscation using Trace-Informed Compositional Program SynthesisProceedings of the ACM on Programming Languages10.1145/36897898:OOPSLA2(2211-2241)Online publication date: 8-Oct-2024
  • (2024)Mitigating Debugger-based Attacks to Java Applications with Self-debuggingACM Transactions on Software Engineering and Methodology10.1145/363197133:4(1-38)Online publication date: 18-Apr-2024
  • Show More Cited By

Index Terms

  1. Deobfuscation of virtualization-obfuscated software: a semantics-based approach

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CCS '11: Proceedings of the 18th ACM conference on Computer and communications security
      October 2011
      742 pages
      ISBN:9781450309486
      DOI:10.1145/2046707
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 17 October 2011

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. deobfuscation
      2. dynamic analysis
      3. virtualization

      Qualifiers

      • Research-article

      Conference

      CCS'11
      Sponsor:

      Acceptance Rates

      CCS '11 Paper Acceptance Rate 60 of 429 submissions, 14%;
      Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

      Upcoming Conference

      CCS '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)57
      • Downloads (Last 6 weeks)3
      Reflects downloads up to 14 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Comparing malware evasion theory with practiceProceedings of the Twentieth USENIX Conference on Usable Privacy and Security10.5555/3696899.3696903(61-80)Online publication date: 12-Aug-2024
      • (2024)Control-Flow Deobfuscation using Trace-Informed Compositional Program SynthesisProceedings of the ACM on Programming Languages10.1145/36897898:OOPSLA2(2211-2241)Online publication date: 8-Oct-2024
      • (2024)Mitigating Debugger-based Attacks to Java Applications with Self-debuggingACM Transactions on Software Engineering and Methodology10.1145/363197133:4(1-38)Online publication date: 18-Apr-2024
      • (2024)COVER: Enhancing virtualization obfuscation through dynamic scheduling using flash controller-based secure moduleComputers & Security10.1016/j.cose.2024.104038146(104038)Online publication date: Nov-2024
      • (2023)Securing Embedded Devices through Obfuscation with Predictable Size and Execution OverheadProceedings of the 2023 International Conference on embedded Wireless Systems and Networks10.5555/3639940.3640007(421-426)Online publication date: 15-Dec-2023
      • (2023)Function-Level Code Obfuscation Detection Through Self-Attention-Guided Multi-Representation FusionInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402350066334:04(651-673)Online publication date: 11-Dec-2023
      • (2023)Static Analysis of JNI Programs via Binary DecompilationIEEE Transactions on Software Engineering10.1109/TSE.2023.324163949:5(3089-3105)Online publication date: 1-May-2023
      • (2023)Reverse Engineering of Obfuscated Lua Bytecode via Interpreter Semantics TestingIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.328925418(3891-3905)Online publication date: 2023
      • (2023)No Free Lunch: On the Increased Code Reuse Attack Surface of Obfuscated Programs2023 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)10.1109/DSN58367.2023.00039(313-326)Online publication date: Jun-2023
      • (2023)Evaluating Defensive Countermeasures for Software-Based Hardware AbstractionE-Business and Telecommunications10.1007/978-3-031-36840-0_13(281-304)Online publication date: 22-Jul-2023
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media