Skip to main content
SpringerLink
Log in

Assisting Malware Analysis with Symbolic Execution: A Case Study

  • Conference paper
  • First Online:
Cyber Security Cryptography and Machine Learning (CSCML 2017)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 10332))

Abstract

Security analysts spend days or even weeks in trying to understand the inner workings of malicious software, using a plethora of manually orchestrated tools. Devising automated tools and techniques to assist and speed up the analysis process remains a major endeavor in computer security. While manual intervention will likely remain a key ingredient in the short and mid term, the recent advances in static and dynamic analysis techniques have the potential to significantly impact the malware analysis practice. In this paper we show how an analyst can use symbolic execution techniques to unveil critical behavior of a remote access trojan (RAT). Using a tool we implemented in the Angr framework, we analyze a sample drawn from a well-known RAT family that leverages thread injection vulnerabilities in the Microsoft Win32 API. Our case study shows how to automatically derive the list of commands supported by the RAT and the sequence of system calls that are activated for each of them, systematically exploring the stealthy communication protocol with the server and yielding clues to potential threats that may pass unnoticed by a manual inspection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A context can formally be defined in terms of live variables, i.e., the set of locations that execution paths starting at SEP might read from before writing to.

References

  1. Baldoni, R., Coppa, E., D’Elia, D.C., Demetrescu, C., Finocchi, I.: A survey of symbolic execution techniques. CoRR, abs/1610.00502 (2016)

    Google Scholar 

  2. Ball, T., Bounimova, E., Cook, B., Levin, V., Lichtenberg, J., McGarvey, C., Ondrusek, B., Rajamani, S.K., Ustuner, A.: Thorough static analysis of device drivers. In: Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems, EuroSys 2006, pp. 73–85. ACM, New York (2006)

    Google Scholar 

  3. Brumley, D., Hartwig, C., Kang, M.G., Liang, Z., Newsome, J., Poosankam, P., Song, D., Yin, H.: Bitscope: automatically dissecting malicious binaries. Technical report, CMU-CS-07-133 (2007)

    Google Scholar 

  4. Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D., Yin, H.: Automatically identifying trigger-based behavior in malware. In: Lee, W., Wang, C., Dagon, D. (eds.) Botnet Detection, pp. 65–88. Springer, Boston (2008)

    Chapter  Google Scholar 

  5. Caballero, J., Yin, H., Liang, Z., Song, D.: Polyglot: automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of the 14th ACM Conference on Computer and Communications Security, CCS 2007, pp. 317–329. ACM, New York (2007)

    Google Scholar 

  6. Cadar, C., Dunbar, D., Engler, D.: KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI 2008, pp. 209–224. USENIX Association, Berkeley (2008)

    Google Scholar 

  7. Cho, C.Y., Shin, E.C.R., Song, D.: Inference and analysis of formal models of botnet command and control protocols. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 426–439. ACM, New York (2010)

    Google Scholar 

  8. Christodorescu, M., Jha, S.: Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th Conference on USENIX Security Symposium, SSYM 2003, vol. 12. USENIX Association, Berkeley (2003)

    Google Scholar 

  9. Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantics-aware malware detection. In: Proceedings of the 2005 IEEE Symposium on Security and Privacy, SP 2005, pp. 32–46. IEEE Computer Society, Washington, DC (2005)

    Google Scholar 

  10. Cui, W., Kannan, J., Wang, H.J.: Discoverer: automatic protocol reverse engineering from network traces. In: Proceedings of 16th USENIX Security Symposium on USENIX Security Symposium, SS 2007, pp. 14:1–14:14. USENIX Association, Berkeley (2007)

    Google Scholar 

  11. Cui, W., Peinado, M., Chen, K., Wang, J.H. and Irun-Briz, L.: Automatic reverse engineering of input formats. In: Proceedings of the 15th ACM Conference on Computer and Communications Security, CCS 2008, pp. 391–402. ACM, New York (2008)

    Google Scholar 

  12. Egele, M., Scholte, T., Kirda, E., Kruegel, C.: A survey on automated dynamic malware-analysis techniques and tools. ACM Comput. Surv. 44(2), 6:1–6:42 (2008)

    Google Scholar 

  13. Godefroid, P., Levin, M.Y., Molnar, D.A.: Automated whitebox fuzz testing. In: Proceedings of the Network and Distributed System Security Symposium, NDSS 2008 (2008)

    Google Scholar 

  14. Illera, A.G., Oca, F.: Introducing ponce: one-click symbolic execution. http://research.trust.salesforce.com/Introducing-Ponce-One-click-symbolic-execution/. Accessed Mar 2017

  15. King, J.C.: Symbolic execution and program testing. Commun. ACM 19(7), 385–394 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  16. Kindsight Security Labs: Malware report - Q2 2012 (2012). http://resources.alcatel-lucent.com/?cid=177650. Accessed Mar 2017

  17. RSA Security LLC: Current state of cybercrime (2016). https://www.rsa.com/content/dam/rsa/PDF/2016/05/2016-current-state-of-cybercrime.pdf. Accessed Mar 2017

  18. Moser, A., Kruegel, C., Kirda, E.: Exploring multiple execution paths for malware analysis. In: Proceedings of the 2007 IEEE Symposium on Security and Privacy, SP 2007, pp. 231–245 (2007)

    Google Scholar 

  19. Narayan, J., Shukla, S.K., Clancy, T.C.: A survey of automatic protocol reverse engineering tools. ACM Comput. Surv. 48(3), 40:1–40:26 (2015)

    Article  Google Scholar 

  20. Peng, F., Deng, Z., Zhang, X., Xu, D., Lin, Z., Su, Z.: X-force: force-executing binary programs for security applications. In: Proceedings of the 23rd USENIX Conference on Security Symposium, SEC 2014, pp. 829–844. USENIX Association, Berkeley (2014)

    Google Scholar 

  21. Rodríguez-Gómez, R.A., Maciá-Fernández, G., García-Teodoro, P.: Survey and taxonomy of botnet research through life-cycle. ACM Comput. Surv. 45(4), 45:1–45:33 (2013)

    Article  Google Scholar 

  22. Saudel, F., Salwan, J.: Triton: a dynamic symbolic execution framework. In: Symposium sur la sécurité des technologies de l’information et des communications, SSTIC, Rennes, France, pp. 31–54. SSTI, 3–5 June 2015

    Google Scholar 

  23. Schwartz, E.J., Avgerinos, T., Brumley, D.: All you ever wanted to know about dynamic taint analysis and forward symbolic execution (but might have been afraid to ask). In: Proceedings of the 2010 IEEE Symposium on Security and Privacy, SP 2010, pp. 317–331. IEEE Computer Society, Washington, DC (2010)

    Google Scholar 

  24. Sharif, M.I., Lanzi, A., Giffin, J.T., Lee, W.: Impeding malware analysis using conditional code obfuscation. In: Proceedings of the Network and Distributed System Security Symposium, NDSS 2008 (2008)

    Google Scholar 

  25. Shoshitaishvili, Y., Wang, R., Hauser, C., Kruegel, C., Vigna, G.: Firmalice - automatic detection of authentication bypass vulnerabilities in binary firmware. In: 22nd Annual Network and Distributed System Security Symposium, NDSS 2015 (2015)

    Google Scholar 

  26. Shoshitaishvili, Y., Wang, R., Salls, C., Stephens, N., Polino, M., Dutcher, A., Grosen, J., Feng, S., Hauser, C., Krügel, C., Vigna, G.: SOK: (state of) the art of war: offensive techniques in binary analysis. IEEE Symposium on Security and Privacy, SP 2016, pp. 138–157 (2016)

    Google Scholar 

  27. Norman Solutions: Norman sandbox analyzer. http://download01.norman.no/product_sheets/eng/SandBox_analyzer.pdf. Accessed Mar 2017

  28. Song, D., et al.: BitBlaze: a new approach to computer security via binary analysis. In: Sekar, R., Pujari, A.K. (eds.) ICISS 2008. LNCS, vol. 5352, pp. 1–25. Springer, Heidelberg (2008). doi:10.1007/978-3-540-89862-7_1

    Chapter  Google Scholar 

  29. Stephens, N., Grosen, J., Salls, C., Dutcher, A., Wang, R., Corbetta, J., Shoshitaishvili, Y., Kruegel, C., Vigna, G.: Driller: augmenting fuzzing through selective symbolic execution. In: 23nd Annual Network and Distributed System Security Symposium, NDSS 2016 (2016)

    Google Scholar 

  30. Ugarte-Pedrero, X., Balzarotti, D., Santos, I., Bringas, P.G.: RAMBO: run-time packer analysis with multiple branch observation. In: Caballero, J., Zurutuza, U., Rodríguez, R.J. (eds.) DIMVA 2016. LNCS, vol. 9721, pp. 186–206. Springer, Cham (2016). doi:10.1007/978-3-319-40667-1_10

    Google Scholar 

  31. Villeneuve, N., Sancho, D.: The “Lurid” downloader. Trend Micro Incorporated (2011). http://la.trendmicro.com/media/misc/lurid-downloader-enfal-report-en.pdf. Accessed Mar 2017

  32. Wang, Z., Ming, J., Jia, C., Gao, D.: Linear obfuscation to combat symbolic execution. In: Atluri, V., Diaz, C. (eds.) ESORICS 2011. LNCS, vol. 6879, pp. 210–226. Springer, Heidelberg (2011). doi:10.1007/978-3-642-23822-2_12

    Chapter  Google Scholar 

  33. Willems, C., Holz, T., Freiling, F.: Toward automated dynamic malware analysis using CWSandbox. IEEE Secur. Priv. 5(2), 32–39 (2007)

    Article  Google Scholar 

  34. Yadegari, B., Debray, S.: Symbolic execution of obfuscated code. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015, pp. 732–744. ACM (2015)

    Google Scholar 

Download references

Acknowledgments

We are grateful to the anonymous CSCML 2017 referees for their many useful comments. This work is partially supported by a grant of the Italian Presidency of Ministry Council and by CINI Cybersecurity National Laboratory within the project “FilieraSicura: Securing the Supply Chain of Domestic Critical Infrastructures from Cyber Attacks” (www.filierasicura.it) funded by CISCO Systems Inc. and Leonardo SpA.

Author information

Authors and Affiliations

  1. Software Analysis and Optimization Laboratory, Department of Computer, Control, and Management Engineering, Cyber Intelligence and Information Security Research Center, Sapienza University of Rome, Rome, Italy

    Roberto Baldoni, Emilio Coppa, Daniele Cono D’Elia & Camil Demetrescu

Authors
  1. Roberto Baldoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emilio Coppa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniele Cono D’Elia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Camil Demetrescu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniele Cono D’Elia .

Editor information

Editors and Affiliations

  1. Ben-Gurion University of the Negev , Beer-Sheva, Israel

    Shlomi Dolev

  2. Tata Consultancy Services (India) , Chennai, Tamil Nadu, India

    Sachin Lodha

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Baldoni, R., Coppa, E., D’Elia, D.C., Demetrescu, C. (2017). Assisting Malware Analysis with Symbolic Execution: A Case Study. In: Dolev, S., Lodha, S. (eds) Cyber Security Cryptography and Machine Learning. CSCML 2017. Lecture Notes in Computer Science(), vol 10332. Springer, Cham. https://doi.org/10.1007/978-3-319-60080-2_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-60080-2_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-60079-6

  • Online ISBN: 978-3-319-60080-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation