Skip to main content
SpringerLink
Log in

On the Automatic Analysis of the Practical Resistance of Obfuscating Transformations

  • Published:
Automatic Control and Computer Sciences Aims and scope Submit manuscript

Abstract

A method is developed for assessing the practical persistence of obfuscating transformations of programs. The method is based on the calculation of the similarity index for the original, obfuscated, and deobfuscated programs. Candidates are proposed for the similarity indices, which are based on such program characteristics as the control flow graph, symbolic execution time, and degree of coverage for symbolic execution. The control flow graph is considered the basis for building other candidates for program similarity indicators. On its basis, a new candidate is proposed for the similarity index, which, when calculated, finds the Hamming distances between the adjacency matrices of the control flow graphs of the compared programs. A scheme for estimating (analyzing) the persistence of obfuscating transformations is constructed. According to this scheme, the characteristics of the original, obfuscated, and deobfuscated programs are calculated and compared in accordance with the chosen comparison model. In particular, the developed scheme is suitable for comparing programs based on similarity indices. This paper develops and implements one of the key units of the constructed scheme, which is a block for obtaining the program characteristics compiled for the x86/x86_64 architecture. The developed unit allows finding the control flow graph, the time for symbolic execution, and the degree of coverage for symbolic execution. Selected results of operation of the constructed block are provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1.
Fig. 2.

Similar content being viewed by others

REFERENCES

  1. Siegmund, J., Program comprehension: Past, present, and future, IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2016, vol. 5, pp. 13–20.

  2. Avidan, E. and Feitelson, D.G., From obfuscation to comprehension, Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension, 2015, pp. 178–181.

  3. Pozdeev, A.G., Krivopalov, V.N., Romashkin, E.V., and Radchenko, E.D., Mathematical and software tools for program obfuscation, Prikl. Diskretn. Mat., 2009, vol. 1, pp. 52–53.

    Google Scholar 

  4. Chernov, A.V., Analyzing confusing program transformations, 2002. http://www.citforum.ru/security/articles/analysis/.

  5. Kuzurin, N., Shokurov, A., Varnovsky, N., and Zakharov, V., On the concept of software obfuscation in computer security, International Conference on Information Security, Berlin–Heidelberg: Springer, 2007, pp. 281–298.

  6. Diffie, W. and Hellman, M., New directions in cryptography, IEEE Trans. Inf. Theory, 1976, vol. 22, no. 6, pp. 644–654.

    Article  MathSciNet  Google Scholar 

  7. Collberg, C.S. and Thomborson, C., Watermarking, tamper-proofing, and obfuscation tools for software protection, IEEE Trans. Software Eng., 2002, vol. 28, no. 8, pp. 735–746.

    Article  Google Scholar 

  8. Lee, B., Kim, Y., and Kim, J., binOb+: A framework for potent and stealthy binary obfuscation, Proceedings of the 5th ACM Symposium on Information, Computer and Communications Security, ASIACCS 2010, 2010, pp. 271–281.

  9. Borello, J.M. and Me, L., Code obfuscation techniques for metamorphic viruses, J. Comput. Virol., 2008, vol. 4, no. 3, pp. 211–220.

    Article  Google Scholar 

  10. Moser, A., Kruegel, C., and Kirda, E., Limits of static analysis for malware detection, Proceedings of Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), 2007, pp. 421–430.

  11. Baiardi, F. and Sgandurra, D., An obfuscation-based approach against injection attacks, Proceedings of the Sixth International Conference on Availability, Reliability and Security (ARES), 2011, pp. 51–58.

  12. Nurmukhametov, A.R., Use of diversifying and obfuscating transformations to change the signature of program code, Tr. Inst. Sist. Program. Ross. Akad. Nauk, 2016, vol. 28, no. 5, pp. 93–104.

    Google Scholar 

  13. Kosolapov, Y.V., About detection of code reuse attacks, Model. Anal. Inf. Sist., 2019, vol. 26, no. 2, pp. 213–228.

    Article  Google Scholar 

  14. Collberg, C., Thomborson, C., and Low, D., A Taxonomy of Obfuscating Transformations, Technical Report 148, The University of Auckland, 1997.

  15. Walenstein, A., El-Ramly, M., Cordy, J.R., Evans, W.S., Mahdavi, K., Pizka, M., Ramalingam, G., and von Gudenberg, J.W., Similarity in programs, Duplication, Redundancy, and Similarity in Software, 2007, pp. 1–8.

    Google Scholar 

  16. Chipounov, V., Kuznetsov, V., and Candea, G., The S2E platform: Design, implementation, and applications, ACM Trans. Comput. Syst., 2012, vol. 30, no. 1, pp. 1–49.

    Article  Google Scholar 

  17. Saudel, F. and Salwan, J., Triton: A dynamic symbolic execution framework, Symposium Sur La Security Des Technologies de L’information et Des Communications, SSTIC, 2015, pp. 31–54.

  18. Wang, Z., Ming, J., Jia, C., and Gao, D., Linear obfuscation to combat symbolic execution, Proceedings of Computer Security – ESORICS 2011, 2011, pp. 210–226.

  19. Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D., and Yin, H., Automatically identifying trigger-based behavior in malware, Botnet Detect., Adv. Inf. Secur., 2008, vol. 36, pp. 65–88.

    Article  Google Scholar 

  20. King, J.C., Symbolic execution and program testing, Commun. ACM, 1976, vol. 19, no. 7, pp. 385–394.

    Article  MathSciNet  Google Scholar 

  21. Cadar, C., Dunbar, D., and Engler, D.R., KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs, 8th USENIX Symposium on Operating Systems Design and Implementation, 2008, pp. 209–224.

  22. Shoshitaishvili, Y., et al., SoK: (State of) the art of war: Offensive techniques in binary analysis, IEEE Symposium on Security and Privacy, 2016, pp. 138–157.

  23. Sharif, M.I., Lanzi, A., Giffin, J.T., and Lee, W., Impeding malware analysis using conditional code obfuscation, Proceedings of NDSS, 2008, pp. 1–13.

  24. Udupa, S.K., Debray, S.K., and Madou, M., Deobfuscation: Reverse engineering obfuscated code, Proceedings of the 12th Working Conference on Reverse Engineering (WCRE'05), 2005, pp. 44–53.

  25. Nagarajan, V., Gupta, R., Zhang, X., Madou, M., and De Sutter, B., Matching control flow of program versions, IEEE International Conference on Software Maintenance, 2007, pp. 84–93.

  26. Bonfante, G., Kaczmarek, M., and Marion, J.Y., Control flow graphs as malware signatures, International Workshop on the Theory of Computer Viruses, 2007, pp. 1–6.

  27. Park, Y., Reeves, D., Mulukutla, V., and Sundaravel, B., Fast malware classification by automated behavioral graph matching, Proceedings of the Sixth Annual Workshop on Cyber Security and Information Intelligence Research, 2010, pp. 1–4.

  28. Kinable, J. and Kostakis, O., Malware classification based on call graph clustering, J. Comput. Virol., 2011, vol. 7, no. 4, pp. 233–245.

    Article  Google Scholar 

  29. Lim, H.I., Comparing control flow graphs of binary programs through match propagation, IEEE 38th Annual Computer Software and Applications Conference, 2014, pp. 598–599.

  30. Dullien, T. and Rolles, R., Graph-Based Comparison of Executable Objects, 2005, pp. 1–8.

  31. Chan, P.P.F. and Collberg, C., A method to evaluate CFG comparison algorithms, 14th International Conference on Quality Software, 2014, pp. 95–104.

  32. Axenovich, M., Kezdy, A., and Martin, R., On the editing distance of graphs, J. Graph Theory, 2008, vol. 58, no. 2, pp. 123–138.

    Article  MathSciNet  Google Scholar 

  33. Borisov, P.D. and Kosolapov, Y.V., On the choice of characteristics for assessing the stability of obfuscating transformations, Sovremennye informatsionnye tekhnologii: Tendentsii i perspektivy razvitiya. Trudy XXV nauchnoi konferentsii SITO-2019 (Modern Information Technologies: Trends and Development Prospects. Proc. 25th Sci. Conf. SITO-2019), 2019, pp. 42–44.

  34. Lehman, M.M. and Belady, L.A., Program Evolution. Processes of Software Change, Academic Press, 1985.

    Google Scholar 

  35. Schnappinger, M., Osman, M.H., Pretschner, A., Pizka, M., and Fietzke, A., Software quality assessment in practice: A hypothesis-driven framework, Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2018, pp. 1–6.

  36. Borisov, P.D. and Kosolapov, Y.V., A model of the experimental analysis of the robustness of obfuscation algorithms, Sovremennye informatsionnye tekhnologii: Tendentsii i perspektivy razvitiya. Trudy XXV nauchnoi konferentsii SITO-2019 (Modern Information Technologies: Trends and Development Prospects. Proc. 25th Sci. Conf. SITO-2019), 2019, pp. 37–39.

  37. IDA Pro. https://www.hex-rays.com/products/ida/.

  38. The LLVM Compiler Infrastructure. https://llvm.org/.

  39. McSema. https://github.com/trailofbits/mcsema.

  40. Junod, P., Rinaldini, J., Wehrli, J., and Michieliny, J., Obfuscator-LLVM – software protection for the masses, 2015 IEEE/ACM 1st International Workshop on Software Protection (SPRO), 2015, pp. 3–9.

Download references

Author information

Authors and Affiliations

  1. Southern Federal University, 344090, Rostov-on-Don, Russia

    P. D. Borisov & Yu. V. Kosolapov

Authors
  1. P. D. Borisov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu. V. Kosolapov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to P. D. Borisov or Yu. V. Kosolapov.

Ethics declarations

CONFLICT OF INTEREST

The authors declare that they have no conflicts of interest.

ADDITIONAL INFORMATION

Petr D. Borisov, orcid.org/0000-0002-8919-8310, graduate student.

Yury V. Kosolapov, orcid.org/0000-0002-1491-524X, PhD.

Additional information

Translated by E. Oborin

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Borisov, P.D., Kosolapov, Y.V. On the Automatic Analysis of the Practical Resistance of Obfuscating Transformations. Aut. Control Comp. Sci. 54, 619–629 (2020). https://doi.org/10.3103/S0146411620070044

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3103/S0146411620070044

Keywords:

Search

Navigation