Skip to main content
Log in

On the Detection of Exploitation of Vulnerabilities That Leads to the Execution of a Malicious Code

  • Published:
Automatic Control and Computer Sciences Aims and scope Submit manuscript

An Erratum to this article was published on 01 December 2022

This article has been updated

Abstract

Software protection from exploitation of possible unknown vulnerabilities can be ensured both by searching for (for example, using symbolic execution) and subsequently eliminating vulnerabilities and by means of detection and/or intrusion prevention systems. In the latter case, this problem is usually solved by forming a profile of normal program execution and an unacceptable deviation from the normal behavior is regarded as an anomaly or attack. The task considered in this study is to protect a given executable file (program) P from exploiting unknown vulnerabilities. The method proposed for this purpose is to construct the normal execution profile of program P, in which not only a set of legal chains of system and library functions of length l is taken into consideration, but so is the distance between adjacent function calls. This distance is calculated as the difference in the call addresses of respective functions. Taking into account distances between function calls allows detecting the execution of a malicious shell code using system and/or library function calls when the distance between at least one of the calls used in the shell code and the preceding call is untypical for program P. This study constructs an algorithm and a system for detecting abnormal code execution and describes several tests conducted when P is the Firefox browser.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.

Similar content being viewed by others

Change history

Notes

  1. As of February 20, 2020.

REFERENCES

  1. Khraisat, A. and Gondal, I., Vamplew, P., and Kamruzzaman, J., Survey of intrusion detection systems: techniques, datasets and challenges, Cybersecurity, 2019, vol. 2, no. 1, p. 20.  https://doi.org/10.1186/s42400-019-0038-7

    Article  Google Scholar 

  2. Forrest, S., Hofmeyr, S., and Somayaji, A., The evolution of system-call monitoring, in Ann. Computer Security Applications Conf. (ACSAC), Anaheim, Calif., 2008, IEEE, 2008, pp. 418–430.  https://doi.org/10.1109/ACSAC.2008.54

  3. Gupta, H., Sharma, H., and Kaur, S., Malware characterization using windows API call sequences, J. Cyber Secur. Mobility, 2018, vol. 7, no. 4, pp. 363–378.  https://doi.org/10.13052/2245-1439.741

    Article  Google Scholar 

  4. Veeramani, R. and Rai, N., Windows API based malware detection and framework analysis, Int. J. Sci. Eng. Res., 2012, vol. 3, no. 3, pp. 1–6.

    Google Scholar 

  5. Singh, A., Arora, R., and Pareek, H., Malware analysis using multiple API sequence mining control flow graph, 2017. arXiv:1707.02691 [cs.CR]

  6. Bernardi, M.L., Cimitile, M., Distante, D., Martinelli, F., and Mercaldo, F., Dynamic malware detection and phylogeny analysis using process mining, Int. J. Inf. Secur., 2019, vol. 18, no. 3, pp. 257–284.  https://doi.org/10.1007/s10207-018-0415-3

    Article  Google Scholar 

  7. Viljanen, L., A survey of application level intrusion detection, Technical Report, Series of Publications C, Report C-2004-61, Helsinki, 2004.

  8. Creech, G., Developing a high-accuracy cross platform Host-Based Intrusion Detection System capable of reliably detecting zero-day attacks, PhD Thesis, Canberra: Univ. of New South Wales, 2014.

  9. Hu, H., Shinde, S., Adrian, S., Chua, Z.L., Saxena, P., and Liang, Z., Data-oriented programming: On the expressiveness of non-control data attacks, in IEEE 2016 Symp. on Security and Privacy (SP), San Jose, Calif., 2016, IEEE, 2016, pp. 969–986.  https://doi.org/10.1109/SP.2016.62

  10. Ispoglou, K.K., AlBassam, B., Jaeger, T., and Payer, M., Block oriented programming: Automating data-only attacks, Proc. 2018 ACM SIGSAC Conf. on Computer and Communications Security, Toronto, 2018, New York: Association for Computing Machinery, 2018, pp. 1868–1882.  https://doi.org/10.1145/3243734.3243739

  11. Kosolapov, Y.V., On detecting code reuse attacks, Autom. Control Comput. Sci., 2020, vol. 54, pp. 573–583. https://doi.org/10.3103/S0146411620070111

  12. Wagner, D. and Soto, P., Mimicry attacks on host-based intrusion detection systems, in Proc. 9th ACM Conf. on Computer and Communications Security, Washington, 2002, Atluri, V., Ed., New York: Association for Computing Machinery, 2002, pp. 255–264.  https://doi.org/10.1145/586110.586145

  13. Snow, K.F., Monrose, F., Davi, L., Dmitrienko, A., Liebchen, C., and Sadeghi, A.-R., Just-in-time code reuse: On the effectiveness of fine-grained address space layout randomization, in IEEE Symp. on Security and Privacy, Berkeley, Calif., 2013, IEEE, 2013, pp. 574–588.  https://doi.org/10.1109/SP.2013.45

  14. Stalmans, E. and El-Sherei, S., Macro-less code Exec in MSWord. https://sensepost.com/blog/2017/macro-less-code-exec-in-msword/. Cited December 12, 2019.

  15. Borisov, P.D. and Kosolapov, Yu.V., On the automatic analysis of the practical resistance of obfuscating transformations, Autom. Control Comput. Sci., 2020, vol. 54, pp. 619–629.  https://doi.org/10.3103/S0146411620070044

  16. API Monito. http://www.rohitab.com/apimonitor. Cited November 28, 2019.

  17. ListDLLs. https://docs.microso.com/en-us/sysinternals/downloads/listdlls. Cited November 28, 2019.

  18. Vervier, M., Orru, M., Wever, B.J., and Sesterhenn, E., Browser security whitepaper. https://browser-security.x41-dsec.de/X41-Browser-Security-White-Paper.pdf. Cited December 5, 2019.

  19. Gawlik, R. and Holz, T., SoK: Make JIT-spray great again, in 12th USENIX Workshop on Offensive Technologies (WOOT 18), Baltimore, 2018, Baltimore: USENIX Association, 2018.

  20. Offensive security, Exploitdb/exploits/windows/remote/42484.html. Cited December 5, 2019. https://github.com/ o.ensive-security/exploitdb/blob/master/exploits/windows/remote/42484.html.

  21. 0vercl0k, CVE-2019-9810. https://github.com/0vercl0k/CVE-2019-9810. Cited December 5, 2019.

  22. Exploit database. https://www.exploit-db.com/. Cited December 5, 2019.

  23. CVE-2017-5375_ASM.JS_JIT-Spray. https://github.com/rh0dev/expdev/tree/master. Cited December 30, 2019.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Y. V. Kosolapov.

Ethics declarations

The author declares that he has no conflicts of interest.

Additional information

Translated by S. Kuznetsov

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kosolapov, Y.V. On the Detection of Exploitation of Vulnerabilities That Leads to the Execution of a Malicious Code. Aut. Control Comp. Sci. 55, 827–837 (2021). https://doi.org/10.3103/S0146411621070233

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3103/S0146411621070233

Keywords:

Navigation