Abstract
Software protection from exploitation of possible unknown vulnerabilities can be ensured both by searching for (for example, using symbolic execution) and subsequently eliminating vulnerabilities and by means of detection and/or intrusion prevention systems. In the latter case, this problem is usually solved by forming a profile of normal program execution and an unacceptable deviation from the normal behavior is regarded as an anomaly or attack. The task considered in this study is to protect a given executable file (program) P from exploiting unknown vulnerabilities. The method proposed for this purpose is to construct the normal execution profile of program P, in which not only a set of legal chains of system and library functions of length l is taken into consideration, but so is the distance between adjacent function calls. This distance is calculated as the difference in the call addresses of respective functions. Taking into account distances between function calls allows detecting the execution of a malicious shell code using system and/or library function calls when the distance between at least one of the calls used in the shell code and the preceding call is untypical for program P. This study constructs an algorithm and a system for detecting abnormal code execution and describes several tests conducted when P is the Firefox browser.
Similar content being viewed by others
Change history
19 February 2023
An Erratum to this paper has been published: https://doi.org/10.3103/S0146411622070094
Notes
As of February 20, 2020.
REFERENCES
Khraisat, A. and Gondal, I., Vamplew, P., and Kamruzzaman, J., Survey of intrusion detection systems: techniques, datasets and challenges, Cybersecurity, 2019, vol. 2, no. 1, p. 20. https://doi.org/10.1186/s42400-019-0038-7
Forrest, S., Hofmeyr, S., and Somayaji, A., The evolution of system-call monitoring, in Ann. Computer Security Applications Conf. (ACSAC), Anaheim, Calif., 2008, IEEE, 2008, pp. 418–430. https://doi.org/10.1109/ACSAC.2008.54
Gupta, H., Sharma, H., and Kaur, S., Malware characterization using windows API call sequences, J. Cyber Secur. Mobility, 2018, vol. 7, no. 4, pp. 363–378. https://doi.org/10.13052/2245-1439.741
Veeramani, R. and Rai, N., Windows API based malware detection and framework analysis, Int. J. Sci. Eng. Res., 2012, vol. 3, no. 3, pp. 1–6.
Singh, A., Arora, R., and Pareek, H., Malware analysis using multiple API sequence mining control flow graph, 2017. arXiv:1707.02691 [cs.CR]
Bernardi, M.L., Cimitile, M., Distante, D., Martinelli, F., and Mercaldo, F., Dynamic malware detection and phylogeny analysis using process mining, Int. J. Inf. Secur., 2019, vol. 18, no. 3, pp. 257–284. https://doi.org/10.1007/s10207-018-0415-3
Viljanen, L., A survey of application level intrusion detection, Technical Report, Series of Publications C, Report C-2004-61, Helsinki, 2004.
Creech, G., Developing a high-accuracy cross platform Host-Based Intrusion Detection System capable of reliably detecting zero-day attacks, PhD Thesis, Canberra: Univ. of New South Wales, 2014.
Hu, H., Shinde, S., Adrian, S., Chua, Z.L., Saxena, P., and Liang, Z., Data-oriented programming: On the expressiveness of non-control data attacks, in IEEE 2016 Symp. on Security and Privacy (SP), San Jose, Calif., 2016, IEEE, 2016, pp. 969–986. https://doi.org/10.1109/SP.2016.62
Ispoglou, K.K., AlBassam, B., Jaeger, T., and Payer, M., Block oriented programming: Automating data-only attacks, Proc. 2018 ACM SIGSAC Conf. on Computer and Communications Security, Toronto, 2018, New York: Association for Computing Machinery, 2018, pp. 1868–1882. https://doi.org/10.1145/3243734.3243739
Kosolapov, Y.V., On detecting code reuse attacks, Autom. Control Comput. Sci., 2020, vol. 54, pp. 573–583. https://doi.org/10.3103/S0146411620070111
Wagner, D. and Soto, P., Mimicry attacks on host-based intrusion detection systems, in Proc. 9th ACM Conf. on Computer and Communications Security, Washington, 2002, Atluri, V., Ed., New York: Association for Computing Machinery, 2002, pp. 255–264. https://doi.org/10.1145/586110.586145
Snow, K.F., Monrose, F., Davi, L., Dmitrienko, A., Liebchen, C., and Sadeghi, A.-R., Just-in-time code reuse: On the effectiveness of fine-grained address space layout randomization, in IEEE Symp. on Security and Privacy, Berkeley, Calif., 2013, IEEE, 2013, pp. 574–588. https://doi.org/10.1109/SP.2013.45
Stalmans, E. and El-Sherei, S., Macro-less code Exec in MSWord. https://sensepost.com/blog/2017/macro-less-code-exec-in-msword/. Cited December 12, 2019.
Borisov, P.D. and Kosolapov, Yu.V., On the automatic analysis of the practical resistance of obfuscating transformations, Autom. Control Comput. Sci., 2020, vol. 54, pp. 619–629. https://doi.org/10.3103/S0146411620070044
API Monito. http://www.rohitab.com/apimonitor. Cited November 28, 2019.
ListDLLs. https://docs.microso.com/en-us/sysinternals/downloads/listdlls. Cited November 28, 2019.
Vervier, M., Orru, M., Wever, B.J., and Sesterhenn, E., Browser security whitepaper. https://browser-security.x41-dsec.de/X41-Browser-Security-White-Paper.pdf. Cited December 5, 2019.
Gawlik, R. and Holz, T., SoK: Make JIT-spray great again, in 12th USENIX Workshop on Offensive Technologies (WOOT 18), Baltimore, 2018, Baltimore: USENIX Association, 2018.
Offensive security, Exploitdb/exploits/windows/remote/42484.html. Cited December 5, 2019. https://github.com/ o.ensive-security/exploitdb/blob/master/exploits/windows/remote/42484.html.
0vercl0k, CVE-2019-9810. https://github.com/0vercl0k/CVE-2019-9810. Cited December 5, 2019.
Exploit database. https://www.exploit-db.com/. Cited December 5, 2019.
CVE-2017-5375_ASM.JS_JIT-Spray. https://github.com/rh0dev/expdev/tree/master. Cited December 30, 2019.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The author declares that he has no conflicts of interest.
Additional information
Translated by S. Kuznetsov
About this article
Cite this article
Kosolapov, Y.V. On the Detection of Exploitation of Vulnerabilities That Leads to the Execution of a Malicious Code. Aut. Control Comp. Sci. 55, 827–837 (2021). https://doi.org/10.3103/S0146411621070233
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3103/S0146411621070233