Skip to main content
Springer Nature Link
Log in

An instrumentation based algorithm for stack overflow detection

  • Original Paper
  • Published:
Journal of Computer Virology and Hacking Techniques Aims and scope Submit manuscript

Abstract

Despite all the efforts made by the scientific community in terms of computer security, buffer overflow vulnerabilities continue being the biggest security flaw in applications, since they compromise the security of the system through memory corruption. To tackle this problem, there are different techniques based on the binary analysis of the application in question. With this objective in mind, the present paper proposes an algorithm based on the dynamic instrumentation of binaries, that is, dynamic local variables belonging to the functions of the program are detected, and a check is performed to see whether there is an overflow of memory between them. The results obtained show how the proposed algorithm is able to detect buffer overflow errors in the stack frames of a function.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Common vulnerabilities and exposures. http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1328

  2. Common vulnerabilities and exposures. https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-2883

  3. Dynamorio. http://www.dynamorio.org/

  4. Ida dissasembler. https://www.hex-rays.com/products/ida/

  5. Metasploit | penetration testing software, pen testing security. https://www.metasploit.com/

  6. Ravnas, O.: Frida. https://www.frida.re/

  7. Offensive security exploit database archive. https://www.exploit-db.com

  8. Pin—a dynamic binary instrumentation tool. https://software.intel.com/enus/articles/pin-a-dynamic-binary-instrumentation-tool

  9. Pin tutorial. https://software.intel.com/sites/default/files/article/256675/pintutorial-academiasinica-1.ppt

  10. Valgrind Developers. http://valgrind.org/

  11. Álvarez, S.: Radare2. http://radare.org/r/index.html

  12. Bruening, D., Zhao, Q.: Practical memory checking with Dr. memory. In: Proceedings of the 9th Annual IEEE/ACM International Symposium on Code Generation and Optimization, CGO ’11, pp. 213–223. IEEE Computer Society, Washington, DC, USA (2011). http://dl.acm.org/citation.cfm?id=2190025.2190067

  13. Carballal, C.: Estudio, análisis y modelado de memorias caché compartidas bajo administración dinámica. Master’s thesis, Universidad de Buenos Aires (2011). URL http://materias.fi.uba.ar/7500/carballalabarzua-tesisingenieriainformatica.pdf

  14. Cugliari, A., Graziano, M.: Smashing the stack. Master’s thesis, Politecnico di Torino (2010). http://www.mgraziano.info/docs/stsi2010.pdf

  15. D’ Alessio, S., Mariani, S.: Pindemonium: a DBI-based generic unpacker for windows executables (2016). http://hdl.handle.net/10589/120861

  16. Dang, T.H., Maniatis, P., Wagner, D.: The performance cost of shadow stacks and stack canaries. In: Proceedings of the 10th ACM Symposium on Information, Computer and Communications Security, ASIA CCS ’15, pp. 555–566. ACM, New York, NY, USA (2015). https://doi.org/10.1145/2714576.2714635

  17. Davi, L., Sadeghi, A.R., Winandy, M.: Ropdefender: A detection tool to defend against return-oriented programming attacks. In: Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security, ASIACCS ’11, pp. 40–51. ACM, New York, NY, USA (2011). https://doi.org/10.1145/1966913.1966920

  18. Fu, D., Shi, F.: Buffer overflow exploit and defensive techniques. In: 2012 Fourth International Conference on Multimedia Information Networking and Security, pp. 87–90 (2012). https://doi.org/10.1109/MINES.2012.81

  19. Ganz, J., Peisert, S.: ASLR: how robust is the randomness? In: 2017 IEEE Cybersecurity Development (SecDev), pp. 34–41 (2017). https://doi.org/10.1109/SecDev.2017.19

  20. Gao, Y., Lu, Z., Luo, Y.: Survey on malware anti-analysis. In: Fifth International Conference on Intelligent Control and Information Processing, pp. 270–275 (2014)

  21. Gentsch, C.: Evaluation of open source static analysis security testing (SAST) tools for c. Technical report, DLR DW (2020). URL https://elib.dlr.de/133945/

  22. Henderson, A., Prakash, A., Yan, L.K., Hu, X., Wang, X., Zhou, R., Yin, H.: Make it work, make it right, make it fast: building a platform-neutral whole-system dynamic binary analysis platform. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 248–258. Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2610384.2610407

  23. Howard, M.: Security development lifecycle (SDL) banned function calls (2011). https://msdn.microsoft.com/en-us/library/bb288454.aspx

  24. Jia, X., Zhang, C., Su, P., Yang, Y., Huang, H., Feng, D.: Towards efficient heap overflow discovery. In: 26th USENIX Security Symposium (USENIX Security 17), pp. 989–1006. USENIX Association, Vancouver, BC (2017). https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/jia

  25. Koziol, J., Heasman, J., Lindner, F., Richarte, G., Aitel, D., Anley, C., Eren, S., Mehta, N., Hassell, R.: The Shellcoder’s Handbook: Discovering and Exploiting Security Holes, 2 revised edn. Wiley, Indianapolis, IN (2007)

    Google Scholar 

  26. Ma, J., Zhang, P., Dong, G., Shao, S., Zhang, J.: Twalker: an efficient taint analysis tool. In: 2014 10th International Conference on Information Assurance and Security, pp. 18–22 (2014). https://doi.org/10.1109/ISIAS.2014.7064628

  27. Nethercote, N.: Dynamic binary analysis and instrumentation. Technical report UCAM-CL-TR-606, University of Cambridge, Computer Laboratory (2004). https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-606.pdf

  28. Petsios, T., Kemerlis, V.P., Polychronakis, M., Keromytis, A.D.: Dynaguard: armoring canary-based protections against brute-force attacks. In: Proceedings of the 31st Annual Computer Security Applications Conference, ACSAC 2015, pp. 351–360. ACM, New York, NY, USA (2015). https://doi.org/10.1145/2818000.2818031

  29. Prandini, M., Ramilli, M.: Return-oriented programming. IEEE Secur. Priv. 10(6), 84–87 (2012). https://doi.org/10.1109/MSP.2012.152

    Article  Google Scholar 

  30. Radu, D., Dang, B.: Shellcode analysis using dynamic binary instrumentation. Technical report (2011)

  31. Rodriguez, R.J., Gaston, I.R., Alonso, J.: Towards the detection of isolation-aware malware. IEEE Latin Am. Trans. 14(2), 1024–1036 (2016). https://doi.org/10.1109/TLA.2016.7437254

    Article  Google Scholar 

  32. Saito, T., Watanabe, R., Kondo, S., Sugawara, S., Yokoyama, M.: A survey of prevention/mitigation against memory corruption attacks. In: 2016 19th International Conference on Network-Based Information Systems (NBiS), pp. 500–505 (2016)

  33. Serebryany, K., Bruening, D., Potapenko, A., Vyukov, D.: Addresssanitizer: a fast address sanity checker. In: Proceedings of the 2012 USENIX Conference on Annual Technical Conference, USENIX ATC’12, pp. 28–28. USENIX Association, Berkeley, CA, USA (2012). http://dl.acm.org/citation.cfm?id=2342821.2342849

  34. Seward, J., Nethercote, N.: Using valgrind to detect undefined value errors with bit-precision. In: Proceedings of the Annual Conference on USENIX Annual Technical Conference, ATEC ’05, p. 2. USENIX Association, USA (2005)

  35. Shudrak, M.: WinHeap explorer: efficient and transparent heap-based bug detection in machine code. In: 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 94–101 (2017). https://doi.org/10.1109/QRS.2017.20

  36. van der Veen, V., dutt Sharma, N., Cavallaro, L., Bos, H.: Memory errors: the past, the present, and the future. In: Balzarotti, D., Stolfo, S.J., Cova, M. (eds.) Research in Attacks, Intrusions, and Defenses, pp. 86–106. Springer, Berlin (2012)

    Chapter  Google Scholar 

  37. Wu, B., Li, M., Zhang, B., Zhang, Q., Tang, C.: Directed symbolic execution for binary vulnerability mining. In: 2014 IEEE Workshop on Electronics, Computer and Applications, pp. 614–617 (2014). https://doi.org/10.1109/IWECA.2014.6845694

  38. Ye, T., Zhang, L., Wang, L., Li, X.: An empirical study on detecting and fixing buffer overflow bugs. In: 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 91–101 (2016)

  39. Zou, Q., Huang, W., An, J., Fan, W.: Identify stack overflow exploits with dynamic binary instrumentation. In: 2015 International Conference on Industrial Informatics—Computing Technology, Intelligent Technology, Industrial Information Integration, pp. 263–267 (2015). https://doi.org/10.1109/ICIICII.2015.147

Download references

Author information

Authors and Affiliations

  1. Institute of Research of Informatics (i3a) in Albacete, Universidad de Castilla-La Mancha, Albacete, Spain

    J. Carrillo-Mondéjar, J. M. Castelo-Gómez, J. Roldán-Gómez & J. L. Martínez

Authors
  1. J. Carrillo-Mondéjar
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. J. M. Castelo-Gómez
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. J. Roldán-Gómez
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. J. L. Martínez
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to J. Carrillo-Mondéjar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work has been supported by the MINECO and European Commission (FEDER funds) under Project RTI2018-098156-B-C52, by the JCCM under the Project SB-PLY/17/180501/00035, by the Spanish Education, Culture and Sports Ministry under Grants FPU 17/03105 and FPU 17/02007, by the University of Castilla-La Mancha under the contract 2018-PREDUCLM-7476.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Carrillo-Mondéjar, J., Castelo-Gómez, J.M., Roldán-Gómez, J. et al. An instrumentation based algorithm for stack overflow detection. J Comput Virol Hack Tech 16, 245–256 (2020). https://doi.org/10.1007/s11416-020-00359-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-020-00359-7

Keywords