Skip to main content
SpringerLink
Log in

Software System Exploration Using Library Call Analysis

  • Conference paper
  • First Online:
Model-driven Simulation and Training Environments for Cybersecurity (MSTEC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 12512))

  • 481 Accesses

Abstract

The ability to analyze software systems without access to the source code, offers many advantages including the detection of vulnerabilities so that they may be fixed before an adversary can exploit them in a zero day attack. This type of analysis also has an important role in education as it allows students to use their imagination and creativity in the exploration process. In this paper, we use two techniques for black-box testing based on our previous work, where we demonstrated how library calls may be intercepted using wrappers as well as using the kernel to separate the memory of a process into regions, based on the (statically/dynamically) linked libraries that a program uses. By monitoring function calls to libraries or the main executable, we can determine if a high-level execution signature (which depends not only on the occurrence, but also the sequence and number of calls) fits a pattern of a possible attack against a system under test. We can, then, (a) determine whether a call should go ahead, (b) determine whether the arguments are acceptable and (c) ensure that we will be informed when there is suspicion of foul play. We then demonstrate how these techniques may be used in student training sessions to explore the structure of software systems and determine how such systems respond to specific input sequences designed to trigger bugs or demonstrate unexpected behavior.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-flow integrity. In: Proceedings of the 12th ACM Conference on Computer and Communications Security, CCS 2005, pp. 340–353. ACM, New York (2005). https://doi.org/10.1145/1102120.1102165. http://doi.acm.org/10.1145/1102120.1102165

  2. Andersen, S., Abella, V.: Data Execution Prevention (2004). https://technet.microsoft.com/en-us/library/bb457155.aspx

  3. Bittau, A., Belay, A., Mashtizadeh, A., Mazières, D., Boneh, D.: Hacking blind. In: 2014 IEEE Symposium on Security and Privacy, pp. 227–242, May 2014. https://doi.org/10.1109/SP.2014.22

  4. Bletsch, T., Jiang, X., Freeh, V.W., Liang, Z.: Jump-oriented programming: a new class of code-reuse attack. In: Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security, ASIACCS 2011, pp. 30–40. ACM, New York (2011). https://doi.org/10.1145/1966913.1966919. http://doi.acm.org/10.1145/1966913.1966919

  5. Checkoway, S., Davi, L., Dmitrienko, A., Sadeghi, A.R., Shacham, H., Winandy, M.: Return-oriented programming without returns. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 559–572. ACM, New York (2010). https://doi.org/10.1145/1866307.1866370. http://doi.acm.org/10.1145/1866307.1866370

  6. Chen, L.H., Hsu, F.H., Hwang, Y., Su, M.C., Ku, W.S., Chang, C.H.: Armory: an automatic security testing tool for buffer overflow defect detection. Comput. Electr. Eng. 39(7), 2233–2242 (2013). https://doi.org/10.1016/j.compeleceng.2012.07.005

    Article  Google Scholar 

  7. Cheng, Y., Zhou, Z., Miao, Y., Ding, X., Deng, H.R.: ROPecker: a generic and practical approach for defending against ROP attacks. In: Symposium on Network and Distributed System Security (NDSS) (2014)

    Google Scholar 

  8. Common Vulnerabilities and Exposures: CVE-2013-2028, February 2013. https://www.cvedetails.com/cve/CVE-2013-2028/

  9. Cowan, C., et al.: StackGuard: automatic adaptive detection and prevention of buffer-overflow attacks. In: Proceedings of the 7th Conference on USENIX Security Symposium - Volume 7, SSYM 1998, p. 5. USENIX Association, Berkeley (1998). http://dl.acm.org/citation.cfm?id=1267549.1267554

  10. Crane, S., Larsen, P., Brunthaler, S., Franz, M.: Booby trapping software. In: Proceedings of the 2013 Workshop on New Security Paradigms Workshop, NSPW 2013, pp. 95–106. ACM, New York (2013). https://doi.org/10.1145/2535813.2535824

  11. CVE-2016-7054: Chacha20/poly1305 heap-buffer-overflow (2016). https://www.openssl.org/news/secadv/20161110.txt

  12. CVE\_2016\_7054: Chacha20/poly1305 heap-buffer-overflow (2016). https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-7054

  13. Designer, S.: Getting around non-executable stack (and fix) (1997). http://seclists.org/bugtraq/1997/Aug/63

  14. Fratric, I.: ROPGuard: runtime prevention of return-oriented programming attacks (2012). http://www.ieee.hr/_download/repository/Ivan_Fratric.pdf

  15. Hamad, M., Hammadeh, Z.A.H., Saidi, S., Prevelakis, V., Ernst, R.: Prediction of abnormal temporal behavior in real-time systems. In: The 33rd ACM/SIGAPP Symposium On Applied Computing (SAC 2018) (2018). https://dl.acm.org/citation.cfm?id=3167172

  16. Hiroaki, E.: ProPolice: GCC extension for protecting applications from stack-smashing attacks, January 2003

    Google Scholar 

  17. Lu, S., Seo, M., Lysecky, R.: Timing-based anomaly detection in embedded systems. In: 20th Asia and South Pacific Design Automation Conference, ASP-DAC 2015, January 2015. https://doi.org/10.1109/ASPDAC.2015.7059110

  18. Pappas, V., Polychronakis, M., Keromytis, A.D.: Transparent ROP exploit mitigation using indirect branch tracing. In: Proceedings of the 22Nd USENIX Conference on Security, SEC 2013, pp. 447–462. USENIX Association, Berkeley (2013). http://dl.acm.org/citation.cfm?id=2534766.2534805

  19. PaX, T.: Address Space Layout Randomization (2001). https://pax.grsecurity.net/docs/aslr.txt

  20. Pincus, J., Baker, B.: Beyond stack smashing: recent advances in exploiting buffer overruns. IEEE Secur. Privacy 2(4), 20–27 (2004). https://doi.org/10.1109/MSP.2004.36

    Article  Google Scholar 

  21. Prevelakis, V.: Use of HTTP protocol by the TU-BS Sophos Repository. Technical report, TU Braunschweig (2017)

    Google Scholar 

  22. Roemer, R., Buchanan, E., Shacham, H., Savage, S.: Return-oriented programming: systems, languages, and applications. ACM Trans. Inf. Syst. Secur. 15(1), 2:1–2:34 (2012). https://doi.org/10.1145/2133375.2133377. http://doi.acm.org/10.1145/2133375.2133377

  23. Shacham, H.: The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86). In: Proceedings of the 14th ACM Conference on Computer and Communications Security, CCS 2007, pp. 552–561. ACM, New York (2007). https://doi.org/10.1145/1315245.1315313. http://doi.acm.org/10.1145/1315245.1315313

  24. Shacham, H., Page, M., Pfaff, B., Goh, E.J., Modadugu, N., Boneh, D.: On the effectiveness of address-space randomization. In: Proceedings of the 11th ACM Conference on Computer and Communications Security, CCS 2004, pp. 298–307. ACM, New York (2004). https://doi.org/10.1145/1030083.1030124. http://doi.acm.org/10.1145/1030083.1030124

  25. Snow, K.Z., Monrose, F., Davi, L., Dmitrienko, A., Liebchen, C., Sadeghi, A.R.: Just-in-time code reuse: on the effectiveness of fine-grained address space layout randomization. In: Proceedings of the 2013 IEEE Symposium on Security and Privacy, SP 2013, pp. 574–588. IEEE Computer Society, Washington, DC (2013). https://doi.org/10.1109/SP.2013.45. http://dx.doi.org/10.1109/SP.2013.45

  26. Tian, D., Xiong, X., Hu, C., Liu, P.: Defeating buffer overflow attacks viavirtualization. Comput. Electr. Eng. 40(6), 1940–1950 (2014). http://dx.doi.org/10.1016/j.compeleceng.2013.11.032. http://www.sciencedirect.com/science/article/pii/S0045790613003145

  27. Tsantekidis, M., Prevelakis, V.: Library-level policy enforcement. In: SECURWARE 2017, The Eleventh International Conference on Emerging Security Information, Systems and Technologies. Rome, Italy (2017). http://www.thinkmind.org/index.php?view=article&articleid=securware_2017_2_20_30034

  28. Tsantekidis, M., Prevelakis, V.: Sophos bogus update report. Technical report, TU Braunschweig (2017)

    Google Scholar 

  29. Tsantekidis, M., Prevelakis, V.: Efficient Monitoring of Library Call Invocation. In: Sixth International Conference on Internet of Things: Systems, Management and Security (IOTSMS). pp. 387–392. Granada, Spain (2019). DOI: 10.1109/IOTSMS48152.2019.8939203

    Google Scholar 

  30. Volckaert, S., Coppens, B., Sutter, B.D.: Cloning your gadgets: complete rop attack immunity with multi-variant execution. IEEE Trans. Dependable Secure Comput. 13(4), 437–450 (2016). https://doi.org/10.1109/TDSC.2015.2411254

    Article  Google Scholar 

  31. Wagle, P., Cowan, C.: Stackguard: simple stack smash protection for GCC. In: Proceedings of the GCC Developers Summit, pp. 243–255 (2003)

    Google Scholar 

  32. i386 WX, O.: (2003). http://marc.info/?l=openbsd-misc&m=105056000801065

  33. Zeng, Q., Zhao, M., Liu, P.: HeapTherapy: an efficient end-to-end solution against heap buffer overflows. In: 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 485–496, June 2015. https://doi.org/10.1109/DSN.2015.54

Download references

Acknowledgments

This work is supported by the European Commission through the following H2020 projects: THREAT-ARREST under Grant Agreement No. 786890 and CONCORDIA under Grant Agreement No. 830927.

Author information

Authors and Affiliations

  1. Institute of Computer and Network Engineering, TU Braunschweig, Braunschweig, Germany

    Marinos Tsantekidis & Vassilis Prevelakis

Authors
  1. Marinos Tsantekidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vassilis Prevelakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marinos Tsantekidis .

Editor information

Editors and Affiliations

  1. Foundation for Research and Technology - Hellas, Heraklion, Greece

    George Hatzivasilis

  2. Technical University of Crete, Chania, Greece

    Sotiris Ioannidis

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tsantekidis, M., Prevelakis, V. (2020). Software System Exploration Using Library Call Analysis. In: Hatzivasilis, G., Ioannidis, S. (eds) Model-driven Simulation and Training Environments for Cybersecurity. MSTEC 2020. Lecture Notes in Computer Science(), vol 12512. Springer, Cham. https://doi.org/10.1007/978-3-030-62433-0_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62433-0_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62432-3

  • Online ISBN: 978-3-030-62433-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation