Skip to main content
SpringerLink
Log in

A Survey on Function and System Call Hooking Approaches

  • Published:
Journal of Hardware and Systems Security Aims and scope Submit manuscript

Abstract

Functions and system calls are effective indicators of the behavior of a process. These subroutines are useful for identifying unauthorized behavior caused by malware or for developing a better understanding of the lower-level operations of an application. Code obfuscation, however, often prevents user monitoring and modification of subroutine calls. Subroutine hooking offers a solution to this limitation. Function and system call hooking approaches allow for subroutine instrumentation, making hooking a valuable and versatile skill across industry and academia. In this survey, we present several criteria for the classification and selection of hooking tools and techniques as well as an examination of the major hooking approaches used on Windows, Linux, macOS, iOS, and Android operating systems. We also evaluate and compare the performance of different subroutine hooking tools and techniques based on computing resource utilization such as CPU time, memory, and wall-clock time. To the best of our knowledge, this is the first paper that encompasses both system call and function hooking techniques and tools across the major desktop and mobile operating systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. API monitor. http://www.rohitab.com/apimonitor. [Online; accessed 22-December-2016]

  2. AppInit DLLs and secure boot. https://msdn.microsoft.com/en-us/library/windows/desktop/dn280412(v=vs.85).aspx. [Online; acces- sed 20-December-2016]

  3. BitBlaze: binary analysis for computer security. http://bitblaze.cs.berkeley.edu/. [Online; accessed 25-June-2017]

  4. Cydia substrate. http://www.cydiasubstrate.com. [Online; accessed 20-December-2016]

  5. dlsym(3)—linux man page. http://man7.org/linux/man-pages/man3/dlsym.3.html. [Online; accessed 16-December-2016]

  6. dtruss(1m)—Mac OS X man pages. https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/dtruss.1m.html. [Online; accessed 16-December-2016]

  7. DYLD(1)—Mac OS X man pages. https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/dyld.1.html. [Online; accessed 16-December-2016]

  8. EasyHook. https://easyhook.github.io. [Online; accessed 22-December-2016]

  9. Frida. https://www.frida.re. [Online; accessed 19-December-2016]

  10. Instruments user guide. https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/. [Online; accessed 20-December-2016]

  11. Introspy-Android. http://isecpartners.github.io/Introspy-Android/. [Online; accessed 19-December-2016]

  12. ld.so(8)—linux man page. http://man7.org/linux/man-pages/man8/ld.so.8.html. [Online; accessed 22-December-2016]

  13. Microsoft detours. https://www.microsoft.com/en-us/research/project/detours/. [Online; accessed 02-July-2017]

  14. Nektra: advanced computing. http://www.nektra.com/products/spystudio-api-monitor/. [Online; accessed 02-July-2017]

  15. POSIX.1-2008 The pen Group Base Specifications Issue 7. http://pubs.opengroup.org/onlinepubs/9699919799/. [Online; accessed 20-January-2017]

  16. ptrace—linux manual page. http://man7.org/linux/man-pages/man2/ptrace.2.html. [Online; accessed 14-December-2016]

  17. SpyStudio Overview. http://www.nektra.com/products/spystudio-api-monitor/ http://www.nektra.com/products/spystudio-api-monitor/. [Online; accessed 18-December-2016]

  18. strace for android. https://github.com/alireza7991?tab=repositories. [Online; accessed 20-December-2016]

  19. Strace for NT. http://seriss.com/people/erco/ftp/winnt/strace/. [Online; accessed 10-January-2017]

  20. strace(1)—linux man page. https://linux.die.net/man/1/strace. [Online; accessed 14-December-2016]

  21. Theos/setup. http://iphonedevwiki.net/index.php/Theos/Setup. [Online; accessed 20-December-2016]

  22. WinAPIOverride. http://jacquelin.potier.free.fr/winapioverride32/. [Online; accessed 18-December-2016]

  23. Windows API index. https://msdn.microsoft.com/en-us/library/windows/desktop/ff818516(v=vs.85).aspx. [Online; accessed 20-January-2017]

  24. Windows API index. https://msdn.microsoft.com/en-us/library/windows/desktop/ff818516(v=vs.85).aspx. [Online; accessed 02-July-2017]

  25. Xcode 8. https://developer.apple.com/xcode/. [Online; accessed 20-December-2016]

  26. Xposed module repository. http://repo.xposed.info. [Online; accessed 19-December-2016]

  27. Abimbola AA, Munoz JM, Buchanan WJ (2006) Nethost-sensor: monitoring a target host’s application via system calls. Inf Secur Tech Rep 11(4):166–175

    Article  Google Scholar 

  28. Andersson S, Clark A, Mohay G, Schatz B, Zimmermann J (2005) A framework for detecting network-based code injection attacks targeting Windows and UNIX. In: Proceedings of the 21st annual computer security applications conference, ACSAC ’05. IEEE Computer Society, Washington, DC, USA, pp 49–58

  29. Babun L, Aksu H, Uluagac AS (2017) Identifying counterfeit smart grid devices: a lightweight system level framework. In: 2017 international conference on communications (ICC)

  30. Backes M, Gerling S, Hammer C, Maffei M, von Styp-Rekowsky P (2013) Appguard: enforcing user requirements on Android apps. In: Proceedings of the 19th international conference on tools and algorithms for the construction and analysis of systems, TACAS’13. Springer, Berlin, pp 543–548

  31. Bovet D, Cesati M (2005) Understanding the Linux kernel Oreilly & Associates Inc

  32. Davis B, Chen H (2013) Retroskeleton: retrofitting Android apps. In: Proceeding of the 11th annual international conference on mobile systems, applications, and services, MobiSys ’13. ACM, New York, pp 181–192

  33. Eder T, Rodler M, Vymazal D, Zeilinger M (2013) ANANAS - A framework for analyzing android applications. In: 2013 eighth international conference on availability, reliability and security (ARES), pp 711–719

  34. Enck W, Octeau D, McDaniel P, Chaudhuri S (2011) A study of android application security. In: Proceedings of the 20th USENIX conference on security, SEC’11. USENIX Association, Berkeley, pp 21–21

  35. Holy Father (2004) Hooking Windows API-Technics of hooking API functions on Windows. CodeBreakers-Journal, 1(2)

  36. Garfinkel T (2003) Traps and pitfalls practical problems in system call interposition based security tools. In: Proceedings of network and distributed systems security symposium, pp 163–176

  37. Gregg B, Mauro J (2011) DTrace: dynamic tracing in oracle Solaris, Mac OS X and freeBSD, 1st edn. Prentice Hall Press, Upper Saddle River

    Google Scholar 

  38. Guo PJ, Engler D (2011) Using system call interposition to automatically create portable software packages. In: Proceedings of the 2011 USENIX conference on USENIX annual technical conference, USENIXATC’11. USENIX Association, Berkeley, pp 21–21

  39. Hunt G, Brubacher D (1999) Detours: binary interception of Win32 functions. In: Proceedings of the 3rd conference on USENIX windows NT symposium - Volume 3, WINSYM’99. USENIX Association, Berkeley, pp 14–14

  40. Jeong Y, Lee H, Cho S, Han S, Park M (2014) A kernel-based monitoring approach for analyzing malicious behavior on Android. In: Proceedings of the 29th annual ACM symposium on applied computing, SAC ’14. ACM, New York, pp 1737–1738

  41. Keniston J, Mavinakayanahalli A, Panchamukhi P, Prasad V (2007) Ptrace, utrace, uprobes lightweight, dynamic tracing of user apps. In: Proceedings of the 2007 Linux symposium, pp 215–224

  42. Kim S-W (2012) Intercepting system API calls. https://software.intel.com/en-us/articles/intercepting-system-api-calls. [Online; accessed 18-December-2016]

  43. Kim T, Zeldovich N (2013) Practical and effective sandboxing for non-root users. In: Presented as part of the 2013 USENIX annual technical conference (USENIX ATC 13). USENIX, San Jose, pp 139–144

  44. Liu ST, Huang Hc, Chen YM (2011) A system call analysis method with mapreduce for malware detection. In: 2011 IEEE 17th international conference on parallel and distributed systems, pp 631–637

  45. Zhao F, Tan L, Zhang X (2012) Advanced operating and distributed system android and iOS platform study final report

  46. Ligh MH, Adair S, Hartstein B, Richards M (2011) Malware analyst’s codebook and DVD: tools and techniques for fighting malicious code. Wiley, New York

    Google Scholar 

  47. Madani P, Vlajic N (2016) Towards sequencing malicious system calls. In: 2016 IEEE conference on communications and network security (CNS), pp 376–377

  48. Marhusin MF, Larkin H, Lokan C, Cornforth D (2008) An evaluation of API calls hooking performance. In: Proceedings of the 2008 international conference on computational intelligence and security - volume 01, CIS ’08. IEEE Computer Society, Washington, pp 315–319

  49. Mehdi B, Ahmed F, Khayyam SA, Farooq M (2010) Towards a theory of generalizing system call representation for in-execution malware detection. In: 2010 IEEE international conference on communications, pp 1–5

  50. (2015). Microsoft. Visual studio, Microsoft portable executable and common object file format specification. Technical report, Microsoft

  51. Myers DS, Bazinet AL (2004) Intercepting arbitrary functions on Windows, UNIX, and Macintosh OS X platforms. Technical report, Center for Bioinformatics and Computational Biology, Institute for Advanced Computer Studies University of Maryland

  52. Qin F, Wang C, Li Z, Kim Hs, Zhou Y, Wu Y (2006) Lift: a low-overhead practical information flow tracking system for detecting security attacks. In: 2006 39th annual IEEE/ACM international symposium on microarchitecture (MICRO’06), pp 135– 148

  53. Richter JM, Nasarre C (2007) Windows via C/C++, 5th edn. Microsoft Press, USA

    Google Scholar 

  54. Rubanov VV, Shatokhin EA (2011) Runtime verification of linux kernel modules based on call interception. In: 2011 fourth IEEE international conference on software testing, verification and validation, pp 180–189

  55. Russello G, Jimenez AB, Naderi H, van der Mark W (2013) FireDroid: hardening security in almost-stock android. In: Proceedings of the 29th annual computer security applications conference, ACSAC ’13. ACM, New York, pp 319–328

  56. Russinovich ME, Solomon DA, Ionescu A (2012) Windows internals, Part 1: covering windows server 2008 R2 and Windows 7, 6th edition. Microsoft Press, USA

    Google Scholar 

  57. Mohd Shaid SZ, Maarof MA (2015) In memory detection of Windows API call hooking technique. In: 2015 international conference on computer, communications, and control technology (i4CT), pp 294–298

  58. Silberschatz A, Galvin PB, Gagne G (2008) Operating system concepts, 8th edn. Wiley Publishing, New York

    MATH  Google Scholar 

  59. Song D, Brumley D, Yin H, Caballero J, Jager I, Kang MG, Liang Z, Newsome J, Poosankam P, Saxena P (2008) BitBlaze: a new approach to computer security via binary analysis. In: Proceedings of the 4th international conference on information systems security. Keynote Invited paper., Hyderabad, India

  60. Sun M, Zheng M, Lui JCS, Jiang X (2014) Design and implementation of an android host-based intrusion prevention system. In: Proceedings of the 30th annual computer security applications conference, ACSAC ’14. ACM, New York, pp 226–235

  61. Sze WK, Sekar R (2015) Provenance-based integrity protection for windows. In: Proceedings of the 31st annual computer security applications conference, ACSAC 2015. ACM, New York, pp 211–220

  62. Vogl S, Pfoh J, Kittel T, Eckert C (2014) Persistent data-only malware: function hooks without code. In: NDSS

  63. Wampler DR (2007) Methods for detecting Kernel Rootkits. PhD thesis, Louisville, KY, USA. AAI3293571

  64. Willems C, Holz T, Freiling F (2007) Toward automated dynamic malware analysis using CWSandbox. IEEE Secur Priv 5(2):32– 39

    Article  Google Scholar 

  65. Wißfeld M, von Styp-Rekowsky P, Backes M Callee-side method hook injection on the new Android runtime ART

  66. Xu K, Li Y, Deng RH (2016) Iccdetector: Icc-based malware detection on android. IEEE Trans Inf Forensics Secur 11(6):1252–1264

    Article  Google Scholar 

  67. Xu R, Saïdi H, Anderson R (2012) Aurasium: practical policy enforcement for android applications. In: Proceedings of the 21st USENIX conference on security symposium, Security’12. USENIX Association, Berkeley, pp 27–27

  68. Ye Y, Wang D, Li T, Dongyi Y (2007) IMDS intelligent malware detection system. In: Proceedings of the 13th ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’07. ACM, New York, pp 1043–1047

  69. Yin H, Liang Z, Song D (2008) HookFinder: identifying and understanding malware hooking behaviors. In: Proceedings of the 15th annual network and distributed system security symposium (NDSS’08)

  70. Yucheng G, Peng W, Juwei L, Qingping G (2011) A way to detect computer trojan based on DLL preemptive injection. In: 2011 tenth international symposium on distributed computing and applications to business, engineering and science (DCABES), pp 255–258

  71. Zdziarski J (2012) Hacking and securing iOS applications: stealing data, hijacking software, and how to prevent it. O’Reilly Media, Inc., Sebastopol

    Google Scholar 

Download references

Acknowledgments

This work was partly supported by the US NSF-CAREER-CNS-1453647, US DOE DE-OE0000779, and US NSF-REU-CNS-1461119. Any opinions, findings, and conclusions or recommendations expressed in this work are those of the authors and do not necessarily reflect the views of the funding agencies.

Author information

Authors and Affiliations

  1. Florida International University-Engineering Center, Miami, FL, 33174, USA

    Juan Lopez, Leonardo Babun, Hidayet Aksu & A. Selcuk Uluagac

Authors
  1. Juan Lopez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leonardo Babun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hidayet Aksu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. Selcuk Uluagac
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonardo Babun.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lopez, J., Babun, L., Aksu, H. et al. A Survey on Function and System Call Hooking Approaches. J Hardw Syst Secur 1, 114–136 (2017). https://doi.org/10.1007/s41635-017-0013-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41635-017-0013-2

Keywords

Search

Navigation