Abstract
This paper studies the idea of using large-scale diversification to protect operating systems and make malware ineffective. The idea is to first diversify the system call interface on a specific computer so that it becomes very challenging for a piece of malware to access resources, and to combine this with the recursive diversification of system library routines indirectly invoking system calls. Because of this unique diversification (i.e. a unique mapping of system call numbers), a large group of computers would have the same functionality but differently diversified software layers and user applications. A malicious program now becomes incompatible with its environment. The basic flaw of operating system monoculture – the vulnerability of all software to the same attacks – would be fixed this way.
Specifically, we analyze the presence of system calls in the ELF binaries. We study the locations of system calls in the software layers of Linux and examine how many binaries in the whole system use system calls. Additionally, we discuss the different ways system calls are coded in ELF binaries and the challenges this causes for the diversification process. Also, we present a diversification tool and suggest several solutions to overcome the difficulties faced in system call diversification. The amount of problematic system calls is small, and our diversification tool manages to diversify the clear majority of system calls present in standard-like Linux configurations. For diversifying all the remaining system calls, we consider several possible approaches.
This research has been funded by MATINE project 3301.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Apvrille, A., Strazzere, T.: Reducing the window of opportunity for android malware gotta catch ’em all. Int. J. Ambient Comput. Intell. 8(1–2), 61–71 (2012)
Bruschi, D., Cavallaro, L., Lanzi, A.: An efficient technique for preventing mimicry and impossible paths execution attacks. In: Performance, Computing, and Communications Conference, 2007, IPCCC 2007. IEEE Internationa, pp. 418–425, April 2007
Chew, M., Song, D.: Mitigating buffer overflows by operating system randomization (2002)
Cohen, F.B.: Operating system protection through program evolution. Comput. Secur. 12(6), 565–584 (1993)
Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscation tranformations. Technical report 148, The University of Auckland (1997)
TIS Committee: Tool Interface Standard. Executable and Linking Format (ELF) Specification. Version 1.2. Submitted to Journal of Information Security and Applications (Elsevier), under evaluation (1995)
Cooper, K.D., Harvey, T.J., Waterman, T.: Building a control-flow graph from scheduled assembly code. Technical report 02–399, Rice University (2002)
Falcarin, P., Carlo, S.D., Cabutto, A., Garazzino, N., Barberis, D.: Exploiting code mobility for dynamic binary obfuscation. In 2011 World Congress on Internet Security (WorldCIS), pp. 114–120, February 2011
Jang, M.H., Jang, M.: Security Strategies in Linux Platforms and Applications. Jones & Bartlett Publishers, Burlington (2010)
Jiang, X., Wang, H.J., Xu, D., Wang, Y.-M.: Randsys: thwarting code injection attacks with system service interface randomization. In: IEEE International Symposium on Reliable Distributed Systems, SRDS 2007, pp. 209–218 (2007)
Kerrisk, M.: The Linux Programming Interface. No Starch Press, San Francisco (2010)
Kinder, J., Zuleger, F., Veith, H.: An abstract interpretation-based framework for control flow reconstruction from binaries. In: Jones, N.D., Müller-Olm, M. (eds.) VMCAI 2009. LNCS, vol. 5403, pp. 214–228. Springer, Heidelberg (2009)
Liang, Z., Liang, B., Li, L.: A system call randomization based method for countering code injection attacks. In: International Conference on Networks Security, Wireless Communications and Trusted Computing, NSWCTC 2009, pp. 584–587 (2009)
Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM Conference on Computer and Communications Security, CCS 2003, pp. 290–299. ACM, New York, USA (2003)
Madou, M., Anckaert, B., De Bus, B., De Bosschere, K., Cappaert, J., Preneel, B.: On the effectiveness of source code transformations for binary obfuscation. In: Proceedings of the International Conference on Software Engineering Research and Practice (SERP06), pp. 527–533. CSREA Press (2006)
Popov, I.V., Debray, S.K., Andrews, G.R.: Binary obfuscation using signals. In: USENIX Security (2007)
S. Rauti, J. Holvitie, and V. Leppänen. Towards a Diversification Framework for Operating System Protection. In: Proceedings of International Conference on Computer Systems and Technologies, CompSysTech 2014 (2014)
Rauti, S., Leppänen, V.: Browser extension-based man-in-the-browser attacks against Ajax applications with countermeasures. In: Proceedings of International Conference on Computer Systems and Technologies, CompSysTech 2012, pp. 251–258. ACM Press (2012)
Rauti, S., Leppänen, V.: A proxy-like obfuscator for web application protection. Int. J. Inf. Technol. Secur. 5(1) (2014)
Lee, J.W., Lee, Y.J., Kim, H.K., Hwang, B., Ryu, K.H.: Discovering temporal relation rules mining from interval data. In: Shafazand, H., Tjoa, A.M. (eds.) EurAsia-ICT 2002. LNCS, vol. 2510, pp. 57–66. Springer, Heidelberg (2002)
Rauti, S., Leppänen, V.: Resilient code protection by JavaScript and HTML obfuscation for Ajax applications against man-in-the-browser attacks. Submitted to Journal of Information Security and Applications (Elsevier), under evaluation (2014)
Schwarz, B., Debray, S., Andrews, G.: Disassembly of executable code revisited. In: Proceedings of Ninth Working Conference on Reverse Engineering, pp. 45–54 (2002)
Sobell, M.G.: A Practical Guide to Linux. Addison-Wesley, Boston (1999)
Srivastava, A., Lanzi, A., Giffin, J., Balzarotti, D.: Operating system interface obfuscation and the revealing of hidden operations. In: Holz, T., Bos, H. (eds.) DIMVA 2011. LNCS, vol. 6739, pp. 214–233. Springer, Heidelberg (2011)
Tanenbaum, A.S.: Modern Operating Systems, 3rd edn. Prentice Hall Press, Upper Saddle River (2007)
Theiling, H.: Extracting safe and precise control flow from binaries. In: Proceedings of Seventh International Conference on Real-Time Computing Systems and Applications, pp. 23–30. IEEE (2000)
Wang, S.P.: Mastering Linux. CRC Press, Boca Raton (2011)
Wu, Z., Gianvecchio, S., Xie, M., Wang, H.: Mimimorphism: a new approach to binary code obfuscation. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 536–546. ACM, New York, USA (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Rauti, S., Laurén, S., Hosseinzadeh, S., Mäkelä, JM., Hyrynsalmi, S., Leppänen, V. (2015). Diversification of System Calls in Linux Binaries. In: Yung, M., Zhu, L., Yang, Y. (eds) Trusted Systems. INTRUST 2014. Lecture Notes in Computer Science(), vol 9473. Springer, Cham. https://doi.org/10.1007/978-3-319-27998-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-27998-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27997-8
Online ISBN: 978-3-319-27998-5
eBook Packages: Computer ScienceComputer Science (R0)