Abstract
Attacks against privileged applications can be detected by analyzing the stream of system calls issued during process execution. In the last few years, several approaches have been proposed to detect anomalous system calls. These approaches are mostly based on modeling acceptable system call sequences. Unfortunately, the techniques proposed so far are either vulnerable to certain evasion attacks or are too expensive to be practical. This paper presents a novel approach to the analysis of system calls that uses a composition of dynamic analysis and learning techniques to characterize anomalous system call invocations in terms of both the invocation context and the parameters passed to the system calls. Our technique provides a more precise detection model with respect to solutions proposed previously, and, in addition, it is able to detect data modification attacks, which cannot be detected using only system call sequence analysis.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ammons, G., Ball, T., Larus, J.R.: Exploiting hardware performance counters with flow and context sensitive profiling. In: Proceedings of the Conference on Programming Language Design and Implementation (PLDI 1997) (1997)
Feng, H., Giffin, J., Huang, Y., Jha, S., Lee, W., Miller, B.: Formalizing sensitivity in static analysis for intrusion detection. In: Proceedings of the IEEE Symposium on Security and Privacy, May 2004, IEEE Computer Society Press, Los Alamitos (2004)
Feng, H., Kolesnikov, O., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings of the IEEE Symposium on Security and Privacy, May 2003, IEEE Computer Society Press, Los Alamitos (2003)
Forrest, S.: A Sense of Self for UNIX Processes. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, CA, May 1996, pp. 120–128. IEEE Computer Society Press, Los Alamitos (1996)
Gao, D., Reiter, M., Song, D.: Gray-Box Extraction of Execution Graphs for Anomaly Detection. In: Proceedings of ACM CCS, Washington, DC, USA, October 2004, pp. 318–329. ACM Press, New York (2004)
Gao, D., Reiter, M., Song, D.: On Gray-Box Program Tracking for Anomaly Detection. In: Proceedings of the 13th USENIX Security Symposium, San Diego, CA, USA (August 2004)
Giffin, J., Jha, S., Miller, B.: Detecting Manipulated Remote Call Streams. In: Proceedings of the 11th USENIX Security Symposium, pp. 61–79 (2002)
Giffin, J., Jha, S., Miller, B.: Efficient context-sensitive intrusion detection. In: Proceedings of the 11th Network and Distributed System Security Symposium, San Diego, California (February 2004)
Hind, M., Burke, M., Carini, P., Choi, J.-D.: Interprocedural pointer alias analysis. ACM Transactions on Programming Languages 21(4) (July 1999)
Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: Proceedings of the 14th USENIX Security Symposium (July 2005)
Kruegel, C., Mutz, D., Robertson, W., Valeur, F.: Bayesian Event Classification for Intrusion Detection. In: Omondi, A.R., Sedukhin, S. (eds.) ACSAC 2003. LNCS, vol. 2823, Springer, Heidelberg (2003)
Kruegel, C., Mutz, D., Valeur, F., Vigna, G.: On the Detection of Anomalous System Call Arguments. In: Snekkenes, E., Gollmann, D. (eds.) ESORICS 2003. LNCS, vol. 2808, pp. 326–343. Springer, Heidelberg (2003)
Lee, S., Low, W., Wong, P.: Learning Fingerprints for a Database Intrusion Detection System. In: Gollmann, D., Karjoth, G., Waidner, M. (eds.) ESORICS 2002. LNCS, vol. 2502, Springer, Heidelberg (2002)
Mutz, D.: Context-sensitive Multi-model Anomaly Detection. Ph.d. thesis, UCSB (June 2006)
Mutz, D., Valeur, F., Kruegel, C., Vigna, G.: Anomalous System Call Detection. ACM Transactions on Information and System Security 9(1), 61–93 (2006)
Nystrom, E., Kim, H., Hwu, W.: Importance of heap specialization in pointer analysis. In: Proceedings of Program Analysis for Software Tools and Engineering (2004)
Robertson, W., Vigna, G., Kruegel, C., Kemmerer, R.: Using Generalization and Characterization Techniques in the Anomaly-based Detection of Web Attacks. In: Proceeding of NDSS, San Diego, CA (February 2006)
Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of the IEEE Symposium on Security and Privacy, May 2001, IEEE Computer Society Press, Los Alamitos (2001)
Sekar, R., Venkatakrishnan, V., Basu, S., Du Varney, B.S.D.: Model-carrying code: A practical approach for safe execution of untrusted applications. In: Proceedings of the 19th ACM Symposium on Operating Systems Principles, ACM Press, New York (2003)
SNARE - System iNtrusion Analysis and Reporting Environment, http://www.intersectalliance.com/projects/Snare
Stolcke, A., Omohundro, S.: Hidden Markov Model Induction by Bayesian Model Merging. Advances in Neural Information Processing Systems (1993)
Stolcke, A., Omohundro, S.: Inducing probabilistic grammars by bayesian model merging. In: Proceedings of the International Conference on Grammatical Inference (1994)
Wagner, D., Dean, D.: Intrusion Detection via Static Analysis. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, CA, May 2001, IEEE Press, Los Alamitos (2001)
Wagner, D., Soto, P.: Mimicry Attacks on Host-Based Intrusion Detection Systems. In: Proceedings of ACM CCS, Washington DC, USA, November 2002, ACM Press, New York (2002)
Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: Alternative data models. In: Proceedings of the IEEE Symposium on Security and Privacy, pp. 133–145. IEEE Computer Society Press, Los Alamitos (1999)
Wespi, A., Dacier, M., Debar, H.: Intrusion Detection Using Variable-Length Audit Trail Patterns. In: Debar, H., Mé, L., Wu, S.F. (eds.) RAID 2000. LNCS, vol. 1907, Springer, Heidelberg (2000)
Xu, H., Du, W., Chapin, S.: Context Sensitive Anomaly Monitoring of Process Control Flow to Detect Mimicry Attacks and Impossible Paths. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, Springer, Heidelberg (2004)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mutz, D., Robertson, W., Vigna, G., Kemmerer, R. (2007). Exploiting Execution Context for the Detection of Anomalous System Calls. In: Kruegel, C., Lippmann, R., Clark, A. (eds) Recent Advances in Intrusion Detection. RAID 2007. Lecture Notes in Computer Science, vol 4637. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74320-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-540-74320-0_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74319-4
Online ISBN: 978-3-540-74320-0
eBook Packages: Computer ScienceComputer Science (R0)