Abstract
Many host-based anomaly detection systems monitor process execution at the granularity of system calls. Other recently proposed schemes instead verify the destinations of control-flow transfers to prevent the execution of attack code. This paper formally analyzes and compares real systems based on these two anomaly detection philosophies in terms of their attack detection capabilities, and proves and disproves several intuitions. We prove that for any system-call sequence model, under the same (static or dynamic) program analysis technique, there always exists a more precise control-flow sequence based model. While hybrid approaches combining system calls and control flows intuitively seem advantageous, especially when binary analysis constructs incomplete models, we prove that they have no fundamental advantage over simpler control-flow models. Finally, we utilize the ideas in our framework to make external monitoring feasible at the precise control-flow level. Our experiments show that external control-flow monitoring imposes performance overhead comparable to previous system call based approaches while detecting synthetic and real world attacks as effectively as an inlined monitor.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
GNU Indent Local Heap Overflow Vulnerability, http://www.securityfocus.com/bid/9297/
GnuPG Format String Vulnerability, http://www.securityfocus.com/bid/2797/
imapd Buffer Overflow Vulnerability, http://www.securityfocus.com/bid/130/
thttpd defang Buffer Overflow Vulnerability, http://www.securityfocus.com/bid/8906/
Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-Flow Integrity: Principles, Implementations, and Applications. In: Proceedings of ACM Computer and Communications Security (CCS), Alexandria, Virginia, November 2005, ACM Press, New York (2005)
Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: A theory of secure control flow. In: Lau, K.-K., Banach, R. (eds.) ICFEM 2005. LNCS, vol. 3785, Springer, Heidelberg (2005)
Bhatkar, S., Chaturvedi, A., Sekar, R.: Dataflow anomaly detection. In: IEEE Symposium on Security and Privacy, Oakland, California, May 2006, IEEE Computer Society Press, Los Alamitos (2006)
Chen, H., Wagner, D.: MOPS: An infrastructure for examining security properties of software. In: ACM Conference on Computer and Communications Security (CCS), Washington, DC, November 2002, ACM Press, New York (2002)
Feng, H., Giffin, J., Huang, Y., Jha, S., Lee, W., Miller, B.: Formalizing sensitivity in static analysis for intrusion detection. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, California, May 2004, IEEE Computer Society Press, Los Alamitos (2004)
Feng, H., Kolesnikov, O., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, California, May 2003, IEEE Computer Society Press, Los Alamitos (2003)
Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, California, May 1996, IEEE Computer Society Press, Los Alamitos (1996)
Gao, D., Reiter, M., Song, D.: Gray-box extraction of execution graphs for anomaly detection. In: Proceedings of the 11th ACM Conference on Computer and Communications Security (CCS), Washington, DC, October 2003, ACM Press, New York (2003)
Gao, D., Reiter, M.K., Song, D.: On gray-box program tracking for anomaly detection. In: USENIX Security Symposium, San Diego, California (August 2004)
Garvey, T., Lunt, T.: Model-based intrusion detection. In: Proceedings of the 14th National Computer Security Conf. (NCSC), Baltimore, Maryland (June 1991)
Ghosh, A., Schwartzbard, A., Schatz, M.: Learning program behavior profiles for intrusion detection. In: Proceedings of the 1st USENIX Workshop on Intrusion Detection and Network Monitoring, Santa Clara, California (April 1999)
Giffin, J., Jha, S., Miller, B.: Detecting manipulated remote call streams. In: Proceedings of the 11th USENIX Security Symposium, San Francisco, California, August 2002 (2002)
Giffin, J., Jha, S., Miller, B.: Efficient context-sensitive intrusion detection. In: Proceedings of the 11th Annual Network and Distributed Systems Security Symposium (NDSS), San Diego, California, February 2004 (2004)
Giffin, J.T., Jha, S., Miller, B.P.: Automated discovery of mimicry attacks. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, Springer, Heidelberg (2006)
Hollingsworth, J.K., Miller, B.P., Cargille, J.: Dynamic program instrumentation for scalable performance tools. In: Proceedings of the Scalable High Performance Computing Conference, Knoxville, Tennessee (May 1994)
Ko, C., Fink, G., Levitt, K.: Automated detection of vulnerabilities in privileged programs by execution monitoring. In: Proceedings of the 10th Annual Computer Security Applications Conference (ACSAC), Orlando, Florida (December 1994)
Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: Proceedings of the USENIX Security Symposium, Baltimore, Maryland (August 2005)
Kruegel, C., Mutz, D., Valeur, F., Vigna, G.: On the detection of anomalous system call arguments. In: Snekkenes, E., Gollmann, D. (eds.) ESORICS 2003. LNCS, vol. 2808, Springer, Heidelberg (2003)
Lam, L., Chiueh, T.: Automatic extraction of accurate application-specific sandboxing policy. In: Recent Advances in Intrusion Detection, Sophia Antipolis, France, September 2004 (2004)
Lam, L., Li, W., Chiueh, T.: Accurate and automated system call policy-based intrusion prevention. In: The International Conference on Dependable Systems and Networks (DSN), Philadelphia, PA, USA (June 2006)
Landi, W.: Undecidability of static analysis. ACM Letters on Programming Languages and Systems (LOPLAS) 1(4), 323–337 (1992)
Lee, W., Stolfo, S., Mok, K.: A data mining framework for building intrusion detection models. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, California, May 1999, IEEE Computer Society Press, Los Alamitos (1999)
Sekar, R., Bendre, M., Bollineni, P., Dhurjati, D.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, California, May 2001, IEEE Computer Society Press, Los Alamitos (2001)
Tan, K., Killourhy, K.S., Maxion, R.A.: Undermining an anomaly-based intrusion detection system using common exploits. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, Springer, Heidelberg (2002)
Vigna, G., Kruegel, C.: Handbook of Information Security. ch. Host-based Intrusion Detection Systems. Wiley, Chichester (December 2005)
Wagner, D.: Static Analysis and Computer Security: New Techniques for Software Assurance. Ph.D. dissertation, University of California at Berkeley (2000)
Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, California, May 2001, IEEE Computer Society Press, Los Alamitos (2001)
Wagner, D., Soto, P.: Mimicry attacks on host based intrusion detection systems. In: Proceedings of the Ninth ACM Conference on Computer and Communications Security (CCS), Washington, DC, November 2002, ACM Press, New York (2002)
Xu, H., Du, W., Chapin, S.J.: Context sensitive anomaly monitoring of process control flow to detect mimicry attacks and impossible paths. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, Springer, Heidelberg (2004)
Zhang, T., Zhuang, X., Lee, W., Pande, S.: Anomalous path detection with hardware support. In: Proceedings of the International Conference on Compilers, Architectures and Synthesis of Embedded Systems (CASES), San Francisco, CA (July 2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sharif, M., Singh, K., Giffin, J., Lee, W. (2007). Understanding Precision in Host Based Intrusion Detection. In: Kruegel, C., Lippmann, R., Clark, A. (eds) Recent Advances in Intrusion Detection. RAID 2007. Lecture Notes in Computer Science, vol 4637. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74320-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-540-74320-0_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74319-4
Online ISBN: 978-3-540-74320-0
eBook Packages: Computer ScienceComputer Science (R0)