Abstract
Model-based anomaly detection systems restrict program execution by a predefined model of allowed system call sequences. These systems are useful only if they detect actual attacks. Previous research developed manually-constructed mimicry and evasion attacks that avoided detection by hiding a malicious series of system calls within a valid sequence allowed by the model. Our work helps to automate the discovery of such attacks. We start with two models: a program model of the application’s system call behavior and a model of security-critical operating system state. Given unsafe OS state configurations that describe the goals of an attack, we then find system call sequences allowed as valid execution by the program model that produce the unsafe configurations. Our experiments show that we can automatically find attack sequences in models of programs such as wu-ftpd and passwd that previously have only been discovered manually. When undetected attacks are present, we frequently find the sequences with less than 2 seconds of computation.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Ball, T., Rajamani, S.K.: Bebop: A symbolic model checker for boolean programs. In: 7th International SPIN Workshop on Model Checking of Software. Stanford, California (2000)
Besson, F., Jensen, T., Métayer, D.L., Thorn, T.: Model checking security properties of control-flow graphs. Journal of Computer Security 9, 217–250 (2001)
Chen, H., Wagner, D.: MOPS: An infrastructure for examining security properties of software. In: 9th ACM Conference on Computer and Communications Security (CCS), Washington, DC (November 2002)
Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press, Cambridge (2000)
Esparza, J., Hansel, D., Rossmanith, P., Schwoon, S.: Efficient algorithms for model checking pushdown systems. In: Computer Aided Verification (CAV), Chicago, Illinois (July 2000)
Feng, H.H., Giffin, J.T., Huang, Y., Jha, S., Lee, W., Miller, B.P.: Formalizing sensitivity in static analysis for intrusion detection. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2004)
Forrest, S.: Data sets—synthetic FTP (1998), http://www.cs.unm.edu/~immsec/data/FTP/UNM/-normal/synth/
Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for UNIX processes. In: IEEE Symposium on Security and Privacy, Oakland, California (May 1996)
Gao, D., Reiter, M.K., Song, D.: On gray-box program tracking for anomaly detection. In: USENIX Security Symposium, San Diego, California (August 2004)
Giffin, J.T., Dagon, D., Jha, S., Lee, W., Miller, B.P.: Environment-sensitive intrusion detection. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 185–206. Springer, Heidelberg (2006)
Giffin, J.T., Jha, S., Miller, B.P.: Detecting manipulated remote call streams. In: 11th USENIX Security Symposium, San Francisco, California (August 2002)
Gopalakrishna, R., Spafford, E.H., Vitek, J.: Efficient intrusion detection using automaton inlining. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2005)
Guttman, J.D., Herzog, A.L., Ramsdell, J.D., Skorupka, C.W.: Verifying information flow goals in Security-Enhanced Linux. Journal of Computer Security 13, 115–134 (2005)
Lam, L.C., Chiueh, T.-c.: Automatic extraction of accurate application-specific sandboxing policy. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 1–20. Springer, Heidelberg (2004)
Ramakrishnan, C.R., Sekar, R.: Model-based vulnerability analysis of computer systems. In: 2nd International Workshop on Verification, Model Checking and Abstract Interpretation, Pisa, Italy (September 1998)
Schneider, F.B.: Enforceable security policies. ACM Transactions on Information and System Security 3(1), 30–50 (2000)
Schwoon, S.: Model-Checking Pushdown Systems. Ph.D. dissertation, Technische Universität München (June 2002)
Schwoon, S.: Moped—a model-checker for pushdown systems (2006), http://www.fmi.uni-stuttgart.de/szs/tools/moped/
Sekar, R., Bendre, M., Bollineni, P., Dhurjati, D.: A fast automaton-based method for detecting anomalous program behaviors. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2001)
Tan, K.M.C., Killourhy, K.S., Maxion, R.A.: Undermining an anomaly-based intrusion detection system using common exploits. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, p. 54. Springer, Heidelberg (2002)
Tan, K., Maxion, R.A.: “Why 6?” Defining the operational limits of stide, an anomaly based intrusion detector. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2002)
Tan, K., McHugh, J., Killourhy, K.: Hiding intrusions: From the abnormal to the normal and beyond. In: 5th International Workshop on Information Hiding, Noordwijkerhout, Netherlands (October 2002)
Wagner, D., Dean, D.: Intrusion detection via static analysis. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2001)
Wagner, D., Soto, P.: Mimicry attacks on host based intrusion detection systems. In: 9th ACM Conference on Computer and Communications Security, Washington, DC (November 2002)
Walker, B.J., Kemmerer, R.A., Popek, G.J.: Specification and verification of the UCLA Unix security kernel. Communications of the ACM 23(2) (February 1980)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Giffin, J.T., Jha, S., Miller, B.P. (2006). Automated Discovery of Mimicry Attacks. In: Zamboni, D., Kruegel, C. (eds) Recent Advances in Intrusion Detection. RAID 2006. Lecture Notes in Computer Science, vol 4219. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11856214_3
Download citation
DOI: https://doi.org/10.1007/11856214_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-39723-6
Online ISBN: 978-3-540-39725-0
eBook Packages: Computer ScienceComputer Science (R0)