Abstract
We consider the issue of online anomaly detection from a time sequence of directional data (normalized vectors) in high dimensional systems. In spite of the practical importance, little is known about anomaly detection methods for directional data. Using a novel concept of the effective dimension of the system, we successfully formulated an anomaly detection method which is free from the “curse of dimensionality.” In our method, we derive a probability distribution function (pdf) for an anomaly metric, and use a novel update algorithm for the parameters in the pdf, where the effective dimension is included as a fitting parameter. For directional data from a computer system, we demonstrate the utility of our algorithm in anomaly detection.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Banerjee, A., Dhillon, I., Ghosh, J., Sra, S.: Expectation maximization for clustering on hyperspheres. Technical Report, TR-03-07, Department of Computer Sciences, University of Texas at Austin (2003)
Banerjee, A., Dhillon, I., Ghosh, J., Sra, S.: Generative model-based clustering of directional data. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 19–28. ACM Press, New York (2003)
Berman, A., Plemmons, R.J.: Nonnegative Matrices in the Mathematical Sciences. Classics in applied mathematics, vol. 9. SIAM, Philadelphia (1994)
Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. Journal of the American Society of Information Science 41(6), 391–407 (1990)
Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley, Chichester (2000)
Gupta, M., Neogi, A., Agarwal, M.K., Kar, G.: Discovering dynamic dependencies in enterprise environments for problem determination. In: Proceedings of 14th IFIP/IEEE Workshop on Distributed Systems: Operations and Management, pp. 221–233. IEEE Computer Society Press, Los Alamitos (2003)
IBM. Trade3. http://www-306.ibm.com/software/webservers/appserv/benchmark3.html
Idé, T., Kashima, H.: Eigenspace-based anomaly detection in computer systems. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM Press, New York (2004)
Jaakkola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems 11, 487–493 (1999)
Mardia, K.V.: Multivariate Analysis. Academic Press, London (1980)
Sarkar, S., Boyer, K.: Quantitative measures for change based on feature organization: Eigenvalues and eigenvectors. Computer Vision and Image Understanding 71, 110–136 (1998)
Strang, G.: Linear Algebra and its Applications. Academic Press, London (1976)
The Open Group. Application response measurement — ARM. http://www.opengroup.org/tech/management/arm/
Yamanishi, K., Takeuchi, J.: A unifying framework for detecting outliers and change points from non-stationary time series data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 676–681. ACM Press, New York (2002)
Yamanishi, K., Takeuchi, J., Williams, G., Milne, P.: On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 320–324. ACM Press, New York (2000)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Idé, T., Kashima, H. (2007). Effective Dimension in Anomaly Detection: Its Application to Computer Systems. In: Sakurai, A., Hasida, K., Nitta, K. (eds) New Frontiers in Artificial Intelligence. JSAI JSAI 2003 2004. Lecture Notes in Computer Science(), vol 3609. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71009-7_17
Download citation
DOI: https://doi.org/10.1007/978-3-540-71009-7_17
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-71008-0
Online ISBN: 978-3-540-71009-7
eBook Packages: Computer ScienceComputer Science (R0)