Abstract
We introduce a feature-based method to detect unusual patterns. The property of normality allows us to devise a framework to quickly prune the normal observations. Observations that can not be combined into any significant pattern are considered unusual. Rules that are learned from the dataset are used to construct the patterns for which we compute a score function to measure the interestingness of the unusual patterns. Experiments using the KDD Cup 99 dataset show that our approach can discover most of the attack patterns. Those attacks are in the top set of unusual patterns and have a higher score than the patterns of normal connections. The experiments also show that the algorithm can run very fast.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Beyer, K.S., Goldstein, J., Ramakrishnan, R., Shaft, U.: When is nearest neighbor meaningful? In: Beeri, C., Bruneman, P. (eds.) ICDT 1999. LNCS, vol. 1540, pp. 217–235. Springer, Heidelberg (1999)
Breunig, M.M., Kriegel, H.-P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. SIGMOD Rec. 29(2), 93–104 (2000)
Chawla, N.V., Lazarevic, A., Hall, L.O., Bowyer, K.W.: SMOTEBoost: Improving prediction of the minority class in boosting. In: Lavrač, N., Gamberger, D., Todorovski, L., Blockeel, H. (eds.) PKDD 2003. LNCS, vol. 2838, pp. 107–119. Springer, Heidelberg (2004)
Das, K., Schneider, J.: Detecting anomalous records in categorical datasets. In: KDD 2007: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 220–229. ACM Press, New York (2007)
Knorr, E.M., Ng, R.T.: Algorithms for mining distance-based outliers in large datasets. In: VLDB 1998: Proceedings of the 24rd International Conference on Very Large Data Bases, pp. 392–403. Morgan Kaufmann Publishers Inc., San Francisco (1998)
Ertöz, L., Steinbach, M., Kumar, V.: Finding clusters of different sizes, shapes, and densities in noisy, high dimensional data. In: Proceedings of the third SIAM international conference on data mining, pp. 47–58. Society for Industrial and Applied, Philadelphia (2003)
Fan, H., Zaïane, O.R., Foss, A., Wu, J.: A nonparametric outlier detection for effectively discovering top-N outliers from engineering data. In: Ng, W.-K., Kitsuregawa, M., Li, J., Chang, K. (eds.) PAKDD 2006. LNCS, vol. 3918, pp. 557–566. Springer, Heidelberg (2006)
Hawkins, D.: Identification of outliers. Chapman and Hall, London (1980)
Jarvis, R.A., Patrick, E.A.: Clustering using a similarity measure based on shared near neighbors. IEEE Transactions on Computers C-22(11), 1025–1034 (1973)
Ke, Y., Cheng, J., Ng, W.: Mining quantitative correlated patterns using an information-theoretic approach. In: KDD 2006: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 227–236. ACM Press, New York (2006)
Korn, F., Pagel, B.-U., Faloutsos, C.: On the ’dimensionality curse’ and the ’self-similarity blessing’. IEEE Transactions on Knowledge and Data Engineering 13(1), 96–111 (2001)
Kriegel, H.-P., Hubert, M.S., Zimek, A.: Angle-based outlier detection in high-dimensional data. In: KDD 2008: Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 444–452. ACM, New York (2008)
Lazarevic, A., Kumar, V.: Feature bagging for outlier detection. In: KDD 2005: Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pp. 157–166. ACM Press, New York (2005)
Mannila, H., Pavlov, D., Smyth, P.: Prediction with local patterns using cross-entropy. In: KDD 1999: Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 357–361. ACM, New York (1999)
Nguyen, M.Q., Mark, L., Omiecinski, E.: Unusual Pattern Detection in High Dimensions. In: The Pacific-Asia Conference on Knowledge Discovery and Data Mining (2008)
Newman, C.B.D., Merz, C.: UCI repository of machine learning databases (1998)
Shaft, U., Ramakrishnan, R.: Theory of nearest neighbors indexability. ACM Trans. Database Syst. 31(3), 814–838 (2006)
Papadimitriou, S., Kitagawa, H., Gibbons, P.B., Faloutsos, C.: LOCI: Fast outlier detection using the local correlation integral. In: Proceedings of the 19th International Conference on Data Engineering: 2003, pp. 315–326. IEEE Computer Society Press, Los Alamitos (2003)
Steinwart, I., Hush, D., Scovel, C.: A classification framework for anomaly detection. J. Mach. Learn. Res. 6, 211–232 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Nguyen, M.Q., Omiecinski, E., Mark, L. (2009). A Fast Feature-Based Method to Detect Unusual Patterns in Multidimensional Datasets. In: Pedersen, T.B., Mohania, M.K., Tjoa, A.M. (eds) Data Warehousing and Knowledge Discovery. DaWaK 2009. Lecture Notes in Computer Science, vol 5691. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03730-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-642-03730-6_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03729-0
Online ISBN: 978-3-642-03730-6
eBook Packages: Computer ScienceComputer Science (R0)