Abstract
Finding rare information behind big data is important and meaningful for outlier detection. However, to find such rare information is extremely difficult when the notorious curse of dimensionality exists in high dimensional space. Most of existing methods fail to obtain good result since the Euclidean distance cannot work well in high dimensional space. In this paper, we first perform a grid division of data for each attribute, and compare the density ratio for every point in each dimension. We then project the points of the same area to other dimensions, and then we calculate the disperse extent with defined cluster density value. At last, we sum up all weight values for each point in two-step calculations. After the process, outliers are those points scoring the largest weight. The experimental results show that the proposed algorithm can achieve high precision and recall on the synthetic datasets with the dimension varying from 100 to 10000.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Hawkins, D.: Identification of Outliers. Chapman and Hall, London (1980)
Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: Indentify density based local outliers. In: The Proceedings of the ACM SIGMOD International Conference on Management of Data (2000)
Aggarwal, C.C., Yu, P.S.: Outlier detection for high dimensional data. In: Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data, pp. 37–46. ACM, New York (2001)
Kriegel, H.P., Schubert, M., Zimek, A.: Angle-based outlier detection in high dimensional data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 444–452. ACM (2008)
Hido, S., Tsuboi, Y., Kashima, H., Sugiyama, M., Kanamori, T.: Statistical outlier detection using direct density ratio estimation. Knowledge and Information Systems 26(2), 309–336 (2011)
Papadimitriou, S., Kitagawa, H.: B.Gibbons, P.: LOCI: fast outlier detection using the local correlation integral. In: IEEE 19th International Conference on Data Engineering (2003)
Hinneburg, A., Keim, D.A.: Optimal grid-clustering: Towards breaking the curse of dimensionality in high dimensional clustering. In: Proceedings of the 25th International Conference on Very Large Data Bases (1999)
Ceglar, A., Roddick, J.F., Powers, D.M.W.: CURIO: A fast outlier and outlier cluster detection algorithm for larger datasets. In: Proceedings of the 2nd International Workshop on Integrating Artificial Intelligence and Data Mining, Australia, vol. 84 (2007)
Nag, A.K., Mitra, A.: Multiple outelier detection in multivariate data using self-organizing maps title. Computational Statistics 20, 245–264 (2005)
Kriegel, H.-P., Kröger, P., Schubert, E., Zimek, A.: Outlier Detection in Axis-Parallel Subspaces of High Dimensional Data. In: Theeramunkong, T., Kijsirikul, B., Cercone, N., Ho, T.-B. (eds.) PAKDD 2009. LNCS, vol. 5476, pp. 831–838. Springer, Heidelberg (2009)
Arcene data, http://archive.ics.uci.edu/ml/datasets/Arcene
NIPS result, http://clopinet.com/isabelle/Projects/NIPS2003/analysis.html#svm-resu
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bao, Z., Kameyama, W. (2013). A Novel Proposal for Outlier Detection in High Dimensional Space. In: Li, J., et al. Trends and Applications in Knowledge Discovery and Data Mining. PAKDD 2013. Lecture Notes in Computer Science(), vol 7867. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40319-4_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-40319-4_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40318-7
Online ISBN: 978-3-642-40319-4
eBook Packages: Computer ScienceComputer Science (R0)