Skip to main content
SpringerLink
Log in

An Efficient Algorithm for Distributed Outlier Detection in Large Multi-Dimensional Datasets

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

The distance-based outlier is a widely used definition of outlier. A point is distinguished as an outlier on the basis of the distances to its nearest neighbors. In this paper, to solve the problem of outlier computing in distributed environments, DBOZ, a distributed algorithm for distance-based outlier detection using Z-curve hierarchical tree (ZH-tree) is proposed. First, we propose a new index, ZH-tree, to effectively manage the data in a distributed environment. ZH-tree has two desirable advantages, including clustering property to help search the neighbors of a point, and hierarchical structure to support space pruning. We also design a bottom-up approach to build ZH-tree in parallel, whose time complexity is linear to the number of dimensions and the size of dataset. Second, DBOZ is proposed to compute outliers in distributed environments. It consists of two stages. 1) To avoid calculating the exact nearest neighbors of all the points, we design a greedy method and a new ZH-tree based k-nearest neighbor searching algorithm (ZHkNN for short) to obtain a threshold LW. 2) We propose a filter-and-refine approach, which first filters out the unpromising points using LW, and then outputs the final outliers through refining the remaining points. At last, the efficiency and the effectiveness of ZH-tree and DBOZ are testified through a series of experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hawkins D M. Identification of Outliers. Springer, 1980.

  2. Barnett V, Lewis T. Outliers in Statistical Data. Wiley New York, 1994.

  3. Rousseeuw P J, Leroy A M. Robust Regression and Outlier Detection. John Wiley & Sons, 2003.

  4. Knorr E M, Ng R T. Algorithms for mining distancebased outliers in large datasets. In Proc. the 24th International Conference on Very Large Data Bases, August 1998, pp. 392-403.

  5. Ramaswamy S, Rastogi R, Shim K. Efficient algorithms for mining outliers from large data sets. ACM SIGMOD Record, 2000, 29(2): 427–438.

    Article  Google Scholar 

  6. Angiulli F, Pizzuti C. Outlier mining in large high dimensional data sets. IEEE Transactions on Knowledge and Data Engineering, 2005, 17(2): 203–215.

    Article  MathSciNet  Google Scholar 

  7. Angiulli F, Pizzuti C. Fast outlier detection in high dimensional spaces. In Proc. the 6th European Conference on Principles of Data Mining and Knowledge Discovery, August 2002, pp. 15-27.

  8. Schubert E, Zimek A, Kriegel H P. Local outlier detection reconsidered: A generalized view on locality with applications to spatial, video, and network outlier detection. Data Mining and Knowledge Discovery, 2014, 28(1): 190–237.

    Article  MathSciNet  MATH  Google Scholar 

  9. Shiokawa H, Fujiwara Y, Onizuka M. Scan++: Efficient algorithm for finding clusters, hubs and outliers on large-scale graphs. Proceedings of the VLDB Endowment, 2015, 8(11): 1178–1189.

    Article  Google Scholar 

  10. Aggarwal C C, Yu P S. Outlier detection for high dimensional data. ACM SIGMOD Record, 2001, 30(2): 37–46.

    Article  Google Scholar 

  11. Breunig M M, Kriegel H P, Ng R T, Sander J. LOF: Identifying density-based local outliers. ACM SIGMOD Record, 2000, 29(2): 93–104.

    Article  Google Scholar 

  12. He Z, Xu X, Deng S. Discovering cluster-based local outliers. Pattern Recognition Letters, 2003, 24(9/10): 1641-1650.

    Article  MATH  Google Scholar 

  13. Bay S D, Schwabacher M. Mining distance-based outliers in near linear time with randomization and a simple pruning rule. In Proc. the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2003, pp. 29-38.

  14. Angiulli F, Fassetti F. Very efficient mining of distance based outliers. In Proc. the 16th ACM Conference on Information and Knowledge Management, November 2007, pp. 791-800.

  15. Guttman A. R-trees: A dynamic index structure for spatial searching. ACM SIGMOD Record, 1984, 14(2): 47–57.

    Article  Google Scholar 

  16. Ciaccia P, Patella M, Zezula P. M-tree: An efficient access method for similarity search in metric spaces. In Proc. the 23rd International Conference on Very Large Databases, August 1997, pp. 426-435.

  17. Beyer K, Goldstein J, Ramakrishnan R, Shaft U. When is “nearest neighbor” meaningful? In Proc. the 7th International Conference on Database Theory, January 1999, pp. 217-235.

  18. Wang B, Yang X C, Wang G R, Yu G. Outlier detection over sliding windows for probabilistic data streams. Journal of Computer Science and Technology, 2010, 25(3): 389–400.

    Article  MathSciNet  Google Scholar 

  19. Cao K Y, Wang G R, Han D H, Ding G H, Wang A X, Shi L X. Continuous outlier monitoring on uncertain data streams. Journal of Computer Science and Technology, 2014, 29(3): 436–448.

    Article  MathSciNet  Google Scholar 

  20. Gupta M, Gao J, Aggarwal C, Han J. Outlier detection for temporal data. Synthesis Lectures on Data Mining and Knowledge Discovery, 2014, 5(1): 1–129.

    Article  Google Scholar 

  21. Yan Y, Zhang J, Huang B, Sun X, Mu J, Zhang Z, Moscibroda T. Distributed outlier detection using compressive sensing. In Proc. the 2015 ACM SIGMOD International Conference on Management of Data, May 31-June 04, 2015, pp. 3-16.

  22. Hung E, Cheung D W. Parallel mining of outliers in large database. Distributed and Parallel Databases, 2002, 12(1): 5–26.

    Article  MATH  Google Scholar 

  23. Lozano E, Acuña E. Parallel algorithms for distance-based and density-based outliers. In Proc. the 5th IEEE International Conference on Data Mining, November 2005, pp. 729- 732.

  24. Angiulli F, Basta S, Lodi S, Sartori C. Distributed strategies for mining outliers in large data sets. IEEE Transactions on Knowledge and Data Engineering, 2013, 25(7): 1520–1532.

    Article  Google Scholar 

  25. Jagadish H V, Ooi B C, Tan K, Yu C, Zhang R. iDistance: An adaptive B+−tree based indexing method for nearest neighbor search. ACM Transaction Database System, 2005, 30(2): 364–397.

    Article  Google Scholar 

  26. Ramsak F, Markl V, Fenk R, Zirkel M, Elhardt K, Bayer R. Integrating the UB-tree into a database system kernel. In Proc. the 26th International Conference on Very Large Data Bases, September 2000, pp. 263-272.

  27. O’Malley O. Terabyte sort on apache hadoop. http://sortbenchmark.org/YahooHadoop.pdf, October 2015.

Download references

Author information

Authors and Affiliations

  1. College of Information Science and Engineering, Northeastern University, Shenyang, 110819, China

    Xi-Te Wang, De-Rong Shen, Mei Bai, Tie-Zheng Nie, Yue Kou & Ge Yu

Authors
  1. Xi-Te Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. De-Rong Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mei Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tie-Zheng Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yue Kou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ge Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xi-Te Wang.

Additional information

Special Section on Networking and Distributed Computing for Big Data

This work was supported by the National Basic Research 973 Program of China under Grant No. 2012CB316201, the National Natural Science Foundation of China under Grant Nos. 61033007 and 61472070, and the Fundamental Research Funds for the Central Universities of China under Grant No. N120816001.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, XT., Shen, DR., Bai, M. et al. An Efficient Algorithm for Distributed Outlier Detection in Large Multi-Dimensional Datasets. J. Comput. Sci. Technol. 30, 1233–1248 (2015). https://doi.org/10.1007/s11390-015-1596-0

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-015-1596-0

Keywords

Search

Navigation