skip to main content
10.1145/2339530.2339669acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article

A near-linear time approximation algorithm for angle-based outlier detection in high-dimensional data

Published:12 August 2012Publication History

ABSTRACT

Outlier mining in d-dimensional point sets is a fundamental and well studied data mining task due to its variety of applications. Most such applications arise in high-dimensional domains. A bottleneck of existing approaches is that implicit or explicit assessments on concepts of distance or nearest neighbor are deteriorated in high-dimensional data. Following up on the work of Kriegel et al. (KDD '08), we investigate the use of angle-based outlier factor in mining high-dimensional outliers. While their algorithm runs in cubic time (with a quadratic time heuristic), we propose a novel random projection-based technique that is able to estimate the angle-based outlier factor for all data points in time near-linear in the size of the data. Also, our approach is suitable to be performed in parallel environment to achieve a parallel speedup. We introduce a theoretical analysis of the quality of approximation to guarantee the reliability of our estimation algorithm. The empirical experiments on synthetic and real world data sets demonstrate that our approach is efficient and scalable to very large high-dimensional data sets.

Skip Supplemental Material Section

Supplemental Material

307_t_talk_8.mp4

References

  1. C. C. Aggarwal, A. Hinneburg, and D. A. Keim. On the surprising behavior of distance metrics in high dimensional spaces. In Proceedings of ICDT'01, pages 420--434, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. C. C. Aggarwal and P. S. Yu. Outlier detection for high dimensional data. In Proceedings of SIGMOD'01, pages 37--46, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. Journal of Computer and System Sciences, 58(1):137--147, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. S. D. Bay and M. Schwabacher. Mining distance-based outliers in near linear time with randomization and a simple pruning rule. In Proceedings of KDD'03, pages 29--38, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. V. Braverman, K.-M. Chung, Z. Liu, M. Mitzenmacher, and R. Ostrovsky. Ams without 4-wise independence on product domains. In Proceedings of STACS'10, pages 119--130, 2008.Google ScholarGoogle Scholar
  6. M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. Lof: Identifying density-based local outliers. In Proceedings of SIGMOD'00, pages 93--104, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. M. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of STOC'02, pages 380--388, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. D. P. Dubhashi and A. Panconesi. Concentration of Measure for the Analysis of Randomized Algorithms. Cambridge University Press, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. A. Frank and A. Asuncion. UCI machine learning repository, 2010.Google ScholarGoogle Scholar
  10. A. Ghoting, S. Parthasarathy, and M. E. Otey. Fast mining of distance-based outliers in high-dimensional datasets. Data Mining and Knowledge Discovery, 16(3):349--364, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. M. X. Goemans and D. P. Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM, 42(6):1115--1145, 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. P. Indyk and A. McGregor. Declaring independence via the sketching of sketches. In Proceedings of SODA'08, pages 737--745, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. E. M. Knorr and R. T. Ng. Algorithms for mining distance-based outliers in large datasets. In Proceedings of VLDB'98, pages 392--403, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. H.-P. Kriegel, M. Schubert, and A. Zimek. Angle-based outlier detection in high-dimensional data. In Proceedings KDD'08, pages 444--452, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. H.-P. Kriegel, M. Schubert, and A. Zimek. Outlier detection techniques. In Tutorial at KDD'10, 2010.Google ScholarGoogle Scholar
  16. V. Mahadevan, W. Li, V. Bhalodia, and N. Vasconcelos. Anomaly detection in crowded scenes. In Proceedings of CVPR'10, pages 1975--1981, 2010.Google ScholarGoogle ScholarCross RefCross Ref
  17. E. Müller, M. Schiffer, and T. Seidl. Statistical selection of relevant subspace projections for outlier ranking. In Proceedings of ICDE'11, pages 434--445, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. S. Papadimitriou, H. Kitagawa, P. B. Gibbons, and C. Faloutsos. Loci: Fast outlier detection using the local correlation integral. In Proceedings of ICDE'03, pages 315--326, 2003.Google ScholarGoogle ScholarCross RefCross Ref
  19. S. Ramaswamy, R. Rastogi, and K. Shim. Efficient algorithms for mining outliers from large data sets. In Proceedings of SIGMOD'00, pages 427--438, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Y. Wang, S. Parthasarathy, and S. Tatikonda. Locality sensitive outlier detection: A ranking driven approach. In Proceedings of ICDE'11, pages 410--421, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. R. Wheeler and J. S. Aitken. Multiple algorithms for fraud detection. Knowledge Based Systems, 13(2--3):93--99, 2000.Google ScholarGoogle Scholar

Index Terms

  1. A near-linear time approximation algorithm for angle-based outlier detection in high-dimensional data

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      KDD '12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
      August 2012
      1616 pages
      ISBN:9781450314626
      DOI:10.1145/2339530

      Copyright © 2012 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 12 August 2012

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      Overall Acceptance Rate1,133of8,635submissions,13%

      Upcoming Conference

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader