Skip to main content
SpringerLink
Log in

A pattern-based outlier region detection method for two-dimensional arrays

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Recently, with the prevalence of various sensing devices and numerical simulation software, a large amount of data is being generated in the form of a two-dimensional (2D) array. One of the important tasks for analyzing such arrays is to find anomalous or outlier regions in such a 2D array. In this article, we propose an effective method for detecting outlier regions in an arbitrary 2D array, which show a significantly different pattern from that of their surrounding regions. Unlike most existing methods that determine the outlierness of a region based on how different its average is from that of its neighboring elements, our method exploits the regression models of a region in determining its outlierness. More specifically, this method first divides the array into a number of small subarrays and then builds a regression model for each subarray. In turn, the method iteratively merges adjacent subarrays with similar regression models into larger clusters. After the clustering, the proposed method reports very small clusters as outlier regions at the final step. Lastly, we demonstrate in our experiments the effectiveness of the proposed method on synthetic and real datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Amidan BG, Ferryman TA, Cooley SK (2005) Data outlier detection using the Chebyshev theorem. In: Proceedings of 2005 IEEE Aerospace Conference. IEEE, pp 3814–3819

  2. Bouguettaya A, Yu Q, Liu X, Zhou X, Song A (2015) Efficient agglomerative hierarchical clustering. Expert Syst Appl 42(5):2785–2797

    Article  Google Scholar 

  3. Chawla M, Sharma S, Sivaswamy J, Kishore LA (2009) Method for automatic detection and classification of stroke from brain CT images. In: Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, pp 3581–3584

  4. Chawla S, Sun P (2006) SLOM: a new measure for local spatial outliers. Knowl Inf Syst 9(4):412–429

    Article  Google Scholar 

  5. Chen CCY, Das SK (1992) Breadth-first traversal of trees and integer sorting in parallel. Inf Proces Lett 41(1):39–49

    Article  MathSciNet  MATH  Google Scholar 

  6. Franke C, Gertz M (2009) ORDEN: outlier region detection and exploration in sensor networks. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of data. ACM, pp 1075–1078

  7. Friedman JH, Fisher NI (1999) Bump hunting in high-dimensional data. Stat Comput 9(2):123–143

    Article  Google Scholar 

  8. Janeja VP, Atluri V (2008) Random walks to identify anomalous free-form spatial scan windows. IEEE Trans Knowl Data Eng 20(10):1378–1392

    Article  Google Scholar 

  9. Jim J, Lee W, Song JJ, Lee SB (2017) Optimized combinatorial clustering for stochastic processes. Clust Comput 20(2):1135–1148

    Article  Google Scholar 

  10. Kulldorff M (1997) A spatial scan statistic. Commun Stat Theory Methods 26(6):1481–1496

    Article  MathSciNet  MATH  Google Scholar 

  11. Kulldorff M, Huang L, Pickle L, Duczmal L (2006) An elliptic spatial scan statistic. Stat Med 25(22):3929–3943

    Article  MathSciNet  Google Scholar 

  12. Kutner MH, Nachtsheim CJ, Neter J, Li W (2005) Applied linear statistical models, 5th edn. McGraw Hill, New York

    Google Scholar 

  13. Langfelder P, Zhang B, Horvath S (2007) Defining clusters from a hierarchical cluster tree: the dynamic tree cut package for R. Bioinformatics 24(5):719–720

    Article  Google Scholar 

  14. Li X, Lv J, Yi Z (2018) An efficient representation-based method for boundary point and outlier detection. IEEE Trans Neural Netw Learn Syst 29(1):51–62

    Article  MathSciNet  Google Scholar 

  15. Lu CT, Kou Y, Zhao J, Chen L (2007) Detecting and tracking regional outliers in meteorological data. Inf Sci 177(7):1609–1632

    Article  Google Scholar 

  16. Lu CT, SANTOS JR RFD, Liu X, Kou Y (2011) A graph-based approach to detect abnormal spatial points and regions. Int J Artif Intell Tools 20(04):721–751

    Article  Google Scholar 

  17. Luts J, Laudadio T, Idema AJ, Simonetti AW, Heerschap A, Vandermeulen D, Suykens JA, Van Huffel S (2009) Nosologic imaging of the brain: segmentation and classification using MRI and MRSI. NMR Biomed 22(4):374–390

    Article  Google Scholar 

  18. Mao J, Wang T, Jin C, Zhou A (2017) Feature grouping-based outlier detection upon streaming trajectories. IEEE Trans Knowl Data Eng 29(12):2696–2709

    Article  Google Scholar 

  19. NASA (2010): New map offers a global view of health-sapping air pollution. https://www.nasa.gov/topics/earth/features/health-sapping.html. Accessed 29 Jan 2018

  20. NASA (2018): NASA earth observation. https://neo.sci.gsfc.nasa.gov/. Accessed 29 Jan 2018

  21. Neill DB, Moore AW, Cooper GF (2006) A Bayesian spatial scan statistic. In: Weiss Y, Schölkopf B, Platt JC (eds) Advances in Neural Information Processing Systems 18 (NIPS 2005). Neural Information Processing Systems Foundation, Inc, pp 1003–1010

  22. Patil GP, Taillie C (2004) Upper level set scan statistic for detecting arbitrarily shaped hotspots. Environ Ecol Stat 11(2):183–197

    Article  MathSciNet  Google Scholar 

  23. Prastawa M, Bullitt E, Ho S, Gerig G (2004) A brain tumor segmentation framework based on outlier detection. Med Image Anal 8(3):275–283

    Article  Google Scholar 

  24. Ramteke R, Monali YK (2012) Automatic Medical image classification and abnormality detection using K-nearest neighbour. Int J Adv Comput Res 2(4):190–196

    Google Scholar 

  25. Reed IS, Yu X (1990) Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution. IEEE Trans Acoust Speech Signal Proces 38(10):1760–1770

    Article  Google Scholar 

  26. Song JJ, Lee W (2017) Relevance maximization for high-recall retrieval problem: finding all needles in a Haystack. J Supercompu. https://doi.org/10.1007/s11227-016-1956-8

    Google Scholar 

  27. Stein DW, Beaven SG, Hoff LE, Winter EM, Schaum AP, Stocker AD (2002) Anomaly detection from hyperspectral imagery. IEEE Signal Proces Mag 19(1):58–69

    Article  Google Scholar 

  28. Suhail Z, Sarwar M, Murtaza K (2015) Automatic detection of abnormalities in mammograms. BMC Med Imaging 15(1):53

    Article  Google Scholar 

  29. Tango T, Takahashi K (2005) A flexibly shaped spatial scan statistic for detecting clusters. Int J Health Geogr 4(1):11

    Article  Google Scholar 

  30. Telang A, Deepak P, Joshi S, Deshpande P, Rajendran R (2014) Detecting localized homogeneous anomalies over spatio-temporal data. Data Mining Knowl Discov 28(5–6):1480–1502

    Article  MathSciNet  Google Scholar 

  31. Tran L, Fan L, Shahabi C (2016) Distance-based outlier detection in data streams. Proc VLDB Endow 9(12):1089–1100

    Article  Google Scholar 

  32. You C, Robinson DP, Vidal R (2017) Provable self-representation based outlier detection in a union of subspaces. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp 1–10

  33. Zheng G, Brantley SL, Lauvaux T, Li Z (2017) Contextual spatial outlier detection with metric learning. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 2161–2170

  34. Zhu M, Aggarwal CC, Ma S, Zhang H, Huai J (2017) Outlier detection in sparse data with factorization machines. In: Proceedings of the 2017 ACM Conference on Information and Knowledge Management, pp 817–826

Download references

Acknowledgements

We would like to thank anonymous reviewers for their insightful comments to improve the quality of this article. We also give thanks to Sang-Un Gu for locating and preparing for the real data sets.

Author information

Authors and Affiliations

  1. Division of Computer Science, Sookmyung Women’s University, Seoul, Korea

    Ki Yong Lee

  2. School of Computer Science and Engineering, Kyungpook National University, Daegu, Korea

    Young-Kyoon Suh

Authors
  1. Ki Yong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Young-Kyoon Suh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Young-Kyoon Suh.

Additional information

This manuscript is an extended version of our paper to appear in Big Data Applications and Services 2017, Advances in Intelligent Systems and Computing 770, Springer Nature Singapore Pte Ltd. 2019 (https://doi.org/10.1007/978-981-13-0695-2_5).

This research was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korea government (MSIP) (No. 2015R1C1A1A02037071).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, K.Y., Suh, YK. A pattern-based outlier region detection method for two-dimensional arrays. J Supercomput 75, 170–188 (2019). https://doi.org/10.1007/s11227-018-2418-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-018-2418-2

Keywords

Search

Navigation