Skip to main content
SpringerLink
Log in

Anomaly detection based on superpixels in videos

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Based on superpixels, we propose a novel method for detecting abnormal events in videos. The conventional methods divide the frames into regular grids and consider the grids with low probability as abnormal events. By contrast with traditional approaches, we divide frames into superpixels according to their similarity and compactness, and the superpixels under the scene model mask are used as the anomaly candidates. The anomaly detection is carried out at two scales: the basic grids covered by the superpixel candidates and the actual superpixel itself. Anomaly scores are calculated by comparing the test samples with corresponding templates. Experiments on the public databases show that our method can effectively detect abnormal events in complex scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. We use grid to represent grid cube and superpixel for superpixel cube in the following.

  2. The results have been retained with two significant digits.

References

  1. Huang Q, Zhou C, Wu J, Wang M, Wang B (2019) Deep structure learning for rumor detection on twitter. In: International joint conference on neural networks, IJCNN 2019 Budapest, Hungary, July 14-19, 2019, pp 1–8

  2. Wang H, Wu J, Hu W, Wu X (2019) Detecting and assessing anomalous evolutionary behaviors of nodes in evolving social networks. ACM Trans Knowl Discov Data 13(1):12:1-12:24

    Article  Google Scholar 

  3. Bergmann P, Fauser M, Sattlegger D, Steger C (2020) Uninformed students: student-teacher anomaly detection with discriminative latent embeddings. In: IEEE conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp 4182–4191

  4. Basharat A, Gritai A, Shah M (2008) Learning object motion patterns for anomaly detection and improved object detection. In: IEEE computer society conference on computer vision and pattern recognition CVPR 2008, 24-26 June 2008, Anchorage, Alaska, USA

  5. Kratz L, Nishino K (2009) Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: IEEE computer society conference on computer vision and pattern recognition, CVPR 2009, 20-25 June 2009, Miami, Florida, USA, pp 1446–1453

  6. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In: IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp 733–742

  7. Ionescu RT, Smeureanu S, Alexe B, Popescu M (2017) Unmasking the abnormal events in video. In: IEEE International conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp 2914–2922

  8. Sodemann AA, Ross MP, Borghetti BJ (2012) A review of anomaly detection in automated surveillance. IEEE Trans Syst Man Cybern Part C 42(6):1257–1272

    Article  Google Scholar 

  9. Gandhi T, Trivedi MM (2007) Pedestrian protection systems: Issues, survey, and challenges. IEEE Trans Intell Transp Syst 8(3):413–430

    Article  Google Scholar 

  10. Shao L, Ji L, Liu Y, Zhang J (2012) Human action segmentation and recognition via motion and shape analysis. Pattern Recognit Lett 33(4):438–445

    Article  Google Scholar 

  11. Zhao B, Li F, Xing EP (2011) Online detection of unusual events in videos via dynamic sparse coding. In: IEEE conference on computer vision and pattern recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp 3313–3320

  12. Cong Y, Yuan J, Liu J (2011) Sparse reconstruction cost for abnormal event detection. In: IEEE conference on computer vision and pattern recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp 3449–3456

  13. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 FPS in MATLAB. In: IEEE International conference on computer vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, pp 2720–2727

  14. Mo X, Monga V, Bala R, Fan Z (2014) Adaptive sparse representations for video anomaly detection. IEEE Trans Circuits Syst Video Techn 24(4):631–645

    Article  Google Scholar 

  15. He C, Shao J, Sun J (2018) An anomaly-introduced learning method for abnormal event detection. Multim Tools Appl 77(22):29573–29588

    Article  Google Scholar 

  16. Sultani W, Chen C, Shah M (2018) Real-world anomaly detection in surveillance videos. In: IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp 6479–6488

  17. Liu W, Luo W, Li Z, Zhao P, Gao S (2019) Kraus, S. (ed.) Margin learning embedded prediction for video anomaly detection with A few anomalies. (ed.Kraus, S.) In: Proceedings of the twenty-eighth international joint conference on artificial intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp 3023–3030

  18. Ionescu RT, Khan FS, Georgescu M, Shao L (2019) Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In: IEEE conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp 7842–7851

  19. Nguyen T, Meunier J (2019) Anomaly detection in video sequence with appearance-motion correspondence. In: IEEE international conference on computer vision, ICCV 2019, Seoul, Korea (South), October 27-November 2, 2019, pp 1273–1283

  20. Pang G, Shen C, van den Hengel A (2019) Deep anomaly detection with deviation networks. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining pp. 353-362

  21. Zhou JT et al (2019) Anomalynet: An anomaly detection network for video surveillance. IEEE Trans Inf Forensics Secur 14(10):2537–2550

    Article  Google Scholar 

  22. Hu X, Huang Y, Gao X, Luo L, Duan Q (2019) Squirrel-cage local binary pattern and its application in video anomaly detection. IEEE Trans Inf Forensics Secur 14(4):1007–1022

    Article  Google Scholar 

  23. Giorno AD, Bagnell JA, Hebert M, Leibe, B, Matas, J, Sebe N & Welling M (2016) (eds) A discriminative framework for anomaly detection in large videos. (eds Leibe, B., Matas, J., Sebe, N. & Welling, M.) Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V, Vol. 9909 of Lecture Notes in Computer Science, 334–349

  24. Cong Y, Yuan J, Tang Y (2013) Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Trans Inf Forensics Secur 8(10):1590–1599

    Article  Google Scholar 

  25. Xu J, Denman S, Sridharan S, Fookes C, Rana R (2011) Dynamic texture reconstruction from sparse codes for unusual event detection in crowded scenes, J-MRE ’11, 25–30

  26. Tran H, Hogg DC (2017) Anomaly detection using a convolutional winner-take-all autoencoder. In: British machine vision conference, BMVC 2017, London, UK, September 4-7, 2017

  27. Pappas OA, Achim AM, Bull DR (2015) Superpixel-based statistical anomaly detection for sense and avoid. In: IEEE international conference on image processing, ICIP 2015, Quebec City, QC, Canada, September 27-30, 2015, pp 2229–2233

  28. Huang Z, Li S, Fang L (2018) Fusing information from subpixel to superpixel for hyperspectral anomaly detection. In: IEEE international geoscience and remote sensing symposium, IGARSS 2018, Valencia, Spain, July 22-27, 2018, pp 1260–1263

  29. Li Z et al (2020) Superpixel masking and inpainting for self-supervised anomaly detection. BMVA Press, UK

    Google Scholar 

  30. Johnson N, Hogg DC (1996) Learning the distribution of object trajectories for event recognition. Image Vis Comput 14(8):609–615

    Article  Google Scholar 

  31. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: IEEE conference on computer vision and pattern recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp 1975–1981

  32. Li A et al (2020) Abnormal event detection in surveillance videos based on low-rank and compact coefficient dictionary learning. Pattern Recognit 108:107355

    Article  Google Scholar 

  33. Kamoona AM, Gostar AK, Bab-Hadiashar A, Hoseinnezhad R (2020) Multiple instance-based video anomaly detection using deep temporal encoding-decoding. CoRRarXiv:2007.01548

  34. Chatfield K, Simonyan K, Vedaldi A, Zisserman A (2014) Valstar, M. F., French, A. P. & Pridmore, T. P. (eds) Return of the devil in the details: Delving deep into convolutional nets. (eds Valstar, M. F., French, A. P. & Pridmore, T. P.) British Machine Vision Conference, BMVC 2014, Nottingham, UK, September 1-5, 2014

  35. Chong YS, Tay YH (2017) Cong, F., Leung, A. C. & Wei, Q. (eds) Abnormal event detection in videos using spatiotemporal autoencoder. (eds Cong, F., Leung, A. C. & Wei, Q.) Advances in Neural Networks - ISNN 2017 - 14th International Symposium, ISNN 2017, Sapporo, Hakodate, and Muroran, Hokkaido, Japan, June 21-26, 2017, Proceedings, Part II, Vol. 10262 of Lecture Notes in Computer Science, 189–196 (Springer, 2017)

  36. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked RNN framework. In: IEEE international conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp 341–349

  37. Wang S, Zhu E, Yin J, Porikli F (2016) Anomaly detection in crowded scenes by SL-HOF descriptor and foreground classification. In: International conference on pattern recognition, ICPR 2016, pp 3398–3403

  38. Nayak R, Pati UC, Das SK (2021) A comprehensive review on deep learning-based methods for video anomaly detection. Image Vis Comput 106:104078

    Article  Google Scholar 

  39. Ma X et al (2021) A comprehensive survey on graph anomaly detection with deep learning. CoRRarXiv:2106.07178

  40. Ren X, Malik J (2003) Learning a classification model for segmentation. In: IEEE international conference on computer vision ICCV 2003, 14-17 October 2003, Nice, France, pp 10–17

  41. Ning J, Zhang L, Zhang D, Wu C (2010) Interactive image segmentation by maximal similarity based region merging. Pattern Recognit. 43(2):445–456

    Article  Google Scholar 

  42. Wang S, Lu H, Yang F, Yang M Metaxas, D N, Quan, L, Sanfeliu A, & Gool LV (2011) (eds) Superpixel tracking. (eds Metaxas, D. N., Quan, L., Sanfeliu, A. & Gool, L. V.) In: IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, 1323–1330 (IEEE Computer Society)

  43. Perazzi F, Krähenbühl P, Pritch Y, Hornung A (2012) Saliency filters: contrast based filtering for salient region detection. In: IEEE conference on computer vision and pattern recognition, Providence, RI, USA, June 16-21, 2012, pp 733–740

  44. Tong N, Lu H, Zhang L, Ruan X (2014) Saliency detection with multi-scale superpixels. IEEE Signal Process Lett 21(9):1035–1039

    Article  Google Scholar 

  45. Ghadiri F, Bergevin R, Bilodeau G (2019) From superpixel to human shape modelling for carried object detection. Pattern Recognit. 89:134–150

    Article  Google Scholar 

  46. Farnebäck G (2003) Bigün, J & Gustavsson, T (2003) (eds) Two-frame motion estimation based on polynomial expansion. (eds Bigün, J. & Gustavsson, T.) Image analysis, 13th scandinavian conference, SCIA 2003, Halmstad, Sweden, June 29 - July 2, 2003, Proceedings, Vol. 2749 of Lecture Notes in Computer Science, 363–370

  47. Otsu N (2007) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66

    Article  MathSciNet  Google Scholar 

  48. Li S, Yang Y, Liu C (2018) Anomaly detection based on two global grid motion templates. Signal Process Image Commun 60:6–12

    Article  Google Scholar 

  49. Felzenszwalb PF, Huttenlocher DP (2004) Efficient graph-based image segmentation. Int J Comput Vis 59(2):167–181

    Article  Google Scholar 

  50. Hu Y, Zhang Y, Davis LS (2013) Unsupervised abnormal crowd activity detection using semiparametric scan statistic, pp 767–774

  51. Zhang Y, Lu H, Zhang L, Ruan X (2016) Combining motion and appearance cues for anomaly detection. Pattern Recogn 51:443–452

    Article  Google Scholar 

  52. Vu H, Nguyen TD, Le T, Luo W, Phung DQ (2019) Robust anomaly detection in videos using multilevel representations. In: The thirty-third AAAI conference on artificial intelligence, AAAI 2019, the thirty-first innovative applications of artificial intelligence conference, IAAI 2019, the ninth AAAI symposium on educational advances in artificial intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27-February 1, 2019, pp 5216–5223

  53. xin Luo W, Liu W, Gao S (2017) Remembering history with convolutional LSTM for anomaly detection. In: IEEE international conference on multimedia and expo, ICME 2017, Hong Kong, China, July 10-14, 2017, pp 439–444

  54. Achanta R et al (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282

    Article  Google Scholar 

Download references

Acknowledgements

This work was jointly supported by the National Natural Science Foundation of China (61402049), Science and Technology Research Project of the Department of Education of Liaoning Province (LJKZ1019) and Social Science Planning Fund of Liaoning Province (L21BGL002).

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Bohai University, Science and Technology Road, Jinzhou, 121000, Liaoning, China

    Shifeng Li, Yan Cheng, Ye Tian & Yunfeng Liu

Authors
  1. Shifeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yan Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ye Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunfeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shifeng Li.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Cheng, Y., Tian, Y. et al. Anomaly detection based on superpixels in videos. Neural Comput & Applic 34, 12617–12631 (2022). https://doi.org/10.1007/s00521-022-07120-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07120-9

Keywords

Search

Navigation