Skip to main content

Advertisement

SpringerLink
Log in

A two-stream abnormal detection using a cascade of extreme learning machines and stacked auto encoder

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Identifying anomalous activity is a heavy task, and this has led to the progression in the domain of deep learning for video surveillance. With the development of deep learning, anomaly detection techniques have been widely used to improve the performance of various applications, including vision detection systems. However, it is still difficult to apply them directly to practical applications which usually involve the lack of abnormal samples and diversity. This paper proposes a novel Stacked Auto Encoder (SAE) and Extreme Learning Machine (ELM) abnormal detection framework based on multiples features. These features are connected to speed of movement and appearance and fed to a new neural network architecture as temporal and spatiotemporal streams. The use of ELM algorithms with an exceptionally fast learning speed when dealing with abnormal activity localization problems in addition to excellent generalization abilities, a deep learning network achieves a good performance with quick learning speed to further improve the regression performance. The strength of our proposed approaches is demonstrated by experiments with measured abnormal activities’ data. This approach can accurately identify and precisely locate abnormal events.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Algorithm 2
Fig. 7
Algorithm 3
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Data availability

All the web access links for the datasets and the machine learning tools used were indicated in ‘References’. Some pseudocodes were presented in this paper. Any questions about implementation details could be addressed to the corresponding author.

References

  1. Abdullah F, Gochoo M, Jalal A (2021) Multi-person tracking and crowd behavior detection via particles gradient motion descriptor and improved entropy classifier. Entropy 23:628

    Article  Google Scholar 

  2. Avenue Dataset. Detection of unusual crowd activity. http://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/dataset.html (Accessed Sept 28, 2022)

  3. Aziz S, Youssef F (2018) Traffic sign recognition based on multi-feature fusion and ELM classifier. Proc Comput Sci 127:146–153

    Article  Google Scholar 

  4. Berlin SJ, John M (2020) Spiking neural network based on joint entropy of optical flow features for human action recognition. The visual computer 1–15

  5. Bilinski P, Bremond F (2016) Human violence recognition and detection in surveillance videos. In 2016 13th IEEE international conference on advanced video and signal based surveillance (AVSS), pp 30–36. IEEE

  6. Chidananda K, Siva Kumar AP (2022) Human anomaly detection in surveillance videos: a review. Information and Communication Technology for Competitive Strategies (ICTCS 2020), 791–802

  7. Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal autoencoder. In International symposium on neural networks, pp 189–196. Springer, Cham

  8. Coşar S, Donatiello G, Bogorny V, Garate C, Alvares LO, Brémond F (2017) Toward abnormal trajectory and event detection in video surveillance. IEEE Trans Circ Syst Video Technol 27(3):683–695

    Article  Google Scholar 

  9. Cui X, Liu Q, Gao M, Metaxas DN (2011) Abnormal detection using interaction energy potentials. In: CVPR 2011, pp 3161–3167

  10. Ege Can Ö (2020) Two-stage sparse representation based abnormal crowd event detection in videos

  11. George M, Jose BR, Mathew J, Kokare P (2019) Autoencoder-based abnormal activity detection using parallelepiped spatio-temporal region. IET Comput Vis 13(1):23–30

    Article  Google Scholar 

  12. George M, Jose BR, Mathew J, Kokare P (2019) Autoencoder-based abnormal activity detection using parallelepiped spatiotemporal region. IET Comput Vis 13(1):23–30

    Article  Google Scholar 

  13. Gnouma M, Ejbali R, Zaied M (2017) Human fall detection based on block matching and silhouette area. In ninth international conference on machine vision (ICMV 2016) (Vol 10341 pp 18–22). SPIE

  14. Gnouma M, Ejbali R, Zaied M (2019) Video anomaly detection and localization in crowded scenes. In International joint conference: 12th international conference on computational intelligence in security for information systems (CISIS), pp 87–96. Springer, Cham

  15. Gnouma M, Ladjailia A, Ejbali R, Zaied M (2019) Stacked sparse autoencoder and history of binary motion image for human activity recognition. Multimed Tools Appl 78(2):2157–2179

    Article  Google Scholar 

  16. Gong D, Liu L, Le V, Saha B, Mansour MR, Venkatesh S, Hengel AVD (2019) Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In proceedings of the IEEE/CVF international conference on computer vision, pp 1705–1714

  17. Guangli WU, Liping LIU, Chen Z, Dengtai TAN (2019) Video abnormal event detection based on ELM. In 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), pp 367–371. IEEE

  18. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742

  19. Hassner T, Itcher Y, Kliper-Gross O (2012) Violent flows: real-time detection of violent crowd behavior. In 2012 IEEE computer society conference on computer vision and pattern recognition workshops (pp 1–6). IEEE

  20. Huang GB, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122

    Article  Google Scholar 

  21. Kantorov V, Laptev I (2014) Efficient feature extraction, encoding and classification for action recognition. In proceedings of the IEEE conference on computer vision and pattern recognition, pp 2593–2600

  22. Labbedi S, Gnouma M, Ejbali R, Zaied M (2021) Violent scenes detection based on connected component analysis. In thirteenth international conference on machine vision (Vol 11605, p 1160527). International Society for Optics and Photonics

  23. Lee S, Kim HG, Ro YM (2019) BMAN: bidirectional multi-scale aggregation networks for abnormal event detection. IEEE Trans Image Process 29:2395–2408

    Article  MATH  Google Scholar 

  24. Li N, Chang F (2019) Video anomaly detection and localization via multivariate gaussian fully convolution adversarial autoencoder. Neurocomputing 369:92–105

    Article  Google Scholar 

  25. Li T, Chen X, Zhu F, Zhang Z, Yan H (2021) Two-stream deep spatial-temporal auto-encoder for surveillance video abnormal event detection. Neurocomputing 439:256–270

    Article  Google Scholar 

  26. Li W, Mahadevan V, Vasconcelos N (2013) Anomaly detection and localization in crowded scenes. IEEE Trans Pattern Anal Mach Intell 36(1):18–32

    Google Scholar 

  27. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection - a new baseline. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 6536–6545

  28. Lloyd K, Rosin PL, Marshall D, Moore SC (2017) Detecting violent and abnormal crowd activity using temporal analysis of grey level co-occurrence matrix (GLCM)-based texture measures. Mach Vis Appl 28(3):361–371

    Article  Google Scholar 

  29. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In proceedings of the IEEE international conference on computer vision pp 2720–2727

  30. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp 1975–1981. IEEE

  31. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In 2009 IEEE conference on computer vision and pattern recognition (pp 935–942). IEEE

  32. Mirmahboub B, Samavi S, Karimi N, Shirani S (2012) Automatic monocular system for human fall detection based on variations in silhouette area. IEEE Trans Biomed Eng 60(2):427–436

    Article  Google Scholar 

  33. Mishra P, Varadharajan V, Tupakula U, Pilli ES (2018) A detailed investigation and analysis of using machine learning techniques for intrusion detection. IEEE Commun Surv Tutor 21(1):686–728

    Article  Google Scholar 

  34. Nguyen TN, Meunier J (2019) Anomaly detection in video sequence with appearance-motion correspondence. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 1273–1283

  35. Primartha R, Tama BA (2017) Anomaly detection using random forest: a performance revisited. In 2017 international conference on data and software engineering (ICoDSE), pp 1–6. IEEE

  36. Rayi VK, Mishra SP, Naik J, Dash PK (2021) Adaptive VMD based optimized deep learning mixed kernel ELM autoencoder for single and multistep wind power forecasting Energy 122585

  37. Sezer ES, Can AB (2018) Anomaly detection in crowded scenes using log-Euclidean covariance matrix. In VISIGRAPP (4: VISAPP), pp 279–286

  38. Sudhakaran S, Lanz O (2017) Learning to detect violent videos using convolutional long short-term memory. In 2017 14th IEEE international conference on advanced video and signal based surveillance (AVSS), pp 1-6. IEEE

  39. Sun K, Zhang J, Zhang C, Hu J (2017) Generalized extreme learning machine autoencoder and a new deep neural network. Neurocomputing 230:374–381

    Article  Google Scholar 

  40. Tudor Ionescu R, Smeureanu S, Alexe B, Popescu M (2017) Unmasking the abnormal events in video. In proceedings of the IEEE international conference on computer vision, pp 2895–2903

  41. Vishwakarma DK, Dhiman C (2019) A unified model for human activity recognition using spatial distribution of gradients and difference of Gaussian kernel. Vis Comput 35(11):1595–1613

    Article  Google Scholar 

  42. Wang L, Qiao Y, Tang X (2015) Action recognition with trajectory-pooled deep-convolutional descriptors. In proceedings of the IEEE conference on computer vision and pattern recognition, pp 4305–4314

  43. Wang J, Xu Z (2015) Crowd anomaly detection for automated video surveillance, IET

  44. Wang S, Zhu E, Yin J, Porikli F (2018) Video anomaly detection and localization by local motion based joint video representation and OCELM. Neurocomputing 277:161–175

    Article  Google Scholar 

  45. Wolpert DH (1992) Stacked generalization. Neural Netw 5(2):241–259

    Article  Google Scholar 

  46. Xu D, Ricci E, Yan Y, Song J, Sebe N (2015) Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553

  47. Xu D, Yan Y, Ricci E, Sebe N (2017) Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput Vis Image Underst 156:117–127

    Article  Google Scholar 

  48. Yahia S, Said S, Zaied M (2020) A novel classification approach based on extreme learning machine and wavelet neural networks. Multimed Tools Appl 79(19):13869–13890

    Article  Google Scholar 

  49. Yahia S, Said S, Zaied M (2022) Wavelet extreme learning machine and deep learning for data classification. Neurocomputing 470:280–289

    Article  Google Scholar 

  50. Yuan Y, Feng Y, Lu X (2018) Structured dictionary learning for abnormal event detection in crowded scenes. Pattern Recogn 73:99–110

    Article  Google Scholar 

  51. Zhang Y, Dong L, Li S, Li J (2014) Abnormal crowd behavior detection using interest points. In 2014 IEEE international symposium on broadband multimedia systems and broadcasting, pp 1–4. IEEE

  52. Zhang T, Jia W, Yang B, Yang J, He X, Zheng Z (2017) MoWLD: a robust motion image descriptor for violence detection. Multimed Tools Appl 76(1):1419–1438

    Article  Google Scholar 

  53. Zhou JT, Du J, Zhu H, Peng X, Liu Y, Goh RSM (2019) Anomalynet: an anomaly detection network for video surveillance. IEEE Trans Inf Forensics Secur 14(10):2537–2550

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the financial support of this work by grants from General Direction of Scientific Research (DGRST), Tunisia, under the ARUB program.

Author information

Authors and Affiliations

  1. Research Team in Intelligent Machines, National School of Engineers of Gabes, University of Gabes, Gabes, Tunisia

    Mariem Gnouma, Ridha Ejbali & Mourad Zaied

  2. Department of Computer Sciences, Faculty of Sciences of Gabes, 6072, Erriadh City, Zrig, Gabes, Tunisia

    Mariem Gnouma & Ridha Ejbali

Authors
  1. Mariem Gnouma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ridha Ejbali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mourad Zaied
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mariem Gnouma.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gnouma, M., Ejbali, R. & Zaied, M. A two-stream abnormal detection using a cascade of extreme learning machines and stacked auto encoder. Multimed Tools Appl 82, 38743–38770 (2023). https://doi.org/10.1007/s11042-023-15060-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15060-2

Keywords

Search

Navigation