Skip to main content
SpringerLink
Log in

Fast anomaly detection in video surveillance system using robust spatiotemporal and deep learning methods

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Since from emergence of deep learning techniques, automatic video surveillance has received researchers’ attention. Such deep learning-based methods improve the accuracy but lead to higher computational complexities than the semi-automatic approach. A unique framework is proposed in this study to bridge the gap between automated and semi-automatic operations to save time and boost accuracy. The main advantage of the proposed model is reducing the computational complexity while improving the overall accuracy using the deep learning approach in video anomaly detection applications. The framework consists of keyframe extraction, robust features extraction, automatic features learning, and classification. Extracting keyframes from the input videos reduces the computational burden during features extraction and classification steps. The novel lightweight keyframe extraction algorithm using the histogram and dynamic thresholding technique is proposed in this paper. This research proposes a revolutionary lightweight keyframe extraction approach based on the histogram and dynamic thresholding technique. Efficient motion tracking among frames is a critical research challenge in video anomaly detection. We propose the novel Modified Spatio-Temporal (MST) approach to extract the interest points as features in this paper. Input frames are normalized and pre-processed using Gaussian filtering first in the proposed MST. Then motion tracking and cuboids are estimated from the pre-processed frames. From 3D cuboids, we applied Discrete Wavelet Transform (DWT) with Principal Component Analysis (PCA) to generate the codebook. This codebook passed as sequential input to Recurrent Neural Network (RNN) using Long Term Short Memory (LSTM) classifier called RNN-LSTM. The novel training and classification model of deep learning is designed to estimate the probability of each class (i.e., anomalous or normal) of the video sequence. The experimental results of the proposed MST-RNN-LSTM model achieved significant improvement in computational and detection efficiencies compared to state-of-art methods. The keyframe extraction algorithm discarded 50% of the frames in an input video sequence, resulting in less computing overhead. The accuracy of the proposed model is improved by 4.5% and the processing time is reduced by 21% compared to state-of-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The datasets analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Aköz, Ö, Karsligil, E (2014) Traffic event classification at intersections based on the severity of abnormality Machine Vision and Applications 25. https://doi.org/10.1007/s00138-011-0390-4

  2. Alhayani, B, Kwekha-Rashid, AS, Mahajan, HB et al. (2022) 5G standards for the Industry 4.0 enabled communication systems using artificial intelligence: perspective of smart healthcare system. Appl Nanosci. https://doi.org/10.1007/s13204-021-02152-4.

  3. Bermejo NE, Deniz SO, Bueno GG, Sukthankar R (2011) Violence Detection in Video Using Computer Vision Techniques. In: Real P, Diaz-Pernil D, Molina-Abril H, Berciano A, Kropatsch W (eds) Computer Analysis of Images and Patterns. CAIP 2011. Lecture notes in computer science, vol 6855. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23678-5_39

    Chapter  Google Scholar 

  4. Deepak, K, Srivathsan, G, Roshan, S, Chandrakala S (2020) Deep multi-view representation learning for video anomaly detection using spatiotemporal autoencoders. Circuits Syst Signal Process. https://doi.org/10.1007/s00034-020-01522-7

  5. Deepak K, Chandrakala S, Mohan CK (2021) Residual spatiotemporal autoencoder for unsupervised video anomaly detection. SIViP 15:215–222. https://doi.org/10.1007/s11760-020-01740-1

    Article  Google Scholar 

  6. Esen, E, Arabaci, MA, Soysal, M (2013) Fight detection in surveillance videos. 2013 11th international workshop on content-based multimedia indexing (CBMI). https://doi.org/10.1109/cbmi.2013.6576569.

  7. Hospedales, T, Li, J, Gong, S, Xiang, T. (2011) Identifying Rare and Subtle Behaviors: A Weakly Supervised Joint Topic Model IEEE Trans Pattern Anal Mach Intell 33. https://doi.org/10.1109/TPAMI.2011.81.

  8. Hu J, Zhu E, Wang S, Liu X, Guo X, Yin J (2019) An efficient and robust unsupervised anomaly detection method using ensemble random projection in surveillance videos. Sensors 19(19):4145. https://doi.org/10.3390/s19194145

    Article  Google Scholar 

  9. Huang, C, Wu, Z, Jie, W, Xu, Y, Jiang, Q, Wang, Y (2021) Abnormal event detection using deep contrastive learning for intelligent video surveillance system. IEEE Trans Indust Inf PP. 1–1. https://doi.org/10.1109/TII.2021.3122801.

  10. Jeong H, Yoo Y, Yi K, Choi J (2014) Two-stage online inference model for traffic pattern analysis and anomaly detection. Mach Vis Appl 25:1501–1517. https://doi.org/10.1007/s00138-014-0629-y

    Article  Google Scholar 

  11. Kamijo, S, Matsushita, Y, Ikeuchi, K, Sakauchi, M (n.d.). Traffic monitoring and accident detection at intersections. Proceedings 199 IEEE/IEEJ/JSAI international conference on intelligent transportation systems (cat. No.99TH8383). https://doi.org/10.1109/itsc.1999.821147.

  12. Khaire P, Kumar P (2022) A semi-supervised deep learning based video anomaly detection framework using RGB-D for surveillance of real-world critical environments. Forensic Sci Int Digit Investig 40:301346. https://doi.org/10.1016/j.fsidi.2022.301346

    Article  Google Scholar 

  13. Kotkar, V. (2018) A comparative analysis of machine learning based anomaly detection techniques in video surveillance. J Eng Appl Sci (JEAS) 12(12SI):9376–9381. https://doi.org/10.36478/jeasci.2017.9376.9381

  14. Kotkar V (2020) Scalable anomaly detection framework in video surveillance using Keyframe extraction and machine learning algorithms. J Adv Res Dyn Control Syst 12:395–408. https://doi.org/10.5373/JARDCS/V12I7/20202020

    Article  Google Scholar 

  15. Lessard, A, Bélisle, F, Bilodeau, G-A, Saunier, N (2016). The CountingApp, or How to Count Vehicles in 500 Hours of Video. https://doi.org/10.1109/CVPRW.2016.198.

  16. LI W-X, Mahadevan V, Vasconcelos N (2013) Anomaly Detection and Localization in Crowded Scenes. IEEE Trans Pattern Anal Mach Intell (TPAMI) 36:18–32. https://doi.org/10.1109/TPAMI.2013.111

    Article  Google Scholar 

  17. Liu, W, Luo, W, Lian, D, Gao, S (2018) Future Frame Prediction for Anomaly Detection - A New Baseline. 6536–6545. https://doi.org/10.1109/CVPR.2018.00684.

  18. Liu, Z, Nie, Y, Long, C, Zhang, Q, Li, G (2021) A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction. 13568–13577. https://doi.org/10.1109/ICCV48922.2021.01333.

  19. Lu, C, Shi, J, Jia, J (2013) Abnormal event detection at 150 FPS in MATLAB. 2013 IEEE international conference on computer vision. https://doi.org/10.1109/iccv.2013.338.

  20. Luo, W, Liu, W, Lian, D, Tang, J, Duan, L, Peng, X, Gao, S (2019) Video anomaly detection with sparse coding inspired deep neural networks. IEEE transactions on pattern analysis and machine intelligence, 1–1. https://doi.org/10.1109/tpami.2019.2944377.

  21. Luo, W, Liu, W, Lian, D, Gao, S (2021) Future frame prediction network for video anomaly detection. IEEE Trans Pattern Anal Mach Intell PP. 1–1. https://doi.org/10.1109/TPAMI.2021.3129349.

  22. Mabrouk A, Zagrouba E (2018) Abnormal behavior recognition for intelligent video surveillance systems: A review. Exp Syst Appl 91:480–491. https://doi.org/10.1016/j.eswa.2017.09.029

    Article  Google Scholar 

  23. Mahajan HB, Badarla A (2021) Cross-layer protocol for WSN-assisted IoT smart farming applications using nature inspired algorithm. Wirel Pers Commun 121:3125–3149. https://doi.org/10.1007/s11277-021-08866-6

    Article  Google Scholar 

  24. Mahajan HB, Badarla A, Junnarkar AA (2021) CL-IoT: cross-layer internet of things protocol for intelligent manufacturing of smart farming. J Ambient Intell Humaniz Comput 12:7777–7791. https://doi.org/10.1007/s12652-020-02502-0

    Article  Google Scholar 

  25. Mahajan, HB, Rashid, AS, Junnarkar, AA et al. (2022) Integration of Healthcare 4.0 and blockchain into secure cloud-based electronic health records systems. Appl Nanosci https://doi.org/10.1007/s13204-021-02164-0.

  26. Mehrsan JR, Levine M (2013) An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Comput Vis Image Underst 117:1436–1452. https://doi.org/10.1016/j.cviu.2013.06.007

    Article  Google Scholar 

  27. Mikhail, A, Kamil, IA, Mahajan, H (2017) Increasing SCADA System Availability by Fault Tolerance Techniques. 2017 International conference on computing, communication, Control Autom (ICCUBEA) https://doi.org/10.1109/iccubea.2017.8463911.

  28. Mikhail, A, Kareem, HH, Mahajan, H (2017) Fault Tolerance to Balance for Messaging Layers in Communication Society. 2017 International conference on computing, communication, Control and Automation (ICCUBEA) https://doi.org/10.1109/iccubea.2017.8463871.

  29. Mo X, Monga V, Bala R, Fan Z (2014) Adaptive Sparse Representations for Video Anomaly Detection. Circuits Syst Vid Technol IEEE Trans 24:631–645. https://doi.org/10.1109/TCSVT.2013.2280061

    Article  Google Scholar 

  30. Mohammadi S, Perina A, Kiani H, Murino V (2016) Angry crowds: detecting violent events in videos. Lect Notes Comput Sci:3–18. https://doi.org/10.1007/978-3-319-46478-7_1

  31. Narasimhan MG (2018) S., S.K. dynamic video anomaly detection and localization using sparse denoising autoencoders. Multimed Tools Appl 77:13173–13195. https://doi.org/10.1007/s11042-017-4940-2

    Article  Google Scholar 

  32. Nasaruddin N, Muchtar K, Afdhal A, Dwiyantoro APJ (2020) Deep anomaly detection through visual attention in surveillance videos. J Big Data 7(1). https://doi.org/10.1186/s40537-020-00365-y

  33. Nawaratne, R, Alahakoon, D, Silva, D, Yu, X (2019) Spatiotemporal anomaly detection using deep learning for real-time video surveillance. IEEE Transactions on Industrial Informatics. PP. 1–1. https://doi.org/10.1109/TII.2019.2938527.

  34. Ngo, D, Toan, D, Nguyen, L (2016) Anomaly detection in video surveillance: A novel approach based on sub-trajectory. 1–4. https://doi.org/10.1109/ELINFOCOM.2016.7562953.

  35. Patino L, Ferryman J (2014) Multiresolution semantic activity characterisation and abnormality discovery in videos. Appl Soft Comput 25:485–495. https://doi.org/10.1016/j.asoc.2014.08.039

    Article  Google Scholar 

  36. Popoola O, Wang K (2012) Video-Based Abnormal Human Behavior Recognition—A Review. IEEE Trans Syst Man Cybern Part C (Appl Rev) 42:865–878. https://doi.org/10.1109/TSMCC.2011.2178594

    Article  Google Scholar 

  37. Ramchandran A, Sangaiah AK (2020) Unsupervised deep learning system for local anomaly event detection in crowded scenes. Multimed Tools Appl 79:35275–35295. https://doi.org/10.1007/s11042-019-7702-5

    Article  Google Scholar 

  38. Shirazi S, Mohammad, Morris, B. (2016) Looking at intersections: a survey of intersection monitoring, Behavior and Safety Analysis of Recent Studies. IEEE Transactions on Intelligent Transportation Systems. PP. 1–21. https://doi.org/10.1109/TITS.2016.2568920.

  39. Sivaraman S, Trivedi M (2013) Looking at Vehicles on the Road: A Survey of Vision-Based Vehicle Detection, Tracking, and Behavior Analysis. Intell Transp Syst, IEEE Trans 14:1773–1795. https://doi.org/10.1109/TITS.2013.2266661

    Article  Google Scholar 

  40. Sodemann, A, Ross, M, Borghetti, B. (2011) A Review of Anomaly Detection in Automated Surveillance. IEEE transactions on systems, man, and cybernetics, part C. 42. https://doi.org/10.1109/TSMCC.2012.2215319.

  41. Srinivasan A, Gnanavel VK (2019) Multiple feature set with feature selection for anomaly search in videos using hybrid classification. Multimed Tools Appl 78:7713–7725. https://doi.org/10.1007/s11042-018-6348-z

    Article  Google Scholar 

  42. Sultani, W, Choi, JY (2010) Abnormal traffic detection using intelligent driver model. 2010 20th international conference on pattern recognition. https://doi.org/10.1109/icpr.2010.88.

  43. Sultani, W, Chen, C, Shah, M (2018) Real-world anomaly detection in surveillance videos. 2018 IEEE/CVF conference on computer vision and pattern recognition. https://doi.org/10.1109/cvpr.2018.00678.

  44. Wang T, Snoussi H (2014) Detection of Abnormal Visual Events via Global Optical Flow Orientation Histogram. Inf Forensic Secur IEEE Trans 9:988–998. https://doi.org/10.1109/TIFS.2014.2315971

    Article  Google Scholar 

  45. Wang, J, Wang, M, Liu, Q, Yin G, Zhang Y (2020) Deep anomaly detection in expressway based on edge computing and deep learning. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-020-02574-y

  46. Wang W, Chang F, Liu C (2022) Mutuality-oriented reconstruction and prediction hybrid network for video anomaly detection. SIViP 16:1747–1754. https://doi.org/10.1007/s11760-021-02131-w

    Article  Google Scholar 

  47. Wu, C, Shao, S, Tunc, C, Satam, P, Hariri, S (2021) An explainable and efficient deep learning framework for video anomaly detection. Cluster computing, 1–23. Advance online publication. https://doi.org/10.1007/s10586-021-03439-5

  48. Xue Z, Wu W (2020) Anomaly detection by exploiting the tracking trajectory in surveillance videos. SCIENCE CHINA Inf Sci 63:154101. https://doi.org/10.1007/s11432-018-9792-8

    Article  Google Scholar 

  49. Yang B, Cao J, Ni R, Zou L (2018) Anomaly detection in moving crowds through spatiotemporal autoencoding and additional attention. Adv Multimed 2018:1–8. https://doi.org/10.1155/2018/2087574

    Article  Google Scholar 

  50. Yoa, S, Lee, S, Kim, C, Kim, H (2021) Self-Supervised Learning for Anomaly Detection With Dynamic Local Augmentation. IEEE Access. PP. 1–1. https://doi.org/10.1109/ACCESS.2021.3124525.

  51. Yu B, Liu Y, Sun Q (2017) A content-adaptively sparse reconstruction method for abnormal events detection with low-rank property. IEEE Trans Syst Man Cybern Syst 47(4):704–716

    Article  Google Scholar 

  52. Yun, K, Jeong, H, Yi, K, Kim, S, Choi, J (2014) Motion Interaction Field for Accident Detection in Traffic Surveillance Video. Proceed Int Conf Pattern Recognit 3062–3067. https://doi.org/10.1109/ICPR.2014.528.

  53. Zhang, Y, Lu, H, Zhang, L, Ruan, X (2015) Combining motion and appearance cues for anomaly detection. Pattern Recogn 51. https://doi.org/10.1016/j.patcog.2015.09.005

  54. Zhang, Q, Feng, G, Wu, H (2022) Surveillance video anomaly detection via non-local U-net frame prediction. Multimed Tools Appl. https://doi.org/10.1007/s11042-021-11550-3

Download references

Funding

No Funding.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, PCET’s Pimpri Chinchwad College of Engineering & Research, Pune, Maharashtra, India

    Vijay A. Kotkar & V. Sucharita

  2. Department of Computer Science and Engineering, Narayana College of Engineering, Gudur, India

    Vijay A. Kotkar

Authors
  1. Vijay A. Kotkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. V. Sucharita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vijay A. Kotkar.

Ethics declarations

Conflict of interest

All authors declares that they has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kotkar, V.A., Sucharita, V. Fast anomaly detection in video surveillance system using robust spatiotemporal and deep learning methods. Multimed Tools Appl 82, 34259–34286 (2023). https://doi.org/10.1007/s11042-023-14840-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14840-0

Keywords

Search

Navigation