Skip to main content
SpringerLink
Log in

Efficient abnormality detection using patch-based 3D convolution with recurrent model

  • ORIGINAL PAPER
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Recent advances in the intelligence video monitoring system have received widespread attention for the detection of anomalous human behavior in crowded scenes. Due to the varying crowd densities, low-resolution videos, inter-object occlusions, and complex human crowds, the detection of abnormalities from human activities is extremely challenging. Hence, automatic analysis of behavioral patterns is necessary for accurately modeling crowd behavior and alerting human operators about suspicious activities on the scene. In response to these concerns, we propose a two-stream multi-scale patch-based pyramidal dilated 3D fully connected network (FCN) with attentive bidirectional long short-term memory (2MPD-3DFCN-AttBiDLSTM) for detecting and locating abnormal activities in the frame. This model effectively captures the spatial–temporal features with a dilated convolution network, and thus the motion and optical flow information features are exploited from the continuous frame, which improves the detection accuracy. Also, we introduce a parallel weighted skip connection into the residual learning framework that preserves the rich characteristics of the input data to be learned without the loss of effective features. Based on the attentive mechanism in the bidirectional LSTM model, two directions of temporal and global representations are extracted that enhance the classification of unusual and normal activity in the visual sequences. Experimental analysis is performed with the two publicly available datasets and evaluated in terms of the equal error rate, precision–recall curve, receiver operating characteristic curve, and area under the curve metrics measures. The result shows that the proposed model outperforms the existing model and achieves high detection results in the video surveillance monitoring system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Yao, F.: Deep learning analysis of human behaviour recognition based on convolutional neural network analysis. Behav. Inf. Technol. 27, 1–9 (2020)

    Google Scholar 

  2. Popoola, O.P., Wang, K.: Video-based abnormal human behavior recognition—a review. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 42(6), 865–878 (2012)

    Article  Google Scholar 

  3. Zhou, S., He, J., Yang, H., Chen, D., Zhang, R.: Big data-driven abnormal behavior detection in healthcare based on association rules. IEEE Access. 8, 129002–129011 (2020)

    Article  Google Scholar 

  4. Zhao, Z.Q., Zheng, P., Xu, S.T., Wu, X.: Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3212–3232 (2019)

    Article  Google Scholar 

  5. Fan, H., Luo, C., Zeng, C., Ferianc, M., Que, Z., Liu, S., Niu, X., Luk, W.: F-E3D: FPGA-based acceleration of an efficient 3D convolutional neural network for human action recognition. In: 2019 IEEE 30th International Conference on Application-specific Systems, Architectures and Processors (ASAP), Vol. 2160, pp. 1–8. IEEE (2019)

  6. Shen, Y., Ji, R., Wang, C., Li, X., Li, X.: Weakly supervised object detection via object-specific pixel gradient. IEEE Trans. Neural Netw. Learn. Syst. 29(12), 5960–5970 (2018)

    Article  Google Scholar 

  7. Zhang, J., Wu, C., Wang, Y., Wang, P.: Detection of abnormal behavior in narrow scene with perspective distortion. Mach. Vis. Appl. 30(5), 987–998 (2019)

    Article  Google Scholar 

  8. Aldweesh, A., Derhab, A., Emam, A.Z.: Deep learning approaches for anomaly-based intrusion detection systems: a survey, taxonomy, and open issues. Knowl. Based Syst. 189, 105124 (2020)

    Article  Google Scholar 

  9. Kiran, B.R., Thomas, D.M., Parakkal, R.: An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. J. Imaging 4(2), 36 (2018)

    Article  Google Scholar 

  10. Hosseinzadeh, M., Rahmani, A.M., Vo, B., Bidaki, M., Masdari, M., Zangakani, M.: Improving security using SVM-based anomaly detection: issues and challenges. Soft. Comput. 2020, 1–29 (2020)

    Google Scholar 

  11. Ko, K.E., Sim, K.B.: Deep convolutional framework for abnormal behavior detection in a smart surveillance system. Eng. Appl. Artif. Intell. 67, 226–234 (2018)

    Article  Google Scholar 

  12. Gupta, T., Nunavath, V., Roy, S.: CrowdVAS-net: a deep-CNN based framework to detect abnormal crowd-motion behavior in videos for predicting crowd disaster. In: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 2877–2882. IEEE (2019)

  13. Wan, M., Shang, W., Zeng, P.: Double behavior characteristics for one-class classification anomaly detection in networked control systems. IEEE Trans. Inf. Forensics Secur. 12(12), 3011–3023 (2017)

    Article  Google Scholar 

  14. Xu, D., Ricci, E., Yan, Y., Song, J., Sebe, N.: Learning deep representations of appearance and motion for anomalous event detection. arXiv:1510.01553 (2015)

  15. Foroughi, H., Rezvanian, A., Paziraee, A.: Robust fall detection using human shape and multi-class support vector machine. In: 2008 Sixth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 413–420. IEEE (2008)

  16. Kwon, D., Kim, H., Kim, J., Suh, S.C., Kim, I., Kim, K.J.: A survey of deep learning-based network anomaly detection. Clust. Comput. 1–3 (2019)

  17. Cai, Y., Ma, L., Liu, G.: A night-time anomaly detection system of hog activities based on passive infrared detector. Appl. Eng. Agric. 35(4), 481–493 (2019)

    Article  Google Scholar 

  18. Mishra, S.R., Mishra, T.K., Sarkar, A., Sanyal, G.: Detection of Anomalies in Human Action Using Optical Flow and Gradient Tensor. InSmart Intelligent Computing and Applications, pp. 561–570. Springer, Singapore (2020)

    Google Scholar 

  19. Zhou, S., Shen, W., Zeng, D., Fang, M., Wei, Y., Zhang, Z.: Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Process. Image Commun. 47, 358–368 (2016)

    Article  Google Scholar 

  20. Chang, C.W., Chang, C.Y., Lin, Y.Y.: A hybrid CNN and LSTM-based deep learning model for abnormal behavior detection. Multimedia Tools Appl. 81(9), 11825–11843 (2022)

    Article  Google Scholar 

  21. Rezaee, K., Rezakhani, S.M., Khosravi, M.R., Moghimi, M.K.: A survey on deep learning-based real-time crowd anomaly detection for secure distributed video surveillance. Personal Ubiquitous Comput. (2021)

  22. Nayak, R., Pati, U.C., Das, S.K.: A comprehensive review on deep learning-based methods for video anomaly detection. Image Vis. Comput. 106, 104078 (2021)

    Article  Google Scholar 

  23. Gunale, K.G., Mukherji, P.: Deep learning with a spatiotemporal descriptor of appearance and motion estimation for video anomaly detection. J. Imaging. 4(6), 79 (2018)

    Article  Google Scholar 

  24. Khan, M.U., Park, H.S., Kyung, C.M.: Rejecting motion outliers for efficient crowd anomaly detection. IEEE Trans. Inf. Forensics Secur. 14(2), 541–556 (2018)

    Article  Google Scholar 

  25. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6479–6488 (2018)

  26. Wu, P., Liu, J., Shen, F.: A deep one-class neural network for anomalous event detection in complex scenes. In: IEEE Transactions on Neural Networks and Learning Systems (2019)

  27. Cai, L., Luo, P., Zhou, G.: Multistage analysis of abnormal human behavior in complex scenes. J. Sens. 2019 (2019)

  28. Zhong, J.X., Li, N., Kong, W., Liu, S., Li, T.H., Li, G.: Graph convolutional label noise cleaner: train a plug-and-play action classifier for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1237–1246 (2019)

  29. Xu, M., Yu, X., Chen, D., Wu, C., Jiang, Y.: An efficient anomaly detection system for crowded scenes using variational autoencoders. Appl. Sci. 9(16), 3337 (2019)

    Article  Google Scholar 

  30. Zhou, J.T., Du, J., Zhu, H., Peng, X., Liu, Y., Goh, R.S.: AnomalyNet: an anomaly detection network for video surveillance. IEEE Trans. Inf. Forensics Secur. 14(10), 2537–2550 (2019)

    Article  Google Scholar 

  31. Hu, Y.: Design and implementation of abnormal behavior detection based on deep intelligent analysis algorithms in massive video surveillance. J. Grid Comput. 18, 1–1 (2020)

    Article  Google Scholar 

  32. Xia, L., Li, Z.: A new method of abnormal behavior detection using LSTM network with temporal attention mechanism. J. Supercomput. 1–9 (2020)

  33. Al-Dhamari, A., Sudirman, R., Mahmood, N.H.: Transfer deep learning along with binary support vector machine for abnormal behavior detection. IEEE Access. 8, 61085–61095 (2020)

    Article  Google Scholar 

  34. Li, N., Chang, F., Liu, C.: Spatial-temporal cascade autoencoder for video anomaly detection in crowded scenes. IEEE Trans. Multimedia 23, 203–215 (2020)

    Article  Google Scholar 

  35. Ullah, W., Ullah, A., Haq, I.U., Muhammad, K., Sajjad, M., Baik, S.W.: CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks. Multimedia Tools Appl. 80, 16979–16995 (2021)

    Article  Google Scholar 

  36. Dubey, S., Boragule, A., Gwak, J., Jeon, M.: Anomalous event recognition in videos based on joint learning of motion and appearance with multiple ranking measures. Appl. Sci. 11(3), 1344 (2021)

    Article  Google Scholar 

  37. Deepak, K., Chandrakala, S., Mohan, C.K.: Residual spatiotemporal autoencoder for unsupervised video anomaly detection. SIViP 15(1), 215–222 (2021)

    Article  Google Scholar 

  38. Fan, Y., Wen, G., Li, D., Qiu, S., Levine, M.D., Xiao, F.: Video anomaly detection and localization via gaussian mixture fully convolutional variational autoencoder. Comput. Vis. Image Underst. 195, 102920 (2020)

    Article  Google Scholar 

  39. Sikdar, A., Chowdhury, A.S.: An adaptive training-less framework for anomaly detection in crowd scenes. Neurocomputing 415, 317–331 (2020)

    Article  Google Scholar 

  40. Mahadevan, V., Li, W., Bhalodia, V., Vasconcelos, N.: Anomaly detection in crowded scenes. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1975–1981. IEEE (2010)

  41. Leyva, R., Sanchez, V., Li, C.T.: The LV dataset: a realistic surveillance video dataset for abnormal event detection. In: 2017 5th International Workshop Biometrics and Forensics (IWBF), pp. 1–6. IEEE (2017)

  42. Dou, L., Zhang, Z., Xu, L., Zou, Q.: IKcr_CNN: a novel computational tool for imbalance classification of human nonhistone crotonylation sites based on convolutional neural networks with focal loss. Comput. Struct. Biotechnol. J. 20, 3268–3279 (2022)

    Article  Google Scholar 

Download references

Funding

There is no funding for this study.

Author information

Authors and Affiliations

  1. Department of Computing Technologies, School of Computing, College of Engineering and Technology, Faculty of Engineering and Technology, SRM Institute of Science and Technology, SRM Nagar, Kattankulathur, Chennai, TamilNadu, India

    M. L. Sworna Kokila & V. Bibin Christopher

  2. Department of Electronics and Communication Engineering, Ponjesly College of Engineering, Nagercoil, TamilNadu, India

    R. Isaac Sajan

  3. Department of Electronics and Communication Engineering, Mar Ephraem College of Engineering and Technology, Marthandam, India

    T. S. Akhila

  4. Electronics and Communication Engineering, Marthandam College of Engineering and Technology, Marthandam, India

    M. Joselin Kavitha

Authors
  1. M. L. Sworna Kokila
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. V. Bibin Christopher
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. Isaac Sajan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. T. S. Akhila
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. M. Joselin Kavitha
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors have participated in writing the manuscript and have revised the final version. All authors read and approved the final manuscript.

Corresponding author

Correspondence to M. L. Sworna Kokila.

Ethics declarations

Conflict of interest

Authors declare that they have no conflict of interest.

Data Availability

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Sworna Kokila M L, Bibin Christopher V, Isaac Sajan R, Akhila T S, Joselin Kavitha M. The first draft of the manuscript was written by Sworna Kokila M L, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. Conceptualization and methodology contributed by Sworna Kokila M L;; formal analysis and investigation contributed by Sworna Kokila M L and Joselin Kavitha M; writing—original draft preparation contributed by Sworna Kokila M L and Isaac Sajan R; writing—review and editing contributed by Akhila T S; supervision contributed by Bibin Christopher V.

Ethical approval

This article does not contain any studies with human participants and/or animals performed by any of the authors.

Informed consent

There is no informed consent for this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kokila, M.L.S., Christopher, V.B., Sajan, R.I. et al. Efficient abnormality detection using patch-based 3D convolution with recurrent model. Machine Vision and Applications 34, 54 (2023). https://doi.org/10.1007/s00138-023-01397-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01397-z

Keywords

Search

Navigation