Skip to main content
SpringerLink
Log in

Generative adversarial network based abnormal behavior detection in massive crowd videos: a Hajj case study

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Hajj is an annual Islamic pilgrimage, which is attended by millions of pilgrims every year. Therefore, there are many security management problems. The existing solutions can only solve the problems of a small-scale crowd, which contains a simple abnormal behavior and a clear surveillance video. However, the performance hasn’t reached a satisfactory result for a large-scale crowd. Therefore, we propose an abnormal behavior detection method based on optical flow and generative adversarial network (GAN). There are three main contributions in this paper. Firstly, the dynamic features of the model are extracted based on the optical flows. The effectiveness of the features is validated by experiments. Secondly, we propose an optical flow framework based on GAN and use a transfer learning strategy to detect behavioral abnormalities in large-scale crowd scenes. The framework uses U-Net and Flownet to generate and distinguish the normal and abnormal behaviors of individuals within the massive crowds. Finally, a number of abnormal behavior pilgrimage videos from different scenes is collected and tested. The accuracy of UMN scenes 1, 2, 3, and UCSD reaches 99.4%, 97.1%, 97.6% and 89.26%, respectively. It also achieves 79.63% of detection accuracy in the large-scale crowd videos using Abnormal Behaviors HAJJ dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Alafif T, Hailat Z, Aslan M, Chen X (2017a) On classifying facial races with partial occlusions and pose variations. In: 2017 16th IEEE International conference on machine learning and applications (ICMLA), IEEE, pp 679–684

  • Alafif T, Hailat Z, Aslan M, Chen X (2017b) On detecting partially occluded faces with pose variations. In: 2017 14th International symposium on pervasive systems, algorithms and networks and 2017 11th international conference on frontier of computer science and technology and 2017 third international symposium of creative computing (ISPAN-FCST-ISCC), IEEE, pp 28–37

  • Andreini P, Bonechi S, Bianchini M, Mecocci A, Scarselli F (2020) Image generation by gan and style transfer for agar plate image segmentation. Comput Methods Programs Biomed 184:105268

    Article  Google Scholar 

  • Boulares M, Alafif T, Barnawi A (2020) Transfer learning benchmark for cardiovascular disease recognition. IEEE Access 8:109475–109491

    Article  Google Scholar 

  • Cong Y, Yuan J, Liu J (2011) Sparse reconstruction cost for abnormal event detection. In: CVPR 2011, IEEE, pp 3449–3456

  • Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, Van Der Smagt P, Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766

  • Feng Y, Yuan Y, Lu X (2017) Learning deep event models for crowd anomaly detection. Neurocomputing 219:548–556

    Article  Google Scholar 

  • Ganokratanaa T, Aramvith S, Sebe N (2019) Anomaly event detection using generative adversarial network for surveillance videos. In: 2019 Asia-Pacific signal and information processing association annual summit and conference (APSIPA ASC), IEEE, pp 1395–1399

  • Gibson JJ (1950) The perception of the visual world

  • Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  • Gupta T, Nunavath V, Roy S (2019) Crowdvas-net: A deep-cnn based framework to detect abnormal crowd-motion behavior in videos for predicting crowd disaster. In: 2019 IEEE international conference on systems, man and cybernetics (SMC), IEEE, pp 2877–2882

  • Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742

  • Hung WC, Tsai YH, Liou YT, Lin YY, Yang MH (2018) Adversarial learning for semi-supervised semantic segmentation. arXiv preprint. arXiv:180207934

  • Ionescu RT, Smeureanu S, Alexe B, Popescu M (2017) Unmasking the abnormal events in video. In: Proceedings of the IEEE international conference on computer vision, pp 2895–2903

  • Kim J, Grauman K (2009) Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates. In: 2009 IEEE conference on computer vision and pattern recognition, IEEE, pp 2921–2928

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  • LeCun Y, Boser BE, Denker JS, Henderson D, Howard RE, Hubbard WE, Jackel LD (1990) Handwritten digit recognition with a back-propagation network. In: Advances in neural information processing systems, pp 396–404

  • Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection—a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6536–6545

  • Ma J, Wu H, Zhang J, Zhang L (2020) Sd-fb-gan: saliency-driven feedback gan for remote sensing image super-resolution reconstruction. In: 2020 IEEE International conference on image processing (ICIP), IEEE, pp 528–532

  • Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: 2010 IEEE Computer society conference on computer vision and pattern recognition, IEEE, pp 1975–1981

  • Mathieu M, Couprie C, LeCun Y (2015) Deep multi-scale video prediction beyond mean square error. arXiv preprint. arXiv:151105440

  • Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: 2009 IEEE Conference on computer vision and pattern recognition, IEEE, pp 935–942

  • Musse SR, Thalmann D (1997) A model of human crowd behavior: group inter-relationship and collision detection analysis. In: Computer animation and simulation’97. Springer, pp 39–51

  • Qiu P, Kim S, Lee JH, Choi J (2018) Anomaly detection in a crowd using a cascade of deep learning networks. Information systems design and intelligent applications. Springer, Berlin, pp 596–607

    Chapter  Google Scholar 

  • Ravanbakhsh M, Nabi M, Sangineto E, Marcenaro L, Regazzoni C, Sebe N (2017) Abnormal event detection in videos using generative adversarial nets. In: 2017 IEEE International conference on image processing (ICIP), IEEE, pp 1577–1581

  • Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 234–241

  • Shin W, Cho SB (2018) Cctv image sequence generation and modeling method for video anomaly detection using generative adversarial network. In: International conference on intelligent data engineering and automated learning, Springer, pp 457–467

  • Singh A, Nigam A (2019) Effect of identity mapping, transfer learning and domain knowledge on the robustness and generalization ability of a network: a biometric based case study. J Ambient Intell Humaniz Comput 11:1–18

    Google Scholar 

  • Singh R, Ahmed T, Singh R, Udmale SS, Singh SK (2019) Identifying tiny faces in thermal images using transfer learning. J Ambient Intell Humaniz Comput 11:1–10

    Google Scholar 

  • Tay NC, Connie T, Ong TS, Goh KOM, Teh PS (2019) A robust abnormal behavior detection method using convolutional neural network. Computational science and technology. Springer, Berlin, pp 37–47

    Chapter  Google Scholar 

  • University of Minnesota (2020) Unusual crowd activity dataset of University of Minnesota. http://mha.cs.umn.edu/movies/ crowdactivity-all.avi. Accessed 25 Apr 2020

  • Wang L, Zhou F, Li Z, Zuo W, Tan H (2018) Abnormal event detection in videos using hybrid spatio-temporal autoencoder. In: 2018 25th IEEE International conference on image processing (ICIP), IEEE, pp 2276–2280

  • Wu Y, Tan X, Lu T (2020) A new multiple-distribution gan model to solve complexity in end-to-end chromosome karyotyping. Complexity 2020:15

    Google Scholar 

  • Zhou K, Gao S, Cheng J, Gu Z, Fu H, Tu Z, Yang J, Zhao Y, Liu J (2020) Sparse-gan: Sparsity-constrained generative adversarial network for anomaly detection in retinal oct image. In: 2020 IEEE 17th International symposium on biomedical imaging (ISBI), IEEE, pp 1227–1231

Download references

Acknowledgements

The authors extend their appreciation to the Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia for funding this research work through the project number (227).

Author information

Authors and Affiliations

  1. Computer Science Department, Jamoum University College, Umm Al-Qura University, Mecca, Saudi Arabia

    Tarik Alafif

  2. Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

    Bander Alzahrani, Reem Alotaibi & Ahmed Barnawi

  3. School of Computer Science and Technology, Huazhong University of Science and Technology (HUST), Wuhan, China

    Yong Cao & Min Chen

Authors
  1. Tarik Alafif
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bander Alzahrani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Reem Alotaibi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ahmed Barnawi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Min Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bander Alzahrani.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alafif, T., Alzahrani, B., Cao, Y. et al. Generative adversarial network based abnormal behavior detection in massive crowd videos: a Hajj case study. J Ambient Intell Human Comput 13, 4077–4088 (2022). https://doi.org/10.1007/s12652-021-03323-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-021-03323-5

Keywords

Search

Navigation