Skip to main content
SpringerLink
Log in

Anomalous event detection and localization in dense crowd scenes

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recognizing and localizing anomalous events in crowd scenes is a challenging problem that has attracted the attention of researchers in computer vision. Surveillance cameras record scenes that require an automated examination to identify anomalous events. Existing approaches in the field have utilized different feature descriptors, modeling methods, and recognition strategies to accurately and efficiently detect anomalies in the scene. Existing techniques in the field have focused mainly on performing global frame-level identification of abnormal events. Only a small number of studies have considered locating abnormal action in the frame. Proposed methods are also often evaluated on scenes that contain a sparse number of individuals performing abnormal and normal staged acts. This research aims to detect and locate anomalies in a structured and unstructured dense crowd scene. The proposed model first detects moving objects and individuals in the scene using a deep convolutional neural network and tracks objects and individuals using spatial and temporal features. Then, spatial-temporal features are extracted from consecutive frames of interest points. The extracted features include the histogram of optical flow, velocity and direction of moving objects, and other features that can indicate sudden motion change. A support vector machine model is then used to classify abnormal events into one of seven classes. The proposed methodology is evaluated on Hajj2 dataset that has 18 videos and 7 different types of abnormal events.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Algorithm 2
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Al-Shargie F, Tang TB, Badruddin N, et al. (2018) Towards multilevel mental stress assessment using svm with ecoc: an eeg approach. Med Bio Eng Comput 56(1):125–136

    Article  Google Scholar 

  2. Alafif T, Alzahrani B, Cao Y, et al. (2022) Generative adversarial network based abnormal behavior detection in massive crowd videos: A hajj case study. J Ambient Intell Humaniz Comput 13(8):4077–4088

    Article  Google Scholar 

  3. Alafif T, Hadi A, Allahyani M et al (2022) Hybrid classifiers for spatio-temporal real-time abnormal behaviors detection, tracking, and recognition in massive hajj crowds. https://doi.org/10.48550/ARXIV.2207.11931https://doi.org/10.48550/ARXIV.2207.11931

  4. Almazroey AA, Jarraya SK (2020) Abnormal events and behavior detection in crowd scenes based on deep learning and neighborhood component analysis feature selection. In: Joint European-US workshop on applications of invariance in computer vision. Springer, pp 258–267

  5. Aqeel M, Khan KB, Azam MA, et al. (2020) Detection of anomaly in videos using convolutional autoencoder and generative adversarial network model. In: 2020 IEEE 23rd International Multitopic Conference (INMIC). IEEE, pp 1–6

  6. Bansod SD, Nandedkar AV (2019) Anomalous event detection and localization using stacked autoencoder. In: International conference on computer vision and image processing. Springer, pp 117–129

  7. Bansod SD, Nandedkar AV (2020) Crowd anomaly detection and localization using histogram of magnitude and momentum. Vis Comput 36(3):609–620

    Article  Google Scholar 

  8. Bera A, Kim S, Manocha D (2016) Interactive crowd-behavior learning for surveillance and training. IEEE Comput Graph Appl 36(6):37–45. https://doi.org/10.1109/MCG.2016.113

    Article  Google Scholar 

  9. Bewley A, Ge Z, Ott L, et al. (2016) Simple online and realtime tracking. In: IEEE International Conference on Image Processing (ICIP). https://doi.org/10.1109/icip.2016.7533003, vol 2016. IEEE

  10. Biswas S, Venkatesh Babu R (2017) Anomaly detection via short local trajectories. Neurocomputing 242:63–72

    Article  Google Scholar 

  11. Chen CY, Shao Y (2015) Crowd escape behavior detection and localization based on divergent centers. IEEE Sensors J 15(4):2431–2439. https://doi.org/10.1109/JSEN.2014.2381260

    Article  Google Scholar 

  12. Chen Y, Wang S (2017) A weighted mhof and sparse representation based crowd anomaly detection algorithm. In: IEEE 15th Intl Conf on Dependable, Autonomic and Secure Computing, 15th Intl Conf on Pervasive Intelligence and Computing, 3rd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech). IEEE, pp 760–765

  13. Chibloun A, El Fkihi S, Mliki H, et al. (2018) Abnormal crowd behavior detection using speed and direction models. In: 2018 9th International Symposium on Signal, Image, Video and Communications (ISIVC). IEEE, pp 197–202

  14. Das D, Mishra D (2018) Unsupervised anomalous trajectory detection for crowded scenes. In: 2018 IEEE 13th International Conference on Industrial and Information Systems (ICIIS) pp 27–31

  15. Deng L, Lian D, Huang Z, et al. (2022) Graph convolutional adversarial networks for spatiotemporal anomaly detection. IEEE Trans Neur Netw Learn Syst 33(6):2416–2428. https://doi.org/10.1109/TNNLS.2021.3136171https://doi.org/10.1109/TNNLS.2021.3136171

    Article  MathSciNet  Google Scholar 

  16. Direkoglu C (2020) Abnormal crowd behavior detection using motion information images and convolutional neural networks. IEEE Access 8:80,408–80,416

    Article  Google Scholar 

  17. Feng Y, Yuan Y, Lu X (2017) Learning deep event models for crowd anomaly detection. Neurocomputing 219:548–556

    Article  Google Scholar 

  18. Gnouma M, Ejbali R, Zaied M (2018) Abnormal events’ detection in crowded scenes. Multimed Tools Applic 77(19):24,843–24,864

    Article  Google Scholar 

  19. Gnouma M, Ejbali R, Zaied M (2019) Video anomaly detection and localization in crowded scenes. In: International Joint Conference: 12th International Conference on Computational Intelligence in Security for Information Systems (CISIS 2019) and 10th International Conference on EUropean Transnational Education (ICEUTE 2019). Springer, pp 87–96

  20. Guo C, Lin H, He Z, et al. (2019) Crowd abnormal event detection based on sparse coding. Int J Humanoid Robot 16(04):1941,005

    Article  Google Scholar 

  21. Guo S, Bai Q, Gao S, et al. (2019) An analysis method of crowd abnormal behavior for video service robot. IEEE Access 7:169,577–169,585. https://doi.org/10.1109/ACCESS.2019.2954544

    Article  Google Scholar 

  22. Han Q, Wang H, Yang L, et al. (2020) Real-time adversarial gan-based abnormal crowd behavior detection. J Real-Time Image Proc 17(6):2153–2162

    Article  Google Scholar 

  23. He H, Ma Y (2013) Imbalanced learning: foundations, algorithms, and applications. Wiley-IEEE Press

  24. Hu L, Hu F (2017) Anomaly detection in crowded scenes via sa-mhof and sparse combination. In: 2017 10th International Symposium on Computational Intelligence and Design (ISCID), pp 421–424. https://doi.org/10.1109/ISCID.2017.130

  25. Hu X, Dai J, Huang Y, et al. (2020) A weakly supervised framework for abnormal behavior detection and localization in crowded scenes. Neurocomputing 383:270–281

    Article  Google Scholar 

  26. Hu X, Lian J, Zhang D, et al. (2022) Video anomaly detection based on 3d convolutional auto-encoder. SIViP, 1–9

  27. ping Hu Z, Zhang L, fang Li S, et al. (2020) Parallel spatial-temporal convolutional neural networks for anomaly detection and location in crowded scenes. J Vis Commun Image Represent 67:102,765

    Article  Google Scholar 

  28. Ilyas Z, Aziz Z, Qasim T, et al. (2021) A hybrid deep network based approach for crowd anomaly detection. Multimed Tools Appl, 1–15

  29. Ji QG, Chi R, Lu ZM (2018) Anomaly detection and localisation in the crowd scenes using a block-based social force model. IET Image Process 12(1):133–137

    Article  Google Scholar 

  30. Joshi KV, Patel NM (2021) A cnn based approach for crowd anomaly detection. Int J Next-Gen Comput, 12(1)

  31. Kaltsa V, Briassouli A, Kompatsiaris I, et al. (2015) Swarm intelligence for detecting interesting events in crowded environments. IEEE Trans Image Process 24 (7):2153–2166. https://doi.org/10.1109/TIP.2015.2409559https://doi.org/10.1109/TIP.2015.2409559

    Article  MathSciNet  MATH  Google Scholar 

  32. Khan MUK, Park HS, Kyung CM (2018) Rejecting motion outliers for efficient crowd anomaly detection. IEEE Trans Inform Forens Secur 14(2):541–556

    Article  Google Scholar 

  33. Li A, Miao Z, Cen Y (2016) Global anomaly detection in crowded scenes based on optical flow saliency, IEEE

  34. Li A, Miao Z, Cen Y, et al. (2017) Anomaly detection using sparse reconstruction in crowded scenes. Multimed Tools Applic 76(24):26,249–26,271

    Article  Google Scholar 

  35. Li X, Li W, Liu B, et al. (2018) Object-oriented anomaly detection in surveillance videos. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2018.8461422, pp 1907–1911

  36. Li X, Li W, Liu B, et al. (2019) Object and patch based anomaly detection and localization in crowded scenes. Multimed Tools Appl, 1–16

  37. Li N, Chang F, Liu C (2020) Spatial-temporal cascade autoencoder for video anomaly detection in crowded scenes. IEEE Trans Multimed 23:203–215

    Article  Google Scholar 

  38. Li X, Yang Y, Xu Y, et al. (2021) Rapid detection of crowd abnormal behavior based on the hierarchical thinking. In: Barolli L, Li KF, Miwa H (eds) Advances in intelligent networking and collaborative systems. Springer International Publishing, Cham, pp 361–371

  39. Li N, Zhong JX, Shu X, et al. (2022) Weakly-supervised anomaly detection in video surveillance via graph convolutional label noise cleaning. Neurocomputing 481:154–167

    Article  Google Scholar 

  40. Lin TY, Maire M, Belongie S, et al. (2014) Microsoft coco: common objects in context. https://doi.org/10.48550/ARXIV.1405.0312

  41. Lin H, Deng JD, Woodford BJ (2015) Anomaly detection in crowd scenes via online adaptive one-class support vector machines. In: 2015 IEEE International Conference on Image Processing (ICIP). https://doi.org/10.1109/ICIP.2015.7351239, pp 2434–2438

  42. Lin S, Yang H, Tang X et al (2019) Social mil: interaction-aware for crowd anomaly detection. In: 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp 1–8. https://doi.org/10.1109/AVSS.2019.8909882

  43. Lin S, Yang H (2021) Dual-mode iterative denoiser: tackling the weak label for anomaly detection. In: 2020 25th International Conference on Pattern Recognition (ICPR). https://doi.org/10.1109/ICPR48806.2021.9412673, pp 6742–6749

  44. Liu S, Qi L, Qin H, et al. (2018) Path aggregation network for instance segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8759–8768

  45. Liu S, Xu L, Li Q, et al. (2018) Fault diagnosis of water quality monitoring devices based on multiclass support vector machines and rule-based decision trees. IEEE Access 6:22,184–22,195

    Article  Google Scholar 

  46. Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th international joint conference on artificial intelligence. IJCAI’81, vol 2. Morgan Kaufmann Publishers Inc., San Francisco, pp 674–679

  47. Luo Z, He W, Liwang M et al (2017) Real-time detection algorithm of abnormal behavior in crowds based on gaussian mixture model. In: 2017 12th International Conference on Computer Science and Education (ICCSE). https://doi.org/10.1109/ICCSE.2017.8085486, pp 183–187

  48. Mahadevan V, Li W, Bhalodia V, et al. (2010) Anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 1975–1981

  49. Marsden M, McGuinness K, Little S et al (2016) Holistic features for real-time crowd behaviour anomaly detection. In: 2016 IEEE International Conference on Image Processing (ICIP). https://doi.org/10.1109/ICIP.2016.7532491, pp 918–922

  50. Mehmood A (2021) Efficient anomaly detection in crowd videos using pre-trained 2d convolutional neural networks. IEEE Access 9:138,283–138,295. https://doi.org/10.1109/ACCESS.2021.3118009

    Article  Google Scholar 

  51. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 935–942

  52. Mishra SR, Mishra TK, Sarkar A, et al. (2020) Detection of anomalies in human action using optical flow and gradient tensor. In: Smart intelligent computing and applications. Springer, pp 561–570

  53. Mondal R, Chanda B (2018) Anomaly detection using context dependent optical flow. In: Proceedings of the 11th Indian conference on computer vision, graphics and image processing. ICVGIP, vol 2018. Association for Computing Machinery, New York, pp 1–8

  54. Moustafa AN, Gomaa W (2020) Gate and common pathway detection in crowd scenes and anomaly detection using motion units and lstm predictive models. Multimed Tools Applic 79(29):20,689–20,728

    Article  Google Scholar 

  55. Nain N, Lamba S (2018) Oriented tracklets approach for anomalous scene detection in high density crowd. In: 2018 14th International conference on signal-image technology internet-based systems (SITIS), pp 435–441. https://doi.org/10.1109/SITIS.2018.00073

  56. Pan J, Liang D (2017) Holistic crowd interaction modelling for anomaly detection. In: Chinese conference on biometric recognition. Springer, pp 642–649

  57. Patil N, Biswas PK (2016) Global abnormal events detection in surveillance video — a hierarchical approach. In: 2016 Sixth International Symposium on Embedded Computing and System Design (ISED). https://doi.org/10.1109/ISED.2016.7977085, pp 217–222

  58. Jocher G, Chaurasia A, Stoken A, Borovec J, NanoCode012, Kwon Y, TaoXie, Fang J, imyhxy, Michael K, Lorna, Abhiram V, Montes D, Nadar J, Laughing, tkianai, yxNONG, Skalski P, Wang Z, Hogan A, Fati C, Mammana L, AlexWang1900, Patel D, Yiwei D, You F, Hajek J, Diaconu L, Thanh Minh M (2022) ultralytics/yolov5: v6.1 - TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference. Zenodo, v6.1. https://doi.org/10.5281/zenodo.6222936

  59. Qasim T, Bhatti N (2019) A low dimensional descriptor for detection of anomalies in crowd videos. Math Comput Simul 166:245–252

    Article  MathSciNet  MATH  Google Scholar 

  60. Ramchandran A, Sangaiah AK (2020) Unsupervised deep learning system for local anomaly event detection in crowded scenes. Multimed Tools Applic 79(47):35,275–35,295

    Article  Google Scholar 

  61. Ramos J, Nedjah N, de Macedo Mourelle L, et al. (2018) Visual data mining for crowd anomaly detection using artificial bacteria colony. Multimed Tools Applic 77(14):35,755–17,777

    Article  Google Scholar 

  62. Sabih M, Vishwakarma DK (2021) Crowd anomaly detection with lstms using optical features and domain knowledge for improved inferring. Vis Comput, 1–12

  63. Sabokrou M, Fathy M, Hoseini M et al (2015) Real-time anomaly detection and localization in crowded scenes. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). https://doi.org/10.1109/CVPRW.2015.7301284, pp 56–62

  64. Sabokrou M, Fayyaz M, Fathy M, et al. (2017) Deep-cascade: Cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans Image Process 26(4):1992–2004. https://doi.org/10.1109/TIP.2017.2670780

    Article  MathSciNet  MATH  Google Scholar 

  65. Samat A, Yokoya N, Du P, et al. (2019) Direct, ecoc, nd and end frameworks—which one is the best? An empirical study of sentinel-2a msil1c image classification for arid-land vegetation mapping in the ili river delta, kazakhstan. Remote Sens 11(16):1953

    Article  Google Scholar 

  66. Shao S, Zhao Z, Li B et al (2018) Crowdhuman: a benchmark for detecting human in a crowd. https://doi.org/10.48550/ARXIV.1805.00123

  67. statista (2022) Saudi Arabia: Hajj population. Annual number of Hajj pilgrims to Saudi Arabia from 1999 to 2019. https://www.statista.com/statistics/617696/saudi-arabia-total-hajj-pilgrims/#statisticContainer. Accessed 2022-02-06

  68. Tian Y, Pang G, Chen Y, et al. (2021) Weakly-supervised video anomaly detection with robust temporal feature magnitude learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp 4975–4986

  69. Tomé A, Salgado L (2017) Detection of anomalies in surveillance scenarios using mixture models. In: 2017 International Carnahan Conference on Security Technology (ICCST). https://doi.org/10.1109/CCST.2017.8167830, pp 1–4

  70. Turaga P, Chellappa R, Veeraraghavan A (2010) Advances in video-based human activity analysis: challenges and approaches. In: Zelkowitz MV (ed) Advances in computers, advances in computers. https://doi.org/10.1016/S0065-2458(10)80007-5, vol 80. Elsevier, pp 237–290

  71. Vaquero L, Brea VM, Mucientes M (2022) Tracking more than 100 arbitrary objects at 25 fps through deep learning. Pattern Recogn 121:108,205. https://doi.org/10.1016/j.patcog.2021.108205, https://www.sciencedirect.com/science/article/pii/S0031320321003861

    Article  Google Scholar 

  72. Wang D, Wang S (2021) Abnormal event detection algorithm based on dual attention future frame prediction and gap fusion discrimination. J Electron Imag 30:023,009–023,009

    Article  Google Scholar 

  73. Wang Q, Wang QWKZC, Ma M, et al. (2016) Hybrid histogram of oriented optical flow for abnormal behavior detection in crowd scenes. Int J Pattern Recognit Artif Intell 30:1655,007:1–1655,007:14

    Article  MathSciNet  Google Scholar 

  74. Wang S, Zhu E, Yin J et al (2016) Anomaly detection in crowded scenes by sl-hof descriptor and foreground classification. In: 2016 23rd International Conference on Pattern Recognition (ICPR). https://doi.org/10.1109/ICPR.2016.7900159, pp 3398–3403

  75. Wang T, Qiao M, Lin Z, et al. (2019) Generative neural networks for anomaly detection in crowded scenes. IEEE Trans Inform Forens Secur 14 (5):1390–1399. https://doi.org/10.1109/TIFS.2018.2878538

    Article  Google Scholar 

  76. Wang L, Tan H, Zhou F, et al. (2022) Unsupervised anomaly video detection via a double-flow convlstm variational autoencoder. IEEE Access 10:44,278–44,289. https://doi.org/10.1109/ACCESS.2022.3165977https://doi.org/10.1109/ACCESS.2022.3165977

    Article  Google Scholar 

  77. Wojke N, Bewley A, Paulus D (2017) Simple online and realtime tracking with a deep association metric. https://doi.org/10.48550/ARXIV.1703.07402

  78. Xu Y, Lu L, Xu Z, et al. (2018) Towards intelligent crowd behavior understanding through the stfd descriptor exploration. Sensing and Imaging 19(1):1–22

    Article  MathSciNet  Google Scholar 

  79. Xu M, Yu X, Chen D et al (2019) An efficient anomaly detection system for crowded scenes using variational autoencoders. Appl Sci, 9(16). https://doi.org/10.3390/app9163337, https://www.mdpi.com/2076-3417/9/16/3337

  80. Yang B, Cao J, Ni R, et al. (2018) Anomaly detection in moving crowds through spatiotemporal autoencoding and additional attention. Adv Multim 2018:2087,574:1–2087,574:8

    Google Scholar 

  81. Yang M, Rashidi L, Rajasegarar S et al (2018) Crowd activity change point detection in videos via graph stream mining. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). https://doi.org/10.1109/CVPRW.2018.00059, pp 328–3288

  82. Zhang Y, Lu H, Zhang L, et al. (2016) Combining motion and appearance cues for anomaly detection. Pattern Recogn 51:443–452

    Article  Google Scholar 

  83. Zhang X, Zhang Q, Hu S et al (2018) Energy level-based abnormal crowd behavior detection. Sensors 18(2). https://doi.org/10.3390/s18020423, https://www.mdpi.com/1424-8220/18/2/423

  84. Zhang S, Gong M, Xie Y, et al. (2022) Influence-aware attention networks for anomaly detection in surveillance videos. IEEE Trans Circuits Syst Video Technol 32(8):5427–5437. https://doi.org/10.1109/TCSVT.2022.3148392https://doi.org/10.1109/TCSVT.2022.3148392

    Article  Google Scholar 

  85. Zhao K, Liu B, Li W et al (2018) Anomaly detection and localization: a novel two-phase framework based on trajectory-level characteristics. In: 2018 IEEE International Conference on Multimedia Expo Workshops (ICMEW). https://doi.org/10.1109/ICMEW.2018.8551517, pp 1–6

  86. Zhou S, Shen W, Zeng D, et al. (2015) Unusual event detection in crowded scenes by trajectory analysis. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2015.7178180, pp 1300–1304

Download references

Acknowledgments

The authors extend their appreciation to the Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia for funding this research work through the project number (227).

Author information

Authors and Affiliations

  1. Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

    Areej Alhothali, Amal Balabid, Reem Alharthi, Bander Alzahrani, Reem Alotaibi & Ahmed Barnawi

Authors
  1. Areej Alhothali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amal Balabid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Reem Alharthi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bander Alzahrani
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Reem Alotaibi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ahmed Barnawi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Areej Alhothali or Bander Alzahrani.

Ethics declarations

Conflict of Interests

The authors declare that they have no known conflicts of interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alhothali, A., Balabid, A., Alharthi, R. et al. Anomalous event detection and localization in dense crowd scenes. Multimed Tools Appl 82, 15673–15694 (2023). https://doi.org/10.1007/s11042-022-13967-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13967-w

Keywords

Search

Navigation