Abstract
Human crowds have become hotspot research, particularly in crowd analysis to ensure human safety. Adaptations of machine learning (ML) approaches, especially deep learning, play a vital role in the applications of evacuation, detection, and prediction pertaining to crowd analysis. Further development in the analysis of crowd is needed to understand human behaviors due to the fast growth of crowd in urban megacities. This article presents a comprehensive review of crowd analysis ML-based systems, where it is categorized with respect to its purposes, viz. crowd evacuation that provides efficient evacuation routes, abnormality detection that could detect the occurrence of any irregular movement or behavior, and crowd prediction that could foresee the occurrence of any possible disasters or predict pedestrian trajectory. Moreover, this article reviews the applied techniques of machine learning with a brief discussion on the used software and simulation platforms. This work also classifies crowd evacuation into data-driven methods and goal-driven learning methods that have attracted significant attention due to their potential to adopt virtual agents with learning capabilities. This review finds that convolutional neural networks and recurrent neural networks have shown superiority in abnormality detection and prediction, whereas deep reinforcement learning has shown potential performance in the development of human level capacities of reasoning. These three methods contribute to the modeling and understanding of pedestrian behavior and will enhance further development in crowd analysis to ensure human safety.
Similar content being viewed by others
References
Zhou S et al (2010) Crowd modeling and simulation technologies. ACM Trans Model Comput Simul 20(4):1–35. https://doi.org/10.1145/1842722.1842725
Martinez-Gil F, Lozano M, García-Fernández I, Fernández F (2017) Modeling, evaluation, and scale on artificial pedestrians: a literature review. ACM Comput Surv. https://doi.org/10.1145/3117808
Still GK (2000) Crowd dynamic. University of Warwick, Warwick
Gerakakis I, Gavriilidis P, Dourvas NI, Georgoudas IG, Trunfio GA, Sirakoulis GC (2019) Accelerating fuzzy cellular automata for modeling crowd dynamics. J Comput Sci 32:125–140. https://doi.org/10.1016/j.jocs.2018.10.007
Centorrino P, Corbetta A, Cristiani E, Onofri E (2021) Managing crowded museums: visitors flow measurement, analysis, modeling, and optimization. J Comput Sci 53:101357. https://doi.org/10.1016/j.jocs.2021.101357
Mitsopoulou M, Dourvas NI, Sirakoulis GC, Nishinari K (2019) Spatial games and memory effects on crowd evacuation behavior with Cellular Automata. J Comput Sci 32:87–98. https://doi.org/10.1016/j.jocs.2018.09.003
Shiwakoti N, Shi X, Ye Z (2019) A review on the performance of an obstacle near an exit on pedestrian crowd evacuation. Saf Sci 113(November 2018):54–67. https://doi.org/10.1016/j.ssci.2018.11.016
Draghici A, Van Steen M (2018) A survey of techniques for automatically sensing the behavior of a crowd. ACM Comput Surv. https://doi.org/10.1145/3129343
Wang Q, Liu H, Gao K, Zhang L (2019) Improved multi-agent reinforcement learning for path planning-based crowd simulation. IEEE Access 7:73841–73855. https://doi.org/10.1109/ACCESS.2019.2920913
Karamouzas I, Sohre N, Hu R, Guy SJ (2018) Crowd space: a predictive crowd analysis technique. ACM Trans Graph 37:6. https://doi.org/10.1145/3272127.3275079
Thida M, Yong YL, Climent-Pérez P, Eng H-L, Remagnino P (2013) A literature review on video analytics of crowded scenes. Intell Multimed Surveill 9783642415:1–192. https://doi.org/10.1007/978-3-642-41512-8-2
Zhan B, Monekosso DN, Remagnino P, Velastin SA, Xu LQ (2008) Crowd analysis: a survey. Mach Vis Appl 19(5–6):345–357. https://doi.org/10.1007/s00138-008-0132-4
Jacques JCS, Mussef SR, Jung CR (2010) Crowd analysis using computer vision techniques. IEEE Signal Process Mag 27(5):66–77. https://doi.org/10.1109/MSP.2010.937394
Sjarif NNA, Shamsuddin SM, Hashim SZ (2012) Detection of abnormal behaviors in crowd scene: a review. Int J Adv Soft Comput Appl 4(1):1–33
Swathi HY, Shivakumar G, Mohana HS (2017) Crowd behavior analysis: a survey. In: International conference on recent advances in electronics and communication technology, pp 169–178. https://doi.org/10.1109/ICRAECT.2017.66
Ibrahim AM, Venkat I, Subramanian KG, Khader AT, De Wilde P (2016) Intelligent evacuation management systems: a review. ACM Trans Intell Syst Technols. https://doi.org/10.1145/2842630
Bi H, Gelenbe E (2019) A survey of algorithms and systems for evacuating people in confined spaces. Electron 8(6):1–27. https://doi.org/10.3390/electronics8060711
Zhou M, Dong H, Ioannou PA, Zhao Y, Wang FY (2019) Guided crowd evacuation: approaches and challenges. IEEE/CAA J Autom Sin 6(5):1081–1094. https://doi.org/10.1109/JAS.2019.1911672
Sodemann AA, Ross MP, Borghetti BJ (2012) A review of anomaly detection in automated surveillance. IEEE Trans Syst Man Cybern Part C Appl Rev 42(6):1257–1272. https://doi.org/10.1109/TSMCC.2012.2215319
Sreenu G, Saleem Durai MA (2019) Intelligent video surveillance: a review through deep learning techniques for crowd analysis. J Big Data 6(1):1–27. https://doi.org/10.1186/s40537-019-0212-5
Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539
Wang K, Shi X, Goh APX, Qian S (2019) A machine learning based study on pedestrian movement dynamics under emergency evacuation. Fire Saf J 106:163–176. https://doi.org/10.1016/j.firesaf.2019.04.008
Kuang S, Hu N, Cai W (2019) A data-driven path planning model for crowd capacity analysis. J Comput Sci 34:66–79. https://doi.org/10.1016/j.jocs.2019.05.003
Nguyen ND, Nguyen T, Nahavandi S (2017) System design perspective for human-level agents using deep reinforcement learning: a survey. IEEE Access 5:27091–27102. https://doi.org/10.1109/ACCESS.2017.2777827
Karamouzas I, Overmars M (2012) Simulating and evaluating the local behavior of small pedestrian groups. IEEE Trans Vis Comput Graph 18(3):394–406. https://doi.org/10.1109/TVCG.2011.133
Lee H (2016) Human crowd evacuation framework and analysis using look-ahead-based reinforcement learning algorithm. Int J Digit Hum 1(3):248. https://doi.org/10.1504/ijdh.2016.10000733
Lee J, Won J, Lee J (2018) Crowd simulation by deep reinforcement learning. https://doi.org/10.1145/3230744.3230782
Du L, Han L, Li XY (2014) Distributed coordinated in-vehicle online routing using mixed-strategy congestion game. Transp Res Part B Methodol 67:1–17. https://doi.org/10.1016/j.trb.2014.05.003
Fernández F, Borrajo D, Parker LE (2005) A reinforcement learning algorithm in cooperative multi-robot domains. J Intell Robot Syst Theory Appl 43(2–4):161–174. https://doi.org/10.1007/s10846-005-5137-x
Mnih V et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236
Silver D et al (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489. https://doi.org/10.1038/nature16961
Martinez-Gil F, Lozano M, Fernández F (2014) Strategies for simulating pedestrian navigation with multiple reinforcement learning agents. Auton Agent Multi Agent Syst 29(1):98–130. https://doi.org/10.1007/s10458-014-9252-6
Torrey L (2010) Crowd simulation via multi-Agent reinforcement learning. In: 6th AAAI conference on artificial intelligence and interactive digital entertainment, pp 89–94
Henry P, Vollmer C, Ferris B, Fox D (2010) Learning to navigate through crowded environments. IEEE Int Conf Robot Autom. https://doi.org/10.1109/ROBOT.2010.5509772
Casadiego L, Pelechano N (2015) From one to many: simulating groups of agents with reinforcement learning controllers. Lect Notes Comput Sci (Includ Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 9238:119–123. https://doi.org/10.1007/978-3-319-21996-7_12
F Martinez-Gil, M Lozano, F Fernández (2011) Multi-agent reinforcement learning for simulating pedestrian navigation. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 7113 LNAI, pp 54–69. https://doi.org/10.1007/978-3-642-28499-1_4.
Yang Z, Merrick K, Member S, Jin L, Abbass HA, Member S (2018) For continuous action control. IEEE Trans Neural Networks Learn Syst 29(11):5174–5184
Guo Q, Zeng X, Hu S, Phoummixay S, Ye Y (2021) Learning a deep network with cross-hierarchy aggregation for crowd counting. Knowl-Based Syst 213:106691. https://doi.org/10.1016/j.knosys.2020.106691
Neelapu R, Devi GL, Rao KS (2018) Deep learning based conventional neural network architecture for medical image classification. Trait du Signal 35(2):169–182. https://doi.org/10.3166/TS.35.169-182
Liu S, Zhao Y, Xue F, Chen B, Chen X (2019) DeepCount: crowd counting with WiFi via deep learning. pp 1–13. http://arxiv.org/abs/1903.05316
Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166. https://doi.org/10.1109/72.279181
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. pp 1–9. http://arxiv.org/abs/1412.3555
Graves A, Mohamed AR, Hinton G (2013) Speech recognition with deep recurrent neural networks. ICASSP IEEE Int Conf Acoust Speech Signal Process Proc 3:6645–6649. https://doi.org/10.1109/ICASSP.2013.6638947
Direkoglu C, Sah M, O’Connor NE (2017) Abnormal crowd behavior detection using novel optical flow-based features. https://doi.org/10.1109/AVSS.2017.8078503
Iwahashi H, et al. (2014) Extraordinary judging using electrodermal activity and acceleration for emergency rescue evacuation support system. In: Proceedings of the 43th International Conference on Parallel Processing Workshops, pp 355–360. https://doi.org/10.1109/ICPPW.2014.53
Varghese EB, Thampi SM (2018) A deep learning approach to predict crowd behavior based on emotion. Springer Nat Switz 11010:296–307. https://doi.org/10.1007/978-3-030-04375-9
Yao Z, Zhang G, Lu D, Liu H (2019) Data-driven crowd evacuation: a reinforcement learning method. Neurocomputing 366:314–327. https://doi.org/10.1016/j.neucom.2019.08.021
Zheng S, Liu H (2019) Improved multi-Agent deep deterministic policy gradient for path planning-based crowd simulation. IEEE Access 7:147755–147770. https://doi.org/10.1109/ACCESS.2019.2946659
Sharma J, Andersen P-A, Granmo O-C, Goodwin M (2020) Deep Q-learning with Q-matrix transfer learning for novel fire evacuation environment. IEEE Trans Syst Man Cybern Syst. https://doi.org/10.1109/tsmc.2020.2967936
Tian K, Jiang S (2018) Reinforcement learning for safe evacuation time of fire in hong Kong-Zhuhai-Macau immersed tube tunnel. Syst Sci Control Eng 6(2):45–56. https://doi.org/10.1080/21642583.2018.1509746
Fragkos G, Apostolopoulos PA, Tsiropoulou EE (2019) “ESCAPE: evacuation strategy through clustering and autonomous operation in public safety systems. Future Internet. https://doi.org/10.3390/fi11010020
Wan Z, Jiang C, Fahad M, Ni Z, Guo Y, He H (2018) Robot-assisted pedestrian regulation based on deep Reinforcement Learning. IEEE Trans Cybern 50(4):1669–1682. https://doi.org/10.1109/TCYB.2018.2878977
Li X, Liang Y, Zhao M, Wang C, Bai H, Jiang Y (2019) Simulation of evacuating crowd based on deep learning and social force model. IEEE Access 7:155361–155371. https://doi.org/10.1109/ACCESS.2019.2949106
Nagananthini C, Yogameena B (2017) Crowd disaster avoidance system (CDAS) by deep learning using eXtended center symmetric local binary pattern (XCS-LBP) texture features. In: International conference on computer vision and image processing, advances in intelligent systems and computing, pp 487–498. https://doi.org/10.1007/978-981-10-2104-6_44
Yi S, Li H, Wang X (2016) Pedestrian behavior understanding and prediction with deep neural networks. ECCV 1(c):263–279. https://doi.org/10.1007/978-3-319-46448-0
Alsalat GY, El-Ramly M, Fahmy AA, Karim Said MD (2018) Detection of mass panic using Internet of Things and machine learning. Int J Adv Comput Sci Appl 9(5):320–329. https://doi.org/10.14569/IJACSA.2018.090542
Jiang FZ, et al. (2017) Supercharging crowd dynamics estimation in disasters via spatio-Temporal deep neural network. In: International conference on data science and advanced analytics, 2018-Janua, pp 184–192. https://doi.org/10.1109/DSAA.2017.11
Duives DC, Wang G, Kim J (2019) Forecasting pedestrian movements using recurrent neural networks: an application of crowd monitoring data. Sensors (Switzerland). https://doi.org/10.3390/s19020382
Lee N, Choi W, Vernaza P, Choy CB, Torr PHS, Chandraker M (2017) DESIRE: distant future prediction in dynamic scenes with interacting agents. In: Proceedings—30th IEEE conference on computer vision and pattern recognition, CVPR, pp 2165–2174. https://doi.org/10.1109/CVPR.2017.233
Xu Y, Piao Z, Gao S (2018) Encoding crowd interaction with deep neural network. CVPR, pp 5275–5284
Johansson A, Helbing D, Shukla PK (2007) Pedestrian model by evolutionary adjustment to video tracking data. Advances 10(2):271–288. https://doi.org/10.1142/S0219525907001355
Pellegrini S, Ess A, Schindler K, Van Gool L (2009) You’ll never walk alone: modeling social behavior for multi-target tracking. Proc IEEE Int Conf Comput Vis Iccv. https://doi.org/10.1109/ICCV.2009.5459260
Scovanner P, Tappen MF (2009) Learning pedestrian dynamics from the real world. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2009.5459224
Helbing D, Molnár P (1995) Social force model for pedestrian dynamics. Phys Rev E 51(5):4282–4286. https://doi.org/10.1103/PhysRevE.51.4282
Den Van Berg J, Lin M, Manocha D (2008) Reciprocal velocity obstacles for real-time multi-agent navigation. Proc IEEE Int Conf Robot Autom. https://doi.org/10.1109/ROBOT.2008.4543489
Kim S, Bera A, Best A, Chabra R, Manocha D (2016) Interactive and adaptive data-driven crowd simulation. Proc IEEE Virtual Real 2016:29–38. https://doi.org/10.1109/VR.2016.7504685
Hartigan JA, Wong MA (1979) A K-means clustering algorithm. R Stat Soc 28(1):100–108. https://doi.org/10.2307/2346830
Li T, Chang H, Wang M, Ni B, Hong R, Yan S (2015) Crowded scene analysis: a survey. IEEE Trans Circuits Syst Video Technol 25(3):367–386. https://doi.org/10.1109/TCSVT.2014.2358029
Aggarwal JK, Ryoo MS (2011) Human activity analysis: a review. ACM Comput Surv. https://doi.org/10.1145/1922649.1922653
Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: IEEE computer society conference on computer vision and pattern recognition workshops, no 1, pp 935–942. https://doi.org/10.1109/CVPRW.2009.5206641
Kratz L, Nishino K (2009) Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models. In: IEEE computer society conference on computer vision and pattern recognition workshops, CVPR Workshops no. June, pp 1446–1453. https://doi.org/10.1109/CVPRW.2009.5206771
Cong Y, Yuan J, Tang Y (2013) Video anomaly search in crowded scenes via spatio-temporal motion context. IEEE Trans Inf Forensics Secur 8(10):1590–1599. https://doi.org/10.1109/TIFS.2013.2272243
Wang B, Ye M, Li X, Zhao F, Ding J (2012) Abnormal crowd behavior detection using high-frequency and spatio-temporal features. Mach Vis Appl 23(3):501–511. https://doi.org/10.1007/s00138-011-0341-0
Yang H, Cao Y, Wu S, Lin W, Zheng S, Yu Z (2012) Abnormal crowd behavior detection based on local pressure model. In: Conf. Handb.—Asia Pacific Signal Inf. Process. Assoc. Annu. Summit Conf. APSIPA ASC
Cong Y, Yuan J, Liu J (2013) Abnormal event detection in crowded scenes using sparse representation. Pattern Recognit 46(7):1851–1864. https://doi.org/10.1016/j.patcog.2012.11.021
Shi X, Chen Z, Wang H, Yeung D-Y, Wong W, Woo W (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. https://doi.org/10.1155/2018/6184713
“Dataset: Detection of Unusual Crowd Activity,” University of Minnesota. http://mha.cs.umn.edu/proj_events.shtml#crowd. Accessed 14 Sept 2020
“Dataset S3 Rapid Dispersion,” Binghamton University. http://cs.binghamton.edu/~mrldata/pets2009. Accessed 14 Sept 2020
Rabiee H, Haddadnia J, Mousavi H (2016) Crowd behavior representation: an attribute-based approach. Springerplus 5:1179. https://doi.org/10.1186/s40064-016-2786-0
Johansson A, Helbing D, Al-Abideen HZ, Al-Bosta S (2008) From crowd dynamics to crowd safety: a video-based analysis. Adv Complex Syst 11(4):497–527. https://doi.org/10.1142/S0219525908001854
Silva C, Bouwmans T, Frélicot C (2015) An extended center-symmetric local binary pattern for background modeling and subtraction in videos. In: 10th international conference on computer vision theory and applications, vol 1, pp 395–402. https://doi.org/10.5220/0005266303950402
Cancela B, Iglesias A, Ortega M, Penedo MG (2014) Unsupervised trajectory modelling using temporal information via minimal paths. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2553–2560. https://doi.org/10.1109/CVPR.2014.327
Yi S, Li H, Wang X (2015) Understanding pedestrian behaviors from stationary crowd groups. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 3488–3496. https://doi.org/10.1109/CVPR.2015.7298971
Wang X, Ma X, Grimson WEL (2009) Unsupervised activity perception in crowded and complicated scenes using hierarchical bayesian models. IEEE Trans Pattern Anal Mach Intell 31(3):539–555. https://doi.org/10.1109/TPAMI.2008.87
Zhou B, Wang X, Tang X (2012) Understanding collective crowd behaviors: learning a mixture model of dynamic pedestrian-agents. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2871–2878. https://doi.org/10.1109/CVPR.2012.6248013
Zhang C, Li H, Wang X, Yang X (2015) Cross-scene crowd counting via deep convolutional neural networks. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 833–841. https://doi.org/10.1109/CVPR.2015.7298684
Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model 2: 935–942
Choi S, Yeo H, Kim J (2018) Network-wide vehicle trajectory prediction in urban traffic networks using deep learning. Transp Res Rec 2672(45):173–184. https://doi.org/10.1177/0361198118794735
Alahi A, Goel K, Ramanathan V, Robicquet A, Fei-Fei L, Savarese S (2016) Social LSTM: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 961–971. https://doi.org/10.1109/CVPR.2016.110
Bi C, Pan G, Yang L, Lin CC, Hou M, Huang Y (2019) Evacuation route recommendation using auto-encoder and Markov decision process. Appl Soft Comput J 84:105741. https://doi.org/10.1016/j.asoc.2019.105741
Abadi M, et al. (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. http://arxiv.org/abs/1603.04467
Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: a matlab-like environment for machine learning. BigLearn, NIPS Work, pp 1–6. http://infoscience.epfl.ch/record/192376/files/Collobert_NIPSWORKSHOP_2011.pdf.
Chollet F, Long J, Robie T (2015) Keras. https://github.com/keras-team/keras
Brockman G, et al. (2016) OpenAI Gym. CoRR, pp 1–4. http://arxiv.org/abs/1606.01540
Juliani A, et al. (2018) Unity: a general platform for intelligent agents. CoRR, vol. abs/1809.0, pp 1–28, 2018. http://arxiv.org/abs/1809.02627
Chen A, He J, Liang M, Su G (2020) Crowd response considering herd effect and exit familiarity under emergent occasions: a case study of an evacuation drill experiment. Phys A Stat Mech its Appl 556:124654. https://doi.org/10.1016/j.physa.2020.124654
Gao J, Zhang J, He J, Gong J, Zhao J (2020) Experiment and simulation of pedestrian’s behaviors during evacuation in an office. Phys A Stat Mech its Appl 545:123749. https://doi.org/10.1016/j.physa.2019.123749
Waqar S, Khan UG, Waseem MH, Qayyum S (2022) The utility of datasets in crowd modelling and analysis: a survey. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-13227-x
Cheng Y, Zheng X (2018) Can cooperative behaviors promote evacuation efficiency? Phys A Stat Mech Appl 492:2069–2078. https://doi.org/10.1016/j.physa.2017.11.124
Nguyen TT, Nguyen ND, Nahavandi S (2020) Deep reinforcement learning for multi-agent systems: a review of challenges, solutions and applications. IEEE Trans Cybern 50(9):3826–3839
Zhou L, Yang P, Chen C, Gao Y (2017) Multiagent reinforcement learning with sparse interactions by negotiation and knowledge transfer. IEEE Trans Cybern 47(5):1238–1250. https://doi.org/10.1109/TCYB.2016.2543238
Haghani M (2020) Empirical methods in pedestrian, crowd and evacuation dynamics: Part II. Field methods and controversial topics. Saf Sci 129:104760. https://doi.org/10.1016/j.ssci.2020.104760
Wijermans N, Conrado C, van Steen M, Martella C, Li J (2016) A landscape of crowd-management support: an integrative approach. Saf Sci 86:142–164. https://doi.org/10.1016/j.ssci.2016.02.027
Acknowledgements
This research is funded by the FRGS 2019 Grant: FRGS/1/2019/ICT02/UIAM/02/2 awarded by the Ministry of Higher Education Malaysia.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Bahamid, A., Mohd Ibrahim, A. A review on crowd analysis of evacuation and abnormality detection based on machine learning systems. Neural Comput & Applic 34, 21641–21655 (2022). https://doi.org/10.1007/s00521-022-07758-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-022-07758-5