Detecting abnormal crowd behaviors based on the div-curl characteristics of flow fields
Graphical abstract
Introduction
Crowd safety management in public places has long been an issue of concern. The use of computer vision algorithms to detect changes in crowd states and abnormal activity can help public security officials to respond to emergencies. Over the past decade, video analysis of crowds has undergone rapid development in various aspects, including crowd segmentation [1], [2], [3], [4], crowd count [5], [6], [7], and event detection [8], [9], [10], [11], [12]. However, understanding crowds remains challenging, especially the analysis of highly crowded scenes.
We address the problem of detecting crowd state changes in public areas. Our method considers the condition of wide-field of view surveillance with dense crowds. This research is needed urgently to deal with unexpected mass incidents or terrorist attacks in crowded spaces such as malls, city streets, train stations, stadiums, etc. In medium and high density scenarios, peoples often focus on the motion behaviors of the whole crowd. When a movement collective is studied as a single entity at a macroscopic level, it can be treated as the flow fluid [13], as in Fig. 1(a). Although several theories of hydromechanics have been adopted to analyze crowds [2], [3], they deserve further study. Inspired by hydromechanics and the feature visualization of flow fields [14], [15], [16], we note that divergence and curl could be quite useful for expressing the underlying flow field topology. Therefore, these two characteristics are used to analyze crowd states. In general, abnormalities refer to phenomena that differ significantly from our expectations [17], which are relative to the current state. The current state provides us with known patterns to predict the subsequent results. We consider abnormal activity to be a relative process; therefore, detecting changes in crowd motion via the “temporal context” will be effective and universal. At the same time, a sudden change in a crowd motion state is a gradual process that happens at the microcosm level. For this reason, these variations must be accumulated to a certain threshold before abnormality can be determined.
In this study, we first compute the original optical flow fields from an input video, and use the local spatial and temporal thermal diffusion processing to construct stable coherent motion flow fields that are consistent with the crowd movement trends. Due to the effects of camera perspective, we also conduct a perspective transform processing to obtain more satisfying coherence motion flow fields. Second, a modified method is proposed to extract the physical characteristics, the divergence and curl, of the motion flow field, to obtain advance characteristics field. Finally, a physical characteristic descriptor of crowd motion (PCM) is established to represent the motion state of the crowd flow field, and the changes in the distribution of the PCM are accumulated to detect crowd motion states and abnormal activities.
The main contributions of this work are summarized as follows: (1) local spatial-temporal thermal diffusion processing and perspective transformation are proposed to construct a coherent motion flow field describing crowd movement trends; (2) the physical characteristics, divergence and curl, of a flow field are introduced to classify crowd motion and to construct a model of motion state; (3) a PCM is formulated that combines the methods of state temporal context and cumulative entropy to obtain a quantified result of the changes in crowd motion state; and (4) we also introduce a new dataset SYSU Crowd for evaluating crowd state detection methods. The dataset consists of six videos of various crowd motion events. It models real life scenarios with dense crowd state change activities.
Section snippets
Related work
It has been shown that crowd behavior can be explained by fluid mechanics [18], [19]. In [20], divergence and curl are used to animate a crowd of humans in data-driven animation models. Wang et al. [21] finds that the proper orthogonal decomposition (POD) can be applied to comprehend complicated phenomena in crowds, and POD is widely used in fluids. Johansson et al. [22] proposes a method to identify critical areas and times of hazards by comparing the density, divergence, and curl. Wang et al.
Motion flow field construction
To capture a crowd’s underlying motion information, the first task is to obtain the flow field of the video sequence. This coherent motion flow field can be described as the collective movement trends. We first directly calculate a dense optical flow field to obtain an original planar flow field as in fluid dynamics. Following the steps given below, we then transform this original flow field into crowd coherent motion flow field that meets the needs of our subsequent feature extraction, as
Extraction of divergence and curl characteristics
Divergence and curl are the major physical characteristics of the flow field and often used in the analysis and visualization of sea flow or electromagnetic fields [14], [15]. As important parameters describing flow fields, these two characteristics reflect their nature and features from different perspectives. These can also be used in fluid dynamics to extract the meaningful structures and phenomena from a vector field, such as seeking vortices and shock waves or separating out a region of
Detection and classification of a crowd motion field
To achieve the purpose of perceiving a crowd state, it is necessary to find and extract motion regions that contain the crowds, and cluster those regions into different groups according to their motion properties. This is implemented through the following two steps.
Step 1. Segmenting the crowd motion field. Divergence and curl reflect various motion flows that are generated by the moving crowds. In the crowd motion regions, the motion vectors have relatively greater magnitudes, we then apply
Detection of crowd state
On the basis of the crowd grouping regions constructed above, the state of these motion regions can be detected. Our aim is to identify which crowd changed, and to know where and when the changes occurred. When the crowd changes its motion state, the original order of the vector is disrupted, and topological structures such as the rotational relationship, the flow direction, and the convergence of the vectors will also change. These changes are reflected in physical quantities such as the
Experiments
In this section, the proposed method is evaluated on our own SYSU Crowd dataset and two publicly available datasets: the PETS 2009 dataset [47] and the UMN dataset [48]. In general, the changes in crowd state mentioned above refer to the phenomenon in which pedestrian groups do not maintain their original motion pattern, including direction, speed, appearance, or even composite structure. Meanwhile, we hope that the proposed method can be adapted to crowded scenarios with groups containing
Conclusion
In this study, problems related to detecting crowd state changes are investigated. We use the divergence and curl features of the flow field to constitute a descriptor of the crowd motion state. The significance of a temporal context comparison of the motion state for detecting crowd state is analyzed, and quantitative criteria for changes in crowd motion state based on accumulation conditional entropy are thus proposed. Our divergence-curl-driven framework can effectively detect crowd
Acknowledgments
This work was supported by National Key Research and Development Program of China (2016YFB1001003), the NSFC (U1611461, 61573387).
Xiao-Han Chen received the B.Eng. degree in computer science and technology from Guangdong Ocean University, Zhanjiang, China; the M.Eng. degree in software engineering from Guangdong University of Technology, Guangzhou, China. He is currently working toward the Ph.D. degree with Sun Yat-sen University, Guangzhou, China. His research interests include computer vision and machine learning, including crowd analysis in video surveillance.
References (50)
- et al.
Gestalt laws based tracklets analysis for human crowd understanding
Pattern Recognit.
(2018) - et al.
Structured dictionary learning for abnormal event detection in crowded scenes
Pattern Recognit.
(2018) - et al.
Social network model for crowd anomaly detection and localization
Pattern Recognit.
(2017) - et al.
Proportional data modeling with hidden markov models based on generalized Dirichlet and Beta-Liouville mixtures applied to anomaly detection in public areas
Pattern Recognit.
(2016) - et al.
Robust individual and holistic features for crowd scene classification
Pattern Recognit.
(2016) - et al.
Congested scene classification via efficient unsupervised feature learning and density estimation
Pattern Recognit.
(2016) - et al.
An energy model approach to people counting for abnormal crowd behavior detection
Neurocomputing
(2012) - et al.
Abnormal crowd behavior detection by using the particle entropy
Optik-Int. J. Light Electron Opt.
(2014) - et al.
Motion-based unusual event detection in human crowds
J. Vis. Commun. Image Represent.
(2011) - et al.
Finding coherent motions and semantic regions in crowd scenes: a diffusion and clustering approach
European Conference on Computer Vision–ECCV 2014
(2014)
A Lagrangian particle dynamics approach for crowd flow segmentation and stability analysis
Computer Vision and Pattern Recognition (CVPR), 2007 IEEE Conference on
A streakline representation of flow in crowded scenes
European Conference on Computer Vision–ECCV 2010
Privacy preserving crowd monitoring: counting people without people models or tracking
Computer Vision and Pattern Recognition (CVPR), 2008 IEEE Conference on
Cross-scene crowd counting via deep convolutional neural networks
Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on
Crowd counting with deep negative correlation learning
Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on
Detection of global and local motion changes in human crowds
IEEE Trans. Circuits Syst. Video Technol.
A bayesian model for crowd escape behavior detection
IEEE Trans. Circuits Syst. Video Technol.
Real-time crowd simulation: a review
The state of the art in flow visualisation: feature extraction and tracking
Computer Graphics Forum
Discrete multiscale vector field decomposition
ACM Transactions on Graphics
Understanding pedestrian behaviors from stationary crowd groups
Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on
The flow of human crowds
Annu. Rev. Fluid Mech.
Modeling, evaluation, and scale on artificial pedestrians: a literature review
ACM Comput. Surv.
Crowd motion capture
Comput. Animat. Virtual Worlds
Cited by (36)
Detection of multiple interacting features of different strength in compressible flow fields
2023, Journal of Computational PhysicsSocial interaction model enhanced with speculation stage for human trajectory prediction
2023, Robotics and Autonomous SystemsA new approach to dominant motion pattern recognition at the macroscopic crowd level
2022, Engineering Applications of Artificial IntelligenceUnderstanding crowd flow patterns using active-Langevin model
2021, Pattern RecognitionCitation Excerpt :In [26], motion trajectories have been analyzed using curl and divergence properties to identify different crowd movements. The authors in [27] have used div-curl features of flow fields to detect abnormal crowd behavior. The authors of [28] have developed a method to detect salient regions and instabilities in the crowd by considering the crowd as a dynamic system.
LSTM based trajectory prediction model for cyclist utilizing multiple interactions with environment
2021, Pattern Recognition
Xiao-Han Chen received the B.Eng. degree in computer science and technology from Guangdong Ocean University, Zhanjiang, China; the M.Eng. degree in software engineering from Guangdong University of Technology, Guangzhou, China. He is currently working toward the Ph.D. degree with Sun Yat-sen University, Guangzhou, China. His research interests include computer vision and machine learning, including crowd analysis in video surveillance.
Jian-Huang Lai received his M.Sc. degree in applied mathematics in 1989 and his Ph.D. in mathematics in 1999 from Sun Yat-sen University, China. He joined Sun Yat-sen University in 1989 as an Assistant Professor, where currently he is a Professor with the School of Data and Computer Science. His current research interests are in the areas of digital image processing, pattern recognition, multimedia communication, wavelet and its applications. He has published over 100 scientific papers in the international journals and conferences on image processing and pattern recognition, e.g. IEEE TPAMI, IEEE TKDE, IEEE TNN, IEEE TIP, IEEE TSMC (Part B), Pattern Recognition, ICCV, CVPR and ICDM. Prof. Lai serves as a standing member of the Image and Graphics Association of China and also serves as a standing director of the Image and Graphics Association of Guangdong.