Elsevier

Pattern Recognition

Volume 88, April 2019, Pages 342-355
Pattern Recognition

Detecting abnormal crowd behaviors based on the div-curl characteristics of flow fields

https://doi.org/10.1016/j.patcog.2018.11.023Get rights and content

Highlights

  • Explore the crowd state change detection problem for the dense crowd scenarios of wide-field of view in the public place.

  • Use a less computation required thermal diffusion processing and perspective transform processing to capture the macroscopic crowd motion information.

  • Model motion information by use of divergence and curl characteristics of the flow field.

  • Exploit a physical characteristic descriptor of crowd motion (PCM).

  • Calculate the conditional entropy of PCM distribution to perceiving the crowd changes.

Abstract

This study proposes a divergence-curl-driven framework for the perception of crowd motion states. In this framework, the characteristics of a flow field, divergence and curl, are used to analyze crowd states. As a collective motion, the movement of a pedestrian crowd shows coherent structural properties. By using the methods of fluid mechanics and the feature visualization of flow fields, a physical characteristic descriptor of crowd motion is established that can model the motion state in a crowd flow field. Given the significance of the temporal comparison of motion states for detecting changes in crowds, a method based on the temporal context of motion is presented to measure changes in the distribution of the physical characteristic descriptors of crowd motion. This method can be used to calculate differences in the distribution of the flow field’s physical characteristics between each state and measure these subtle continuous changes on the sample points, thereby obtaining a quantified metric of changes in a crowd’s motion state. Experiments on crowd event datasets demonstrate the effectiveness of our proposed framework for detecting crowd state changes and abnormal activity.

Introduction

Crowd safety management in public places has long been an issue of concern. The use of computer vision algorithms to detect changes in crowd states and abnormal activity can help public security officials to respond to emergencies. Over the past decade, video analysis of crowds has undergone rapid development in various aspects, including crowd segmentation [1], [2], [3], [4], crowd count [5], [6], [7], and event detection [8], [9], [10], [11], [12]. However, understanding crowds remains challenging, especially the analysis of highly crowded scenes.

We address the problem of detecting crowd state changes in public areas. Our method considers the condition of wide-field of view surveillance with dense crowds. This research is needed urgently to deal with unexpected mass incidents or terrorist attacks in crowded spaces such as malls, city streets, train stations, stadiums, etc. In medium and high density scenarios, peoples often focus on the motion behaviors of the whole crowd. When a movement collective is studied as a single entity at a macroscopic level, it can be treated as the flow fluid [13], as in Fig. 1(a). Although several theories of hydromechanics have been adopted to analyze crowds [2], [3], they deserve further study. Inspired by hydromechanics and the feature visualization of flow fields [14], [15], [16], we note that divergence and curl could be quite useful for expressing the underlying flow field topology. Therefore, these two characteristics are used to analyze crowd states. In general, abnormalities refer to phenomena that differ significantly from our expectations [17], which are relative to the current state. The current state provides us with known patterns to predict the subsequent results. We consider abnormal activity to be a relative process; therefore, detecting changes in crowd motion via the “temporal context” will be effective and universal. At the same time, a sudden change in a crowd motion state is a gradual process that happens at the microcosm level. For this reason, these variations must be accumulated to a certain threshold before abnormality can be determined.

In this study, we first compute the original optical flow fields from an input video, and use the local spatial and temporal thermal diffusion processing to construct stable coherent motion flow fields that are consistent with the crowd movement trends. Due to the effects of camera perspective, we also conduct a perspective transform processing to obtain more satisfying coherence motion flow fields. Second, a modified method is proposed to extract the physical characteristics, the divergence and curl, of the motion flow field, to obtain advance characteristics field. Finally, a physical characteristic descriptor of crowd motion (PCM) is established to represent the motion state of the crowd flow field, and the changes in the distribution of the PCM are accumulated to detect crowd motion states and abnormal activities.

The main contributions of this work are summarized as follows: (1) local spatial-temporal thermal diffusion processing and perspective transformation are proposed to construct a coherent motion flow field describing crowd movement trends; (2) the physical characteristics, divergence and curl, of a flow field are introduced to classify crowd motion and to construct a model of motion state; (3) a PCM is formulated that combines the methods of state temporal context and cumulative entropy to obtain a quantified result of the changes in crowd motion state; and (4) we also introduce a new dataset SYSU Crowd for evaluating crowd state detection methods. The dataset consists of six videos of various crowd motion events. It models real life scenarios with dense crowd state change activities.

Section snippets

Related work

It has been shown that crowd behavior can be explained by fluid mechanics [18], [19]. In [20], divergence and curl are used to animate a crowd of humans in data-driven animation models. Wang et al. [21] finds that the proper orthogonal decomposition (POD) can be applied to comprehend complicated phenomena in crowds, and POD is widely used in fluids. Johansson et al. [22] proposes a method to identify critical areas and times of hazards by comparing the density, divergence, and curl. Wang et al.

Motion flow field construction

To capture a crowd’s underlying motion information, the first task is to obtain the flow field of the video sequence. This coherent motion flow field can be described as the collective movement trends. We first directly calculate a dense optical flow field to obtain an original planar flow field as in fluid dynamics. Following the steps given below, we then transform this original flow field into crowd coherent motion flow field that meets the needs of our subsequent feature extraction, as

Extraction of divergence and curl characteristics

Divergence and curl are the major physical characteristics of the flow field and often used in the analysis and visualization of sea flow or electromagnetic fields [14], [15]. As important parameters describing flow fields, these two characteristics reflect their nature and features from different perspectives. These can also be used in fluid dynamics to extract the meaningful structures and phenomena from a vector field, such as seeking vortices and shock waves or separating out a region of

Detection and classification of a crowd motion field

To achieve the purpose of perceiving a crowd state, it is necessary to find and extract motion regions that contain the crowds, and cluster those regions into different groups according to their motion properties. This is implemented through the following two steps.

Step 1. Segmenting the crowd motion field. Divergence and curl reflect various motion flows that are generated by the moving crowds. In the crowd motion regions, the motion vectors have relatively greater magnitudes, we then apply

Detection of crowd state

On the basis of the crowd grouping regions constructed above, the state of these motion regions can be detected. Our aim is to identify which crowd changed, and to know where and when the changes occurred. When the crowd changes its motion state, the original order of the vector is disrupted, and topological structures such as the rotational relationship, the flow direction, and the convergence of the vectors will also change. These changes are reflected in physical quantities such as the

Experiments

In this section, the proposed method is evaluated on our own SYSU Crowd dataset and two publicly available datasets: the PETS 2009 dataset [47] and the UMN dataset [48]. In general, the changes in crowd state mentioned above refer to the phenomenon in which pedestrian groups do not maintain their original motion pattern, including direction, speed, appearance, or even composite structure. Meanwhile, we hope that the proposed method can be adapted to crowded scenarios with groups containing

Conclusion

In this study, problems related to detecting crowd state changes are investigated. We use the divergence and curl features of the flow field to constitute a descriptor of the crowd motion state. The significance of a temporal context comparison of the motion state for detecting crowd state is analyzed, and quantitative criteria for changes in crowd motion state based on accumulation conditional entropy are thus proposed. Our divergence-curl-driven framework can effectively detect crowd

Acknowledgments

This work was supported by National Key Research and Development Program of China (2016YFB1001003), the NSFC (U1611461, 61573387).

Xiao-Han Chen received the B.Eng. degree in computer science and technology from Guangdong Ocean University, Zhanjiang, China; the M.Eng. degree in software engineering from Guangdong University of Technology, Guangzhou, China. He is currently working toward the Ph.D. degree with Sun Yat-sen University, Guangzhou, China. His research interests include computer vision and machine learning, including crowd analysis in video surveillance.

References (50)

  • S. Ali et al.

    A Lagrangian particle dynamics approach for crowd flow segmentation and stability analysis

    Computer Vision and Pattern Recognition (CVPR), 2007 IEEE Conference on

    (2007)
  • R. Mehran et al.

    A streakline representation of flow in crowded scenes

    European Conference on Computer Vision–ECCV 2010

    (2010)
  • A.B. Chan et al.

    Privacy preserving crowd monitoring: counting people without people models or tracking

    Computer Vision and Pattern Recognition (CVPR), 2008 IEEE Conference on

    (2008)
  • C. Zhang et al.

    Cross-scene crowd counting via deep convolutional neural networks

    Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on

    (2015)
  • Z. Shi et al.

    Crowd counting with deep negative correlation learning

    Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on

    (2018)
  • I.R. de Almeida et al.

    Detection of global and local motion changes in human crowds

    IEEE Trans. Circuits Syst. Video Technol.

    (2017)
  • S. Wu et al.

    A bayesian model for crowd escape behavior detection

    IEEE Trans. Circuits Syst. Video Technol.

    (2014)
  • R. Leggett

    Real-time crowd simulation: a review

    (2004)
  • F.H. Post et al.

    The state of the art in flow visualisation: feature extraction and tracking

    Computer Graphics Forum

    (2003)
  • Y. Tong et al.

    Discrete multiscale vector field decomposition

    ACM Transactions on Graphics

    (2003)
  • M.L. Boas, Mathematical methods in the physical sciences, Wiley, pp....
  • S. Yi et al.

    Understanding pedestrian behaviors from stationary crowd groups

    Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on

    (2015)
  • R.L. Hughes

    The flow of human crowds

    Annu. Rev. Fluid Mech.

    (2003)
  • F. Martinez-Gil et al.

    Modeling, evaluation, and scale on artificial pedestrians: a literature review

    ACM Comput. Surv.

    (2017)
  • N. Courty et al.

    Crowd motion capture

    Comput. Animat. Virtual Worlds

    (2007)
  • Cited by (36)

    • A new approach to dominant motion pattern recognition at the macroscopic crowd level

      2022, Engineering Applications of Artificial Intelligence
    • Understanding crowd flow patterns using active-Langevin model

      2021, Pattern Recognition
      Citation Excerpt :

      In [26], motion trajectories have been analyzed using curl and divergence properties to identify different crowd movements. The authors in [27] have used div-curl features of flow fields to detect abnormal crowd behavior. The authors of [28] have developed a method to detect salient regions and instabilities in the crowd by considering the crowd as a dynamic system.

    View all citing articles on Scopus

    Xiao-Han Chen received the B.Eng. degree in computer science and technology from Guangdong Ocean University, Zhanjiang, China; the M.Eng. degree in software engineering from Guangdong University of Technology, Guangzhou, China. He is currently working toward the Ph.D. degree with Sun Yat-sen University, Guangzhou, China. His research interests include computer vision and machine learning, including crowd analysis in video surveillance.

    Jian-Huang Lai received his M.Sc. degree in applied mathematics in 1989 and his Ph.D. in mathematics in 1999 from Sun Yat-sen University, China. He joined Sun Yat-sen University in 1989 as an Assistant Professor, where currently he is a Professor with the School of Data and Computer Science. His current research interests are in the areas of digital image processing, pattern recognition, multimedia communication, wavelet and its applications. He has published over 100 scientific papers in the international journals and conferences on image processing and pattern recognition, e.g. IEEE TPAMI, IEEE TKDE, IEEE TNN, IEEE TIP, IEEE TSMC (Part B), Pattern Recognition, ICCV, CVPR and ICDM. Prof. Lai serves as a standing member of the Image and Graphics Association of China and also serves as a standing director of the Image and Graphics Association of Guangdong.

    View full text