Learning a deep network with cross-hierarchy aggregation for crowd counting

Highlights

• A novel deep model for crowd counting is proposed.

• We propose cross-hierarchy aggregation to reuse hierarchical features.

• Results show that our method outperforms the state-of-the-art methods.

• Cross-scene evaluation verifies the superior generalization ability of our model.

Abstract

Crowd counting, a significant but challenging task in computer vision, aims at estimating the number of people in an image or video. Recent methods for crowd counting have obtained promising performance due to deep neural networks but most of them ignore the abundant conducive information in hierarchical features. In this paper, a novel Cross-Hierarchy Aggregation Network (CHANet) is proposed to exploit multi-hierarchy information in the crowd features from each hierarchy and aggregate cross-hierarchy features to generate reasonable density maps. Firstly, we propose a CHA module to fully extract local hierarchical features and capture maximum information of the crowd features. The CHA module combines residual and dense connections without over-assigning parameters for feature reuse. Then, we utilize the global hierarchical features from the shallow hierarchies to obtain a more powerful representation ability with a global residual connection. Experimental evaluations on four publicly available crowd counting datasets (ShanghaiTech, UCF-QNRF, WorldExpo’10, and Beijing BRT) demonstrate that the proposed CHANet achieves superior performance compared to other state-of-the-art methods.

Introduction

With the world population growing at a high rate, crowd counting has been arousing a great deal of interest in the past few years. The main objective of crowd counting is to estimate the number of people in a single image or video frame collected by surveillance cameras. Thus, crowd counting is widely applied in video security monitoring, traffic congestion control, urban planning, and other fields [1]. However, it remains difficult because of light change, severe occlusions, uneven crowd distributions, and camera perspective distortions.

Due to the successful application of Convolutional Neural Networks (CNNs) in computer vision including classification [2], [3], detection [4], [5], segmentation [6], [7], and person re-identification [8], [9], researchers have newly proposed lots of methods [10], [11], [12], [13], [14], [15], [16], [17], [18] that use CNNs to extract features from crowd images and generate density maps for crowd counting. And by integrating the generated density maps, we can obtain the number of people in the crowd images. These methods have gained significant progress in addressing the aforementioned problems that hinder the performance of crowd counting. However, two main challenges still exist, as demonstrated in Fig. 1. One is uneven distributions that the crowd is dense in one area and sparse in another in one scene. The other is camera perspective distortions in which the scales of people vary from barely visible tiny points to clear body outlines.

Ideas of multi-column [11], [12], [19], multi-scale [13], [20], [21], [22], and other methods [14], [15], [16] have been implemented on CNN-based frameworks trying to solve the stumbling block that stands in crowd counting. Although these methods are effective in alleviating the problems, they fuse features only from certain layers and aggregate them at a specific stage of the networks. Moreover, they simply mention multi-scale information for various crowd scales when fusing multiple features from different hierarchies. In fact, there is more information than scale information hidden in the hierarchical features. According to Hou et al. [23], deep hierarchies extract high-level features with abundant semantic information while shallow hierarchies extract low-level features with abundant spatial information. Thus, we believe that each hierarchy contains quantitative conducive information including multi-scale information to enhance the accuracy of crowd counting. Furthermore, the information of crowd features extracted from each hierarchy is different but can improve the ability of feature representation and the performance of the model.

To fully extract information from each hierarchy and aggregate cross-hierarchy features from the preceding hierarchies, we propose a novel Cross-Hierarchy Aggregation Network (CHANet) that can learn maximum information of hierarchical features without over-assigning parameters for feature reuse. Residual [24] and dense [25] connections are two different ways to aggregate features from different hierarchies. It has been found that residual connections can pass information directly to the following hierarchies of the network and address degradation problems, while dense connections directly concatenate features from different hierarchies and improve feature reuse. Both connections can reuse the features from preceding hierarchies and allow maximum information flow. Thus, we propose a combination of residual and dense connections named CHA that can fully extract abundant local features from the states of preceding hierarchies and capture maximum information of the crowd features without over-assigning parameters. Specifically, we adopt several CHA modules in our method and each has a set of convolutional layers. In the CHA module, we present local hierarchical aggregation to concatenate the features from each hierarchy. We adopt a 1 × 1 convolution to adaptively aggregate cross-hierarchy information from hierarchical features and enhance the representation ability of the aggregated features. In addition, we introduce a local residual connection to further aggregate local hierarchical features with the input of each CHA module. The CHA module combining residual and dense connections can preserve multiple advantageous information including but not limited to scale information from every hierarchy and ensure maximum information flow in the network. Furthermore, to fuse the global hierarchical features of the whole crowd image from the shallow hierarchies, we present a global residual connection to connect the previous local features generated from the CHA module.

The major contributions of this work are as follows:

• A novel CHANet is proposed for crowd counting to extract conducive crowd information in hierarchical features and aggregate cross-hierarchy features to generate more reasonable density maps.

• We present a CHA module to ensure maximum information flow by combining residual and dense connections and also study the number of convolutional layers in each CHA module as well as the number of CHA modules.

• Experimental evaluations on four crowd counting datasets demonstrate that the proposed CHANet outperforms the state-of-the-art methods.

We give a brief review of CNN-based methods for crowd counting in Section 2. Details of our proposed CHANet are illustrated in Section 3. In Section 4, sufficient experimental results on four datasets are demonstrated to evaluate the performance of our method. Eventually, we summarize this work in Section 5.