Multi-scale structural kernel representation for object detection

Highlights

• The first attempt to integrate high-order statistics into deep CNNs for effective object detection.

• The proposed high-order statistical module preserves the spatial information while taking account into their special geometry structures.

• Performing favorably in comparison to the state-of-the-art methods and showing good generalization ability to other dense prediction tasks.

Abstract

Existing high-performance object detection methods greatly benefit from the powerful representation ability of deep convolutional neural networks (CNNs). Recent researches show that integration of high-order statistics remarkably improves the representation ability of deep CNNs. However, high-order statistics for object detection lie in two challenges. Firstly, previous methods insert high-order statistics into deep CNNs as global representations, which lose spatial information of inputs, and so are not applicable to object detection. Furthermore, high-order statistics have special structures, which should be considered for proper use of high-order statistics. To overcome above challenges, this paper proposes a Multi-scale Structural Kernel Representation (MSKR) for improving performance of object detection. Our MSKR is developed based on the polynomial kernel approximation, which does not only draw into high-order statistics but also preserve the spatial information of input. To consider geometry structures of high-order representations, a feature power normalization method is introduced before computation of kernel representation. Comparing with the most commonly used first-order statistics in existing CNN-based detectors, our MSKR can generate more discriminative representations, and so be flexibly integrated into deep CNNs for improving performance of object detection. By adopting the proposed MSKR to existing object detection methods (i.e., Faster R-CNN, FPN, Mask R-CNN and RetinaNet), it achieves clear improvement on three widely used benchmarks, while obtaining very competitive performance with state-of-the-art methods.

Introduction

Object detection has attracted a lot of attentions in past decades. As one of the fundamental problems in computer vision, it plays a key role in a widely spreading of applications [1], [2]. From the approaches [3] based on traditional handcrafted features to ones [4], [5] based on deep convolutional features, the rapid development of convolutional neural networks (CNNs) [6], [7] greatly improves the performance of object detection. R-CNN [4] is among the first which exploits the powerful representation ability of deep CNNs to characterize the object proposals and achieves a significant improvement compared with the traditional methods. Subsequently, Region of Interest (RoI) pooling layer and Region Proposal Network (RPN) are respectively introduced by Fast R-CNN [8] and Faster R-CNN [5], allowing object detection can be designed into an end-to-end architecture. Such methods require no pre-generated proposals [9], thereby leading to better performance and faster training/testing.

Although Faster R-CNN yields promising performance, it obtains representations by simply performing average pooling on the outputs of one single convolution (conv) layer (i.e., the last conv layer), limiting the robustness and accuracy of detection. As illustrated in Fig. 1(a), Faster R-CNN fails to detect birds with large pose changes, blur and similar background. One solution to improve performance of object detection is extraction of multi-scale feature maps as representations from different conv layers [10], and such methods can be concluded as two parts: concatenation of multi-scale feature maps [11], [12] and pyramidal feature hierarchy [13], [14]. Generally speaking, the feature maps from bottom layers have higher resolutions but weaker semantic information, while feature maps in top layers have high-level semantic information but lower resolutions. The concatenation-based methods can obtain a coarse-to-fine representation for each object proposal by concatenating the outputs of different convolution layers into one single feature map. For the methods based on pyramidal feature hierarchy, the outputs of different convolution layers are employed in a pyramid manner. Moreover, each combination gives its own prediction, and all detection results are fused by using non-maximum suppression. As shown in Fig. 1(b), the methods based on concatenation of multi-scale feature maps (e.g., RON [13]) are able to improve detection performance by enhancing the representations.

All aforementioned methods focus on improving detection performance by extracting multi-scale feature maps, after where the simple first-order pooling (i.e., RoI-Pooling) is performed on feature maps to generate representations. Recently, some researchers show integration of high-order statistics can significantly improve the representation ability of deep CNNs [16], [17]. Among them, B-CNN [16] inserts a second-order noncentral moment into deep CNNs, and element-wise power normalization followed by ℓ₂-normalization is performed. Wang et al. [17]. embed a global Gaussian distribution into deep CNNs. Zhang et al. [18] propose a second-order locality-constrained affine subspace coding method to perform both image classification and image retrieval tasks. These methods obtain promising improvement over first-order pooling based CNN models on challenging fine-grained visual categorization. Li et al. [19] propose a matrix power normalized second-order pooling, showing consistent superiority over various CNN models on large-scale ImageNet classification [20].

Above discussion clearly encourages us to exploit high-order statistics for improving the performance of object detection. However, there exist two challenges to exploit high-order statistics for object detection. First of all, aforementioned high-order methods totally compute global representations for the whole images, which completely lose spatial information of images, and so are not applicable to object detection. On the other hand, high-order statistics have special structures, and previous works [17], [19] have demonstrated that geometry structures should be considered for achieving favorable performance. To handle the first challenge, we introduce a polynomial kernel approximation method inspired by Cai et al. [21] in our previous work [15], where the weight of the high-order statistics inherent in a polynomial kernel can be approximated by rank-1 tensors decomposition [22], and high-order representations can be computed by learning weight parameters. In deep architectures, we can learn weight parameters by using a series of 1 × 1 convolutions and element-wise product operations, and all these operations preserve spatial information. Therefore, the introduced polynomial kernel method can capture high-order statistics while preserving spatial information, having ability to improve performance of dense predication tasks (e.g., object detection).

Recent work [19] shows that matrix power normalization can effectively exploit the geometry of second-order statistics in deep CNN architectures to improve classification performance. Given a set of convolutional features $X\in {\mathbb{R}}^{C\times n},$matrix power normalization of second-order pooling of X (i.e., $\mathbf{M}=\frac{1}{C}{X}^{\top }X$) can be computed by shrinking eigenvalues of Mwith a power function through eigenvalue decomposition (EIG) or singular value decomposition (SVD). However, such kinds of methods can not be directly adopted to approximated kernel representations, where no explicit high-order statistic (i.e., X^⊤X) is computed. Inspired by success of [19], we introduce a feature power normalization method, which can be regarded as transferring matrix power normalization from high-order representations to the original convolutional features. In this way, we can effectively consider geometry of high-order representations based on matrix power normalization, while avoiding computation of explicit high-order statistics. Accordingly, our Multi-scale Structural Kernel Representation (MSKR) considers geometry of high-order kernel representations by performing a feature power normalization before the polynomial kernel approximation. Besides, we embed an attention module [23] into our kernel representations for considering the importance of each convolutional feature. The attention module can jointly encode spatial and channel information.

This paper is an extension of our previous work [15]. There exist two significant differences between MSKR and our previous work (MLKP) in terms of techniques and experiments. Specifically, from technique perspective, MSKR introduces a novel feature power normalization into MLKP, which appropriately makes use of geometry of high-order statistics captured by polynomial kernel approximation of MLKP. Besides, MSKR extends the location-weight network (only spatial information is considered) of MLKP to an attention network, where the latter jointly takes spatial and channel information into consideration. From experiment perspective, we conduct much more experiments to verify the effectiveness of MSKP in terms of detectors, backbone models and tasks comparing with previous work. First, we adopt MSKR to more detectors besides Faster R-CNN [5] used in Wang et al. [15]. Second, we employ light-weight MobileNet [24] as backbone models to assess effect of MSKR in the mobile settings. Additionally, we evaluate the generalization ability of the proposed MSKR on instance segmentation task.

The overview of our proposed MSKR is illustrated in Fig. 2. Given a multi-scale feature map $\mathcal{X},$MSKR first performs feature power normalization, and then kernel representations are computed using 1 × 1 convolution operation and element-wise product. Finally, an attention module is used for re-weighting kernel representations. As shown in Fig. 1(d), MSKR can significantly improve detection performance, especially for objects with complex variations (e.g., large pose changes, blur and similar background). The experiments are conducted on three widely used benchmarks, i.e., PASCAL VOC 2007, PASCAL VOC 2012 [25] and MS COCO [26]. The contributions of this paper are summarized as follows:

• In this paper, we make an attempt to integrate high-order statistics into deep CNNs as representations for effective object detection. To this end, we propose a novel Multi-scale Structural Kernel Representation (MSKR). The proposed MSKR can preserve the spatial information while taking geometry structure of high-order statistics into account.

• To consider the geometry of our high-order kernel representations, we introduce a feature power normalization method before computation of kernel representations, approximately performing matrix power normalization on high-order representations. It can further improve the performance of kernel representations.

• Extensive experiments are conducted on three widely used object detection benchmarks, and the results show MSKR clearly improves performance of many existing deep detectors (e.g., Faster R-CNN [5], FPN [14], Mask R-CNN [27] and RetinaNet [28]), while performing favorably in comparison to the state-of-the-art methods. Besides, the results on instance segmentation task demonstrate that our MSKR has the great potential to improve performance of other dense predication task.