Context-aware co-supervision for accurate object detection

Highlights

• We advocate the importance of equipping two-stage detectors with top-down signals, in order to which provides high-level contextual cues to complement low-level features. In practice, this is implemented by adding a side path in the detection head to predict all object classes in the image, which is co-supervised by image-level semantics and requires little extra overheads.

• Our research reveals the usefulness of combining top-down and bottom-up signals in object detection, and we believe it to be generalized to other tasks. The simplicity and originality of our approach leave much room for future research, in which we will append more powerful modules to enhance contexts and other cues for visual recognition.

Abstract

State-of-the-art object detection approaches are often composed of two stages, namely, proposing a number of regions on an image and classifying each of them into one class. Both stages share a network backbone which builds visual features in a bottom-up manner. In this paper, we advocate the importance of equipping two-stage detectors with top-down signals, in order to which provides high-level contextual cues to complement low-level features. In practice, this is implemented by adding a side path in the detection head to predict all object classes in the image, which is co-supervised by image-level semantics and requires little extra overheads. Our approach is easily applied to two popular object detection algorithms, and achieves consistent performance gain in the MS-COCO dataset.

Introduction

Object detection is a challenging task in computer vision. Recently, with the development of deep learning especially convolutional neural networks, researchers have designed an effective two-stage framework for object detection [1], [2], [3]. Based on a powerful network backbone, these methods first use a proposal stage to extract a number of regions as probable objects, and then apply an individual classifier stage to distinguish the object class within each region.

In another point of view, there are two kinds of approach for understanding the process of perception. Top-down processing is defined as the development of pattern recognition through the use of contextual information. For instance, you are presented with a paragraph written with difficult handwriting. It is easier to understand what the writer wants to convey if you read the whole paragraph rather than reading the words in separate terms. The brain may be able to perceive and understand the gist of the paragraph due to the context supplied by the surrounding words. In the bottom-up processing approach, perception starts at the sensory input, the stimulus. Thus, perception can be described as data-driven. For example, there is a flower at the center of a person’s field. The sight of the flower and all the information about the stimulus is carried from the retina to the visual cortex in the brain. We argue that such guidance is mostly context-free and thus not aware of high-level semantic cues in its surrounding areas. When a number of small proposals are extracted from a complex scene, the corresponding features computed from a small fraction of the feature map are less capable of accurately determining the object classes. With less confident scores, these small objects can be suppressed by large objects and thus mis-detected. Consequently, we propose a context-aware module, which extracts top-down signals to assist bottom-up signals for object detection. An example is shown in Fig. 1.

To assist object detection with richer high-level semantic information, we propose an efficient module which introduces a context-aware module to the detection network. This is achieved by adding an extra classification path to determine which object classes appear in the entire image. This is to say, the features built up in the backbone are expected to have the ability of depicting both each individual proposal and the entire image, which the latter provides top-down information to refine the former outputs. In practice, we propose a global context encoding (GCE) module to summarize all features, and simultaneously feed these global features to complement the features pooled within each proposal. There are two sources of supervision, i.e., co-supervision. The first one lies in the conventional way of classification and regression within each proposal, and the second one is produced by the image-level classification which is obtained by adding a linear layer beyond the GCE features. There is a deep supervision of image-level classification which designed as modified binary cross-entropy (BCE) loss and will be introduced in Section 3. As we can see in Fig. 2, firstly we summarize regional features and send them for image-level classification; secondly, fuse the trained features with ROI-pooled features within each proposal for enhancing regional classification and regression. Our approach is easy to implement and generalizes well to a wide range of detection frameworks, including some popular add-ons such as feature pyramid network (FPN) [4] which enables multi-scale feature extraction. A detailed flowchart can be found in Fig. 3.

Experiments are performed on the MS-COCO dataset [5] which contains a lot of small and medium objects in complex image contents. Two popular baselines are considered, namely, Faster R-CNN [2] and Mask R-CNN [3], both of which are equipped with FPN [4]. Our approach, requiring less than $10\%$ extra time and memory costs, achieves consistent accuracy gain, e.g., $1.01\%$ AP improvement and $1.85\%$ [email protected] improvement beyond Mask R-CNN. Diagnostic experiments suggest that our approach is especially effective in finding small and difficult objects. This work paves the way of future research in combining top-down semantic-aware information with bottom-up regional features, which is also applicable to a wide range of vision tasks.

The remainder of this paper is organized as follows. Section 2 briefly reviews prior work, and Section 3 describes our approach. After experiments are shown in Section 4, we conclude this work in Section 5.