Single-shot weakly-supervised object detection guided by empirical saliency model

Abstract

Even though weakly-supervised object detection (WSOD) has become an effective method to relieve the heavy work of labeling, there are still difficult problems to be solved. WSOD method represented by a Multiple Instance Learning (MIL) have some common problems including running slowly and focusing on discriminative parts rather than the whole object, which will lead to false detection. To improve the efficiency and accuracy, we propose a single-shot weakly-supervised object detection model guided by empirical saliency model (SSWOD). As human vision always focuses on the most attracting parts of the image, saliency maps can usually guide our model to locate the most promising object areas. By this way, our model takes the saliency areas as pseudo ground-truths to realize the WSOD task with only class labels. Moreover, empirical saliency is designed to refine the pseudo ground-truth and improve the detection. Our new framework not only realizes a one-step detection without region proposals, but also reduces computational consumption. Experiments on PASCAL VOC 2007 & 2012 benchmarks demonstrate that SSWOD is 8 times faster and 5 times smaller than previous approaches, surpassing the state-of-the-art WSOD methods by 6.1% mean average precision (mAP).

Introduction

Many algorithms of object detection have emerged [1], [2], [3], [4], [5] due to the development of Convolutional Neural Networks (CNNs) [6], [7]. Thanks to the use of large scale datasets with instance-level annotations, these methods can achieve high accuracy in different tasks such as military reconnaissance, face detection [8], and medical diagnosis [9]. The instances in training images are labeled with both classes and locations. High-accuracy algorithms even explore the relationship between objects to boost the performance [10], [11]. Correlation filters are applied to recent advanced methods [12] to combine spatial features with semantic information. However, these advanced methods all rely on refined annotations, which need intensive labor and financial support. Moreover, it is extremely hard to obtain labels in some special tasks. Thus, Weakly-Supervised Object Detection (WSOD) is becoming an important and promising topic. WSOD refers to the method of training the detection network by images only labeled with locations. Through this way, the labeling cost can be saved, and the training data can be easily expanded. Therefore, increasing researchers are diving into the field of WSOD, that is, training CNN only with image-level labeled images.

Most researchers have made progress on WSOD through Multiple Instance Learning (MIL) [13], [14], [15], [16], [17], [18], but there is still a huge gap (about 30% mAP) in performance with fully-supervised algorithms. Moreover, during training and inference, MIL-based methods need region proposals provided by external algorithms such as Selective Search (SS) [19] or Edge Boxes (EB) [20], which also take long. Also, as shown in Fig. 1, MIL-based methods tend to extract the most obvious features of the object, which usually causes mistakes. Other researchers have tried to use pre-trained models and image-level annotations to generate pseudo ground-truth for training. However, these methods do not complete the end-to-end training, and the inference speed cannot meet the real-time performance.

In this paper, we focus on WSOD problems. To overcome the shortages in computational cost and accuracy of MIL-based approaches, a new framework besides MIL should be proposed, so that we put forward a single-shot weakly-supervised object detection model guided by empirical saliency model (SSWOD). CNNs have strong capability in feature extraction and generalization, and the inference speed is also outstanding under Graphics Processing Unit (GPU) acceleration. So, an end-to-end CNN can perform better in both accuracy and speed than classic machine learning approaches. Therefore, we construct pseudo ground-truth through empirical saliency to avoid MIL, prevent the time consuming prepossessing, and fully utilize CNN for training and inference.

As for a WSOD problem, we can obtain class information from annotations, so how to finish accurate detection without precise location label is the key to the problem. In natural images, the attention mechanism of the human eye determines that most targets are located near the center of an image. Based on this prior, we novelly introduce saliency detection to the field. Through pre-trained saliency models, we can obtain the rough location of the targets. However, human visual attention is not sensitive to size. Even though the position of the object obtained by saliency is relatively accurate, the size of it is often slightly larger than it really is. Thus, directly using saliency information as a pseudo ground-truth will endanger the performance of the model. Therefore, we tried to obtain empirical information through the statistics of a small amount of data in order to modify the pseudo ground-truth constructed by the pre-trained model.

Our contributions can be summarized as follows:

• We build a weak supervised target detection framework based on human visual mechanism, using saliency regions as the guidance to locate the area of the candidate targets. Empirical pseudo ground-truths are introduced for training to prevent the problem that MIL-based methods are easy to focus on part of the target.

• We design a lightweight SSWOD network to achieve end-to-end training and inference. The proposed network has two branches for pseudo ground-truth generation and object detection respectively. Therefore, labeling and detection can be done in a single network during training. A fast single-shot detection can be achieved during inference.

• Our SSWOD network surpasses state-of-the-art WSOD methods by 6.1% mAP on PASCAL VOC 2007 & 2012 benchmarks. It also greatly improves the efficiency up to 8 times, and realizes real-time detection by avoiding the repetitive iterations in MIL-based methods.