Contrastive and consistent feature learning for weakly supervised object localization and semantic segmentation

Abstract

Weakly supervised learning attempts to construct predictive models by learning with weak supervision. In this paper, we concentrate on weakly supervised object localization and semantic segmentation tasks. Existing methods are limited to focusing on narrow discriminative parts or overextending the activations to less discriminative regions even on backgrounds. To mitigate these problems, we regard the background as an important cue that guides the feature activation to cover the entire object to the right extent, and propose two novel objective functions: 1) contrastive attention loss and 2) foreground consistency loss. Contrastive attention loss draws the foreground feature and its dropped version close together and pushes the dropped foreground feature away from the background feature. Foreground consistency loss favors agreement between layers and provides early layers with a sense of objectness. Using both losses leads to balanced improvements over localization and segmentation accuracy by boosting activations on less discriminative regions but restraining the activation in the target object extent. For better optimizing the above losses, we use the non-local attention blocks to replace channel-pooled attention leading to enhanced attention maps considering the spatial similarity. Finally, our method achieves state-of-the-art localization performance on CUB-200-2011, ImageNet, and OpenImages benchmarks regarding top-1 localization accuracy, MaxBoxAccV2, and PxAP. We also demonstrate the effectiveness of our method in improving segmentation performance measured by mIoU on the PASCAL VOC dataset.

Introduction

Weakly supervised learning (WSL) aims to train a convolution neural network (CNN) for the target tasks using only weak supervision, e.g., class label for object localization [8], [27], [46], class label [18], [22] or bounding box [9], [33] for semantic segmentation. They require cheaper annotation cost than fully supervised approaches [2], [5], [24], [25], [41]. Therefore, WSL has received significant attention over the various computer vision tasks [14], [28], [36], [40] due to the lower cost to obtain weak annotations.

We focus on the task of weakly supervised object localization (WSOL) and weakly supervised semantic segmentation (WSSS) utilizing image-level class labels. WSOL pursues both classifying and localizing the target object as a class label and a bounding box. WSSS predicts a class label for each pixel in a given image. In general, WSSS consists of two steps: making a pseudo mask as weak supervision and training segmentation network using pseudo masks. Accordingly, we extend localization to segmentation as both tasks have similar goals to solve. The localization maps extracted from our model can be employed as initial weak supervision for the WSSS task.

Zhou et al. [47] generate a class activation map (CAM) based on the classification network using a global average pooling. CAM highlights the class-specific discriminative regions from the input image. As shown in Fig. 1, it is limited to focusing on narrow discriminative parts (e.g., the face of a person) rather than covering the entire object. To relieve this problem, recent methods apply an adversarial erasing method [8], [27], [44] that erases the most discriminative parts to extend the activations to less discriminative regions. Consequently, they excessively spread out the activations even on backgrounds, which tends to over-estimate bounding boxes. These under-estimated or over-estimated activation maps harm predicting the location of the object (WSOL) or pixel-level probability map (WSSS).

In this paper, we propose four ingredients for both WSOL and WSSS tasks: 1) contrastive attention loss, 2) foreground consistency loss, 3) non-local attention block, 4) dropped foreground mask.

The contrastive attention loss pulls the foreground feature and its erased version close together and pushes the erased foreground feature away from the background feature. It encourages the learned representation to be homogeneous in the object extent and heterogeneous from the backgrounds. The foreground consistency loss favors agreement between layers and provides early layers with a sense of backgroundness. It boosts the foreground activations while restraining the background regions since low-level features are activated on locally distinctive regions, e.g., edges. Moreover, we employ the non-local attention blocks for producing better attention maps which consider the similarity between pixel locations in a feature map. It goes along with our contrastive attention loss as it assigns high weights on similar features. Last but not least, we propose a dropped foreground mask which drops the background region as well as the most discriminative region. It guides the model to avoid spreading attention to backgrounds.

Our method shows competitive performances in three benchmark datasets (CUB-200-2011 [38], ImageNet [30] and OpenImages [20]) for WSOL and PASCAL VOC 2012 [10] for WSSS.

To verify the effectiveness of the proposed method, we measure four evaluation metrics: top-1 localization accuracy, MaxBoxAccV2 [7], PxAP [7], mIoU.

This manuscript extends the conference paper [16], which only focuses on the WSOL task. We newly add (1) an extension of contrastive attention loss; (2) an extensive comparative study with existing WSOL methods using various datasets and metrics; and (3) validation of our method for the WSSS task.