Rethinking semantic-visual alignment in zero-shot object detection via a softplus margin focal loss

Abstract

Zero-shot object detection (ZSD) aims to locate and recognize novel objects without additional training samples. Most existing methods usually map visual features to semantic space, resulting in a hubness problem, and learning an effective feature mapping between the two modalities remains a considerable challenge. In this work, we propose a novel end-to-end framework, Semantic-Visual Auto-Encoder (SVAE) network, to tackle the above issues. Distinct from previous works that utilize fully-connected layers to learn the feature mapping, we implement a 1-dimensional convolution with various shared filters to construct the auto-encoder, which maps semantic features to visual space to alleviate the hubness problem. Specifically, we design a novel loss function, Softplus Margin Focal Loss (SMFL), for object classification channel to align the projected semantic features in visual space and address the class imbalance problem. The SMFL improves the discrimination of projections on positive and negative categories and maintains the property of focal loss. Besides, to promote the localization performance for novel objects, we also provide semantic information for object localization channel and utilize a trainable matrix to align the semantic-visual mapping, considering noises in semantic representations. We conduct extensive experiments on four challenging benchmarks. The experimental results show the competitive performances compared with state-of-the-art approaches. Especially, we achieve 8.39%/6.58% mean average precision (mAP) improvements for ZSD/general-ZSD on Microsoft COCO benchmark.

Introduction

Deep learning has achieved significant success in object detection. However, existing detection models [1], [2], [3], [4], [5] usually rely on large-scale object datasets [6], [7], [8] with fully-annotated locations and categories. It is hard to apply them in the scenarios where data is scarcely labeled from novel categories. One solution is to collect a larger dataset with a wider set of categories, which can be quite laborsome and time-consuming.

Recently, zero-shot object detection (ZSD) [9], [10], [11], [12], [13] has shown an elegant way to address the problem. In this task, a ZSD model is typically trained on an instance set of so-called seen classes and aims to detect the instances from unseen classes during testing. Different from zero-shot recognition (ZSR), the model can not only recognize instances from unseen classes but also localize them. An intuitive and common solution is transferring the knowledge from seen to unseen classes in a shared space through a semantic representation transformation. The representations are usually acquired in the forms of encoded class attributes [14], textual descriptions [15], word vectors [10], [14], [16], [17], etc.

Though the ZSD techniques [9], [13], [14], [16], [18] have made impressive progress in the past few years, there exist some issues: (a) Several approaches [12], [17], [18] map the feature representations from visual to semantic space, resulting in hubness problem ¹ [19], [20]. (b) These existing ZSD models directly learn a projection from visual to semantic space without any constraints. It may lead to misplacing the projection of features from unseen categories during testing. (c) Fully connected (FC) layers without any constraints used in [9], [10], [16], [17], [18] result in high computational complexity. It is hard for models to optimize it and get semantic features well represented in visual space.

To address the above issues, we present a novel end-to-end framework named Semantic-Visual Auto-Encoder network for ZSD task (SVAE-ZSD). Firstly, the encoder projects semantic representations of class labels into the visual space of regions of interest (ROIs). As pointed out in [20], this mapping mechanism can mitigate the hubness problem. Secondly, the decoder projects the predicted visual features back to the semantic space, and $L_2$ regularization is applied in this auto-encoder structure. They constrain the projection to improve the compatibility and robustness of the learned model. Thirdly, for the auto-encoder structure, we use a 1-dimensional (1D) convolution operation proposed in [21] to realize the semantic-visual mapping. Comparing with different weights for each class in FC layers, the shared filters for each class in the convolution reduce the computational complexity. It is simple yet effective and makes contributions to making semantic features well represented for the visual space. Additionally, to align the semantic-visual mapping in the classification subnet, we further propose a softplus margin focal loss, keeping the ability of focal loss to deal with class imbalance problem. The loss function forces the mapping mechanism to maximize the projections of semantic features on positive categories and minimize them on negative categories. It offers the model an ability to distinguish the foreground from the background and detect unseen objects. Furthermore, the semantic information is also applied for the box regression subnet to locate unseen objects. Due to high noises in the semantic vectors, we implement a trainable matrix rather than an element-wise multiplication to get a better synergy between the semantic and the visual space.

The main contributions of this work are summarized as follows:

• We present a novel end-to-end framework, a semantic-visual auto-encoder network based on 1-dimensional convolution, for ZSD task to get semantic features well represented in visual space as well as mitigate the hubness problem. It is simple yet effective.

• We design a softplus margin focal loss function to align the semantic and visual features in the classification subnet. It helps to distinguish the projections of semantic features on positive categories from those on negative categories with margins and relieves the confusion between unseen objects and the background.

• Extensive experiments are carried on four challenging datasets. Our proposed method outperforms state-of-the-art methods with significant margins. The model especially achieves over 6% mAP improvements on Microsoft COCO [7] dataset in ZSD/GZSD setting.

The rest of the paper is organized as follows: Section 2 reviews the related works, followed by Section 3, which describes the proposed approach. The detailed SMFL and other loss functions for the model are explained in Section 4. Experimental evaluation analysis is reported in Section 5, and the conclusions are presented in Section 6.