High-Order-Interaction for weakly supervised Fine-Grained Visual Categorization

Abstract

Fine-Grained Visual Categorization (FGVC) is a challenging task due to the large intra-subcategory and small inter-subcategory variances. Recent studies tackle this task through a weakly supervised manner without using the part annotation from the experts. Of those, methods based on bilinear pooling are one of the main categories for computing the interaction between deep features and have shown high effectiveness. However, these methods mainly focus on the correlation within one specific layer but largely ignore the high interactions between multiple layers. In this study, we argue that considering the high interaction between the features from multiple layers can help to learn more distinguishing fine-grained features. To this end, we propose a High-Order-Interaction (HOI) method for FGVC. In our HOI, an efficient cross-layer trilinear pooling is introduced to calculate the third-order interaction between three different layers. Third-order interactions of different combinations are then fused to form the final representation. HOI can produce more discriminative representations and be readily integrated with the two popular techniques, attention mechanism and triplet loss, to obtain superposed improvement. Extensive experiments conducted on four FGVC datasets show the great superiority of our method over bilinear-based methods and demonstrate that the proposed method achieves the state of the art.

Introduction

In the past few years, the performances of general image classification on large-scale datasets (e.g., ImageNet [1] and Places [2]) have achieved significant improvements, along with the development of deep neural networks (DNNs) [3], [4], [5]. Compared with the general image classification task, the Fine-Grained Visual Categorization (FGVC) is more challenging [6], [7], [8]. The FGVC needs to classify images into different subcategories that have small inter-class variations, such as, bird species [9], vehicle models [10] and aircraft types [11]. Fig. 1(a) shows the challenge of a large intra-subcategory variance and small variance among different similar subcategories through examples of bird images. As can be seen, different types of Gull look very similar, while the same sub-type of Gull can have very large variations. Human experts usually need to find the representative regions to classify an image into the true subcategory. To mimic this characteristic, a straightforward way is to train the CNN network with representative part annotations. However, annotating such regions not only requires expertise but also is time-consuming. To deal with these issues, recent studies focus on exploring the weakly-supervised FGVC [12], [13], [14], [15], in which only subcategory-level annotations are provided for training CNN models.

For constructing an accurate weakly supervised FGVC recognition method, two main concerns need to be addressed: 1) how to locate the informative regions, and 2) how to learn more discriminative features. Regarding the first concern, many methods based on region localization [16], [17], [18], [19] have been developed. These methods mainly consist of two steps, which usually first localize the object parts and then extract corresponding features from those parts for final classification. Despite their success, due to the lack of expert labeling, these methods have difficulties in defining the informative parts and deciding the optimal number of parts. Furthermore, the two-stage scheme may increase the training complexity and computation cost. On the other aspect, several attention based method [20], [21], [22] have been proposed to find the informative regions. In these methods, a specially designed attention module is used to compute the attention weights to highlight important regions. Regarding the second concern, bilinear-based methods [23], [24], [25] have been widely used. These methods concentrate on increasing the feature discrimination by modeling the underlying relationship within the feature regions through bilinear pooling. However, most of them only use the feature from the last convolution layer for computing the final feature representation. In fact, after several max pooling operations, the last convolution layer will discard a lot of local details, leading the feature insufficient to overcome the small inter-subcategory variations and significant intra-subcategory variations. To address this problem, there are some attempts [26], [22] to consider using more layers for further improving the discrimination. The most representative method is the Hierarchical Bilinear Pooling (HBP) proposed in [26]. HBP considers the three-layer feature interaction by stacking several two-layer bilinear pooling features, and their experimental results demonstrated that using more layers can significantly improve the accuracy. However, it still only considers the second-order interaction and fails to exploit the direct interaction between more layers.

To deal with the shortcomings of these methods, we argue that considering the direct correlations between multiple layers can help distinguish the small variations among different subcategories. To this end, we propose a simple but effective method, called High-Order-Interaction (HOI), for the Fine-Grained Visual Categorization task. First, we utilize an attention mechanism to construct a multi-scale feature pyramid that contains semantic information from different layers with various receptive fields. Then, we introduce a cross-layer trilinear pooling operation to compute the high-order interaction among three different layers. The proposed cross-layer trilinear pooling only involves several dot product operations, making it efficient and easy to implement. Finally, we concatenate the high-order interactions of different combinations to obtain the final feature representation. HOI enables the model to discover and focus on discriminative regions in the feature map (as shown in Fig. 1(b)) so that the final representation can be more robust to inter- and intra-subcategory variance. Finally, we leverage the cross-entropy loss and triplet loss to train the whole model.

To summarize, the main contributions of this study are as following:

• We propose a easy and efficient High-Order-Interaction (HOI) method for computing the long-range cross-layer correlations between multiple layers. The visualization of the learned feature maps shows that our proposed HOI can help to learn more discriminative representation for Fine-Grained Visual Categorization.

• We develop a framework that mainly consists of two levels by integrating the attention mechanism, High-Order-Interaction, and multi-task loss functions, which can boost the final performance.

• Extensive experiments on four fine-grained categorization benchmark datasets (i.e., CUB-200–2011, Stanford-Car, FGVC-Aircraft and Oxford-Flower102) demonstrate the effectiveness of our method, which can find discriminative regions for reaching the state-of-the-art performance.