Dual attention interactive fine-grained classification network based on data augmentation

Highlights

• Our method is convenient to obtain rich global image overall attributes.

• We use the global attention of dual attention mechanism to lock the target area.

• We employ interactive fusion of channels to distinguish similar images quickly.

Abstract

The key to fine-grained image classification is to find discriminative regions. Most existing methods only use simple baseline networks or low-recognition attention modules to discover object differences, which will limit the model to finding discriminative regions hidden in images. This article proposes an effective method to solve this problem. The first is a novel layered training method, which uses a new training method to enhance the feature extraction ability of the baseline model. The second step focuses on key regions of the image based on improved long short-term memory (LSTM) and multi-head attention. In the third step, based on the feature map obtained by the dual attention network, spatial mapping is performed by a multi-layer perceptron (MLP). Then the element-by-element mutual multiplication calculation of the channel is performed to obtain a feature map with finer granularity. Finally, the CUB-200-2011, FGVC Aircraft, Stanford Cars, and MedMNIST v2 datasets achieved good performance.

Introduction

Fine-grained visual classification aims to identify very similar images, often subclasses under a large class. This is a more challenging problem than traditional classification due to the inherent little intra-object variation between sub-categories.

Many early classification works StdDev [1] and PS-CNN [2] usually rely on additional annotations to accurately find discriminative regions. However, it is too cumbersome and often prone to errors, resulting in poor results. Therefore, in recent years, most fine-grained research directions have turned to weakly supervise training models, which only rely on the given classification labels for end-to-end learning [3], [4], [5], [6], [7], [8]. In reality, it is difficult to distinguish between the common kingfisher and the great spotted kingfisher; they are very similar in size. However, the ear feathers of the common kingfisher are orange–yellow, and the ear feathers of the great spotted kingfisher are blue. At the same time, the difference between male and female birds of the same species is that the beak is black, while the lower beak of the female is red. It can be seen that subtle differences tend to be concentrated in local areas of objects. So far, most feasible solutions, such as MACNN [9], Cross-X [10], are to rely on neural networks to extract image feature maps, locate local feature regions, and then perform finer learning based on local regions. In this way, subtle differences between two similar objects can be better detected. It is also essential to find the most discriminative local regions for the many local feature regions obtained by localization. For example, in bird classification, the most critical local regions are usually the head, claws, and beaks. However, for some very similar birds, the only area that differs may be the head area. This requires us to learn multiple local regions more deeply and focus on the correct discriminative regions.

Many recent fine-grained classifications tend to amplify localized feature regions WS-DAN [11],MMAL-Net [12]. While this allows better learning of discriminative features of images, it does not consider how features from different enlarged parts are fused together in a synergistic manner. To sum up, fine-grained classification should not only learn the discriminative regions of the image but also increase the feature extraction ability of the baseline network or the good localization of the target region and strengthen the connection of various local features. In this way, more hidden discriminative features can be discovered. Therefore, it is as described in Fig. 1, we propose a dual attention interaction network based on data augmentation and evaluate the performance of the model on three commonly used fine-grained data sets: CUB-200-2011 [13], FGVC Aircraft [14] and Stanford Cars [15]. In addition, experiments were performed on a biomedical dataset MedMNIST v2 [16]. A large number of experimental results show that our model has achieved high performance compared with the state-of-the-art methods. Three innovation points of this article are shown, as follows:

• We propose a cross-layer channel data augmentation framework. This is different from the recent paper SnapMix [17], which directly performs feature fusion processing at the attribute level of the original data. Our data enhancement is based on the training of the residual network layer. The features of each layer are fused with the features before that layer. Hierarchical training can make each neural network layer play the best role. The high-level convolution contains more semantic features, and the bottom-level convolution contains more field of view information, which will avoid feature loss.

• We propose a local attention region framework that uses dual attention networks to localize more local feature regions. Through the LSTM [18] module, the corresponding global features are cyclically focused in turn, and then the attention feature maps are weighted and fused to aggregate the details into representative complementary vectors. Similarly, the multi-head attention [19] module is used to aggregate the extracted attention feature maps. It uses linear conversion processing between Q, K, and V channels, which is equivalent to multiple independent attention calculations, and then performs feature maps. Weighted feature processing. Effectively increase the diversity of image key regions.

• We propose a channel interaction architecture. First, the attention features extracted by dual attention can filter some noise through MLP, perform spatial feature mapping and dimensionality reduction, and generate two more fine-grained features. Then, based on the features extracted by dual attention and the features generated by MLP, the channel multiplication is performed element by element, step by step. Finally, this is equivalent to amplifying the learned features, which is convenient to find rich discriminative features.

The rest of this article is organized as follows. Section 2 briefly introduces related work, while Section 3 provides a detailed overview of the proposed method, in Section 4, performance analysis, ablation experiment, and attention visualization are performed on three data. The fifth chapter discusses the content of the complete text, and the sixth section summarizes the research conclusions.