Multi-instance multi-label image classification: A neural approach

Abstract

In this paper, a multi-instance multi-label algorithm based on neural networks is proposed for image classification. The proposed algorithm, termed multi-instance multi-label neural network (MIMLNN), consists of two stages of MultiLayer Perceptrons (MLP). For multi-instance multi-label image classification, all the regional features are fed to the first-stage MLP, with one MLP copy processing one image region. After that, the MLP in the second stage incorporates the outputs of the first-stage MLPs to produce the final labels for the input image. The first-stage MLP is expected to model the relationship between regions and labels, while the second-stage MLP aims at capturing the label correlation for classification refinement. Error Back-Propagation (BP) approach is adopted to tune the parameters of MIMLNN. In view of that traditional gradient descent algorithm suffers from long-term dependency problem, a refined BP algorithm named Rprop is extended to effectively train MIMLNN. The experiments are conducted on a synthetic dataset and the Corel dataset. Experimental results demonstrate the superior performance of MIMLNN comparing with state-of-the-art algorithms for multi-instance multi-label image classification.

Introduction

With the prevalence of digital imaging devices such as digital cameras and mobile phones, a huge volume of images are produced every day, as a result of which image classification becomes increasingly important. Traditional image classification approaches classify an image to only one class, which may be impractical since in real applications an image could be associated with multiple labels. Furthermore, those traditional approaches usually treat an image as an instance and represent the image using global features only. The issue is that global features cannot well characterize the image contents, in particular when the images are composed of complex objects [1], [2]. In view of this, many multi-instance learning (MIL) [3], [4], [5] and multi-label learning (MLL) [6], [7], [8] algorithms are proposed to tackle image classification tasks.

In MIL, an image is a bag of instances, with each instance representing an image region. As informative regional features are utilized, this approach characterizes images with complex contents quite well. However, MIL targets at binary classification. It assigns an image with one label only. By contrast, MLL deals with multi-label tasks, and it is able to associate an image with multiple labels. But MLL regards an image as a single instance. The regional features are not employed. The issue is that the labels of an image usually relate to different regions. The use of global features is not powerful to discriminate the image labels. Therefore, some researchers have developed multi-instance multi-label learning (MIMLL) based on MIL and/or MLL for multi-label image classification [9], [10], [11].

For MIMLL image classification, each image is composed of a bag of regions and associated with multiple labels. In the learning phase, the relationship between the image regions and labels is unknown. The aim of learning is to figure out the relationship between the regions and labels from the training images, and then the learned relationship can be used to classify unlabeled images. Zhou et al. [9] proposed MIMLBoost and MIMLSVM for multi-instance multi-label scene classification. MIMLBoost transforms a MIML learning task into MIL problem and solve the MIL problem by using MIBoost [12], while MIMLSVM transforms a MIML learning task into MLL problem and tackle the MLL problem by adopting MLSVM [6]. The disadvantage of MIMLBoost and MIMLSVM is that both of them do not take label correlation into account.

Due to their powerful classification capability, neural networks have been widely applied to image classification and pattern recognition [13], [14], [15], [16], [17], [18], [19]. In this paper, we propose an approach based on neural networks for multi-instance multi-label image classification. The proposed approach is termed Multi-Instance Multi-Label Neural Network (MIMLNN). MIMLNN is composed of two stages of MultiLayer Perceptrons (MLP). The first-stage MLP receives the regional features of the input image, with one MLP copy handling one region. The second-stage MLP incorporates the outputs of the first-stage MLPs to generate the final labels for the input image. In the training phase, the first-stage MLP is expected to establish the relationship between the image regions and labels, and the second-stage MLP aims at capturing the label correlation for classification refinement. There are three reasons for us to choose neural networks for label correlation extraction. First, neural network models label correlation using nonlinear function, which is different from many other linear label correlation modeling methods. The nonlinear function is able to characterize more complex label co-occurrence relationship. Second, the label correlation information is stored in neural network weights. Based on the weights, it is possible to explicitly analyze the correlation values of different labels. Third, based on this work, in the future research, we would like to build a system that could feed label correlation values back to the input for performance enhancement. Neural networks are suitable for this task, as the outputs are easy to be fed back just by connecting them to the input. Error Back-Propagation (BP) algorithm is adopted to train MIMLNN. A traditional BP algorithm is gradient descent. Gradient descent algorithm updates neural weights according to the partial derivative magnitude of a predefined error function. Since the error back-propagates by multiplying the derivative of the sigmoid function, the value of which is between 0 and 1, the gradient magnitude could be very small for deep layers when the error back-propagates through multiple layers. As a result, the weights at very deep layers are difficult to update. This is called long-term dependency problem [20], [21]. To solve this problem, we extend the refined BP algorithm Rprop [22] for MIMLNN training. Rprop algorithm updates weights depending on the derivative sign rather than derivative magnitude of the error function. Therefore, the problem of long-term dependency can be solved. To demonstrate the superior performance of MIMLNN, the experiments are conducted on a synthetic image dataset and the popular Corel image dataset. One main advantage of synthetic dataset is that the contents of synthetic images are controllable. For example, we can design specific label correlation for our research.

The rest of this paper is organized as follows. Section 2 describes the architecture and training algorithms of the proposed MIMLNN. The long-term dependency problem is also discussed. The experimental results on the synthetic dataset and Corel dataset are reported in Section 3. Section 4 concludes this paper with final remarks. Before ending this introductory section, it is worth mentioning the contributions of this paper as follows:

• We propose a unique structure based on classical MLPs for image classification. We provide a neural network solution for the multi-instance multi-label learning problem.

• We find out an effective way to train the unique network. We extend a refined BP algorithm for the network training.

• We demonstrate the superior performance of the proposed neural structure by comparing with state-of-the-art algorithms.