Continuous label distribution learning

Highlights

• We propose a novel LDL method named CLDL which can utilize the continuous label distribution information to conduct models.

• We describe labels as a continuous distribution in the latent space, where only a few parameters require to be learned.

• We propose an effective and scalable strategy for learning continuous label distribution based on theoretical analysis.

• We systematically analyze the CLDL method. The analysis illustrates the superiority of CLDL to the existing LDL algorithms.

Abstract

Label distribution learning (LDL) is a suitable paradigm to deal with label ambiguity through learning the correlations among different labels. Most existing label distribution learning methods consider the labels to be discrete and directly establish the mapping from features to labels. However, in many real-world applications, labels naturally form a continuous distribution, which is ignored by the existing methods. As a result, the distribution information of labels can not be accurately described and finally affects the whole learning system. The goal of this paper is to propose a novel approach which can capture the continuous distribution of different labels explicitly and effectively. Specifically, we propose Continuous Label Distribution Learning (CLDL) which describes labels as a continuous density function and learns the distribution information of the labels in the latent space. In this way, the high-order correlations among different labels can be effectively extracted and only a few parameters for describing the continuous distribution need to be learned. Extensive description degree prediction experiments on real-world datasets validate the superiority of CLDL over the existing approaches.

Introduction

Learning with ambiguity is a hot topic in recent machine learning and data mining research. A learning process is essentially building a mapping from the instances to the labels [1]. A classical paradigm to deal with label ambiguity is multi-label learning [2]. Multi-label learning studies the problem where each example is represented by a single instance while associated with a set of labels simultaneously, and the task is to learn a multi-label predictor which maps an instance to a relevant label set [3], [4]. Essentially, multi-label learning considers the relation between the instance and the label to be binary, i.e., whether or not the label is relevant to the instance.

At present, there are a variety of real-world tasks that instances are involved with labels with different importance degrees [5], [6], [7]. Although multi-label learning solves the problem “Which labels describe this instance?”, it does not address the more general issue of label ambiguity — “How do these labels describe this instance?”. Consequently, a soft label instead of a hard one seems to be a reasonable solution. Inspired by this, recently a novel learning paradigm, Label Distribution Learning (LDL), has been proposed. LDL aims at learning the relative importance of each label involved in the description of an instance, i.e., a distribution over the set of labels. Formally speaking, given an instance $\mathit{x}$, LDL assigns each $y\in \mathcal{Y}$ a real value $d_{\mathit{x}}^y$ (label description degree), which indicates the importance of $y$ to $\mathit{x}$. To make the definition proper, [1] suggests that $d_{\mathit{x}}^y\in [0,1]$ and ${\sum }_{y\in \mathcal{Y}}{d}_{\mathit{x}}^y=1$, and the real value function $d$ is called the label distribution function. Since LDL extends the supervision from binary to label distribution, which is more applicable for real-world scenarios.

Due to the utility of dealing with ambiguity explicitly, LDL has been extensively applied in varieties of real-world problems [8], [9], [10], [11]. Many attempts have been done to develop LDL methods in each facet. SA-IIS [7], SA-BFGS [12] and EDL [6] directly optimize the distance between the ground truth and the predicting distribution. SCE-LDL [13] adds the sparsity constraints into the objective function to ameliorate the LDL model. DLDL [14] introduces deep learning into LDL to solve various tasks in computer vision. LDL Forests [15] adopts differentiable decision trees in the LDL model. Besides, many algorithms have been proposed to exploit label correlations in LDL. EDL-LRL [16] exploits local label correlation by capturing low-rank structure on clusters of samples with trace-norm regularization. LALOT [17] casts the label correlations exploration as a ground metric learning problem. LDLSF [18] exploits the label correlations and learns the common features for all labels and specific features for each label simultaneously. GD-LDL-SCL [19] and Adam-LDL-SCL [20] encode the influence of local samples and design a local correlation vector as the additional features for each instance to utilize the label correlations on local samples, which have proven empirically successful.

As shown in Fig. 1, in many real-world applications, e.g., age estimation, labels naturally form a continuous distribution. Taking a close look at Fig. 1(a), one may find that the faces at the neighboring ages look quite similar. This results from the fact that aging is a slow and gradual process. In this sense, although the chronological age is unambiguous, the facial appearance age is ambiguous, where the label information can naturally be considered to form a continuous distribution [7]. However, due to the difficulty of obtaining and leveraging the continuous distribution, existing label distribution learning algorithms, instead, regard the labels as a discrete distribution and directly establish the mapping from features to discrete labels, which ignores the natural continuous characteristics in the labels. As a result, the distribution information of labels can not be effectively extracted, which leads to the degradation of effective information that LDL models can utilize. If the label distribution information can be utilized without losing precision, it can be beneficial to the whole learning system. Therefore, it is essential to effectively utilize the observed data to extract and describe the continuous label distribution information and integrate it into the LDL model.

In light of the above observations, we propose Continuous Label Distribution Learning (CLDL), which utilizes the continuous label distribution hidden behind the given label information to conduct the LDL model. Specifically, CLDL describes labels as a continuous distribution and learns the distribution in the latent space. CLDL first converts discrete label distributions into continuous density functions through the label encoding process. Then CLDL establishes the mapping from instances to continuous density functions. Finally, the predicted label distribution can be obtained through the decoding process. In this way, LDL models can be trained accurately and effectively integrate the label correlation information. In the meantime, only a few parameters for describing the continuous distribution need to be learned to extract high-order correlations among different labels. Additionally, a practical algorithm for CLDL is explored.

In summary, the main contributions of this paper can be stated as follows:

• We propose the continuous label distribution learning (CLDL) method. CLDL is a novel label distribution learning method which can utilize the continuous label distribution information to conduct the LDL model.

• We theoretically describe labels as a continuous distribution in the latent infinite-dimensional space, where only a few parameters require to be learned.

• We propose an effective and scalable strategy for learning the continuous label distribution based on the theoretical analysis.

• We theoretically prove the feasibility of continuous label distribution and systematically analyze the CLDL method. The analysis illustrates the superiority of CLDL to the existing LDL algorithm.

Other chapters of the paper are organized as follows: Section 2 presents some related works of CLDL. Section 3 introduces the proposed CLDL method and gives analyses for CLDL. Section 4 provides the empirical results of the proposed CLDL. Section 5 illustrates the conclusions and the prospects of CLDL.