Partial Multi-label Learning via Constraint Clustering

Abstract

Multi-label learning (MLL) refers to a learning task where each instance is associated with a set of labels. However, in most real-world applications, the labeling process is very expensive and time consuming. Partially multi-label learning (PML) refers to MLL where only a part of the labels are correctly annotated and the rest are false positive labels. The main purpose of PML is to learn and predict unseen multi-label data with less annotation cost. To address the ambiguities in the label set, existing popular PML research attempts to extract the label confidence for each candidate label. These methods mainly perform disambiguation by considering the correlation among labels or/and features. However, in PML because of noisy labels, the true correlation among labels is corrupted. These methods can be easily misled by noisy false-positive labels. In this paper, we propose Partial Multi-Label learning method via Constraint Clustering (PML-CC) to address PML based on the underlying structure of data. PML-CC gradually extracts high-confidence labels and then uses them to extract the rest labels. To find the high-confidence labels, it solves PML as a clustering task while considering extracted information from previous steps as constraints. In each step, PML-CC updates the extracted labels and uses them to extract the other labels. Experimental results show that our method successfully tackles PML tasks and outperforms the state-of-the-art methods on artificial and real-world datasets.

Appendix

1.1 Proof of Formula

In this section, the detail of the optimization of Eq. (1) is given. The goal of this optimization is to find the optimal value for cluster centers ($[Z]_{k\times m\times L}$) and the fuzzy membership ($[U]_{k\times m \times L}$). Where k is the number of classes for each label. Since each label is a binary class we set $k=2$. m is the size of the feature and L is the number of labels. For making the optimization procedure easier for the reader, the procedure is described for a single label. Thus we consider cluster centers ($[Z]_{2\times m}$) and the fuzzy membership ($[U]_{2\times m}$) for only one label. For the rest of the labels, we repeat the procedure. The Eq. (1) does not have a close form solution. To solve this problem an alternating optimization approach is used. Equation (1) is a constraint non-linear optimization form. By using Lagrange multipliers the following function is obtained.

$$\begin{aligned} {\begin{matrix} &{}J (U,Z) = \sum _{j=1}^{2}\sum _{i=1}^{N}U_{ij}d_{ij}(x_i,z_j) +A_1 \sum _{j=1}^{2}\sum _{i=1}^{N}U_{ij}\log U_{ij} + A_2\sum _{m|X_m \in C^i}\sum _{n| X_n \in C^i \atop m\ne n}\sum _{k=1}^{2}\sum _{p=1 \atop p\ne k}^{K}U_{mk}U_{np}\\ &{} +A_3\sum _{m|X_m \in C^i}\sum _{n| X_n \in C^j \atop i\ne j}\sum _{k=1}^{2}U_{mk}U_{nk} +\sum _{i=1}^N \lambda _i ( \sum _{j=1}^2 U_{ij}-1) \\ \end{matrix}} \end{aligned}$$

(11)

Lemma 1. The optimal value for $U_{ij}$ when Z are fixed is equal to :

$$\begin{aligned} U_{ij}= \dfrac{\exp \left( \frac{-d_{ij}(x_i,z_j)-A_2\psi _{ij}-A_3\varPsi _{ij}}{A_1}\right) }{\sum _{p=1}^2\exp \left( \frac{-d_{ip}(x_i,z_p)-A_2\psi _{ip}-A_3\varPsi _{ip}}{A_1}\right) } \end{aligned}$$

(12)

Proof. To find the optimal value for each $U_{ij}$ we take derivative of Eq. (11) respect to each $U_{ij}$ and set it to zero as follows:

$$\begin{aligned} {\begin{matrix} &{}\frac{\partial J (U,Z)}{\partial U_{ij}}=0\\ &{}d_{ij}(x_i,z_j)+A_1(1+logU_{ij})+A_2(\sum \limits _{n|X_i \in C^m, X_n \in C^p \atop m\ne p} \sum \limits _{k=1 \atop \ne j}^{2} U_{nk})+A_3(\sum \limits _{n|X_i,X_n \in C^m}U_{nj})+\lambda _i=0 \end{matrix}} \end{aligned}$$

(13)

By setting $\psi _{ij}$ and $\varPsi _{ij}$ as follows:

$$\begin{aligned} {\begin{matrix} &\psi _{ij}= \sum \limits _{n|X_i \in C^m, X_n \in C^m \atop i\ne n} \sum \limits _{k=1 \atop k\ne j}^{K} U_{nk},\quad \varPsi _{ij} = \sum \limits _{n|X_i \in C^m ,X_n \in C^p \atop m\ne p}U_{nj } \end{matrix}} \end{aligned}$$

(14)

By solving Eq. (13), $U_{ij}$ will be obtained as follows:

$$\begin{aligned} {\begin{matrix} U_{ij}=\exp (-1)\exp \left( \frac{-d_{ij}(x_i,z_j)-A_2\psi _{ij}-A_3\varPsi _{ij}}{A_1}\right) \exp (\frac{-\lambda _i}{A_1}) \end{matrix}} \end{aligned}$$

(15)

Since $\sum _{j=1}^2=1$ the Lagrange multiplier can obtained as follows:

$$\begin{aligned} {\begin{matrix} &{}\sum _{j=1}^2\exp (-1)\exp \left( \frac{-d_{ij}(x_i,z_j)-A_2\psi _{ij}-A_3\varPsi _{ij}}{A_1}\right) \exp (\frac{-\lambda _i}{A_1})=\\ &{}\exp (-1)\exp (\frac{-\lambda _i}{A_1})\sum _{j=1}^2\exp \left( \frac{-d_{ij}(x_i,z_j)-A_2\psi _{ij}-A_3\varPsi _{ij}}{A_1}\right) =1\\ &{}=>\exp (\frac{-\lambda _i}{A_1})=\frac{1}{\exp (-1)\sum _{j=1}^2\exp \left( \frac{-d_{ij}(x_i,z_j)-A_2\psi _{ij}-A_3\varPsi _{ij}}{A_1}\right) } \end{matrix}} \end{aligned}$$

(16)

By substituting Eq. (16) in Eq. (15) the closed form solution for uij (Eq. (12)) will be obtained and completes the proof of lemma.

Lemma 2. If the U (fuzzy memberships) are fixed, the optimal value for Z (cluster centers) are equal to equation (17).

$$\begin{aligned} Z_{jp}=\frac{\sum _{i=1}^{N}U_{ij}x_{ip}}{\sum _{i=1}^{N} U_{ij}} \end{aligned}$$

(17)

Proof. Again the alternative approach is used. First, The U are fixed then the optimal values for Z is obtained by taking derivative of Eq. (11) respect to each cluster center and set it to zero.

$$\begin{aligned} {\begin{matrix} &\frac{\partial J (U,Z)}{\partial Z_{jp}}=0\quad => \sum _{i=1}^{N}2U_{ij}(Z_{jp}-x_{ip})=0\quad => Z_{jp}=\frac{\sum _{i=1}^{N}U_{ij}x_{ip}}{\sum _{i=1}^{N} U_{ij}} \end{matrix}} \end{aligned}$$

(18)

Lemma 3. U and Z are local optimum of J(U, Z) if $Z_{ij}$ and $U_{ij}$ are calculated using Eq. (17) and (12) and $A_1,A_2,A_3 >0$

Proof. Let J(U) be J(U, Z) when Z are fixed, J(Z) be J(U, Z) when U are fixed and $A_1,A_2,A_3 >0$. Then, the Hessian H(J(Z)) and H(J(U)) matrices are calculated as follows:

$$\begin{aligned} {\begin{matrix} h_{fg,ij}(J(U))=\frac{\partial }{\partial _{fg}} \frac{\partial J(U)}{\partial U_{ij}}= \Bigl \{ _{0, \qquad \quad otherwise}^{\frac{A_1}{U_{ij}},\qquad if f=i,g=j} \end{matrix}} \end{aligned}$$

(19)

$$\begin{aligned} {\begin{matrix} h_{fg,il}(J(Z))=\frac{\partial }{\partial _{fg}} \frac{\partial J(Z)}{\partial Z_{ip}}= \Bigl \{ _{0, \quad \quad \qquad otherwise}^{\sum _{j=1}^{2}2U_{ij},\quad if f=i,g=p} \end{matrix}} \end{aligned}$$

(20)

Equation (19) and (20) shows H(J(Z)) and H(J(U)) are diagonal matrices. Since $A_1>0$ and $0< U_{ij} \le 1$, the Hessian matrices are positive definite. Thus Eq. (12) and (17) are sufficient conditions to minimize J(U) and J(Z).

1.2 Additional Excremental Result

Tables 4,5,6 show the performance of our proposed method in the term of ranking loss and coverage respectively.

Table 4. The result of our proposed method and other competitors in term of ranking loss on real-world and synthetic datasets (mean±standard deviation)

Table 5. The result of our proposed method and other competitors in term of one error on real-world and synthetic datasets (mean±standard deviation).

Table 6. The result of PML-CC and other competitors in term of coverage on real-world and synthetic datasets (mean±standard deviation).