Learn structured analysis discriminative dictionary for multi-label classification

Abstract

Multi-label learning is a machine learning classification problem, in which an example belongs to more than one classes at the same time. Recently, multi-label learning has aroused a great deal of attention, and has achieved great success in the fields of text and image classification. In this paper, we propose a new method for multi-label learning, which is named as analysis discriminative dictionary learning for multi-label classification (ADML). We first incorporate analytical discrimination dictionary learning and sparse representation into multi-label classifier to obtain a unified model. The incoherence promoting term and reconstruction error for each label are minimized to obtain the dictionary. We then incorporate an analysis inconsistency promotion term into the model, which minimizes the reconstruction error of the dictionary with the corresponding label of the data. Further, we calculate a linear classifier by taking the label relationships into account. It is worth noting that we implicitly consider the label relationships in the analysis dictionary and linear classifier. Finally, we conduct experiments on 15 datasets to test the performance of the proposed ADML method and baselines. The results show that the proposed ADML method can deliver higher performance than previous multi-label methods.

Appendix:

Proof Proof of Theorem 1

The (11) is solved by the Lagrange dual function, the expression shows as follows:

$$ \begin{aligned} g(\mu)=&inf\{\| X_{l}-D_{l}S_{l} \|_{F}^{2} +\alpha \| D_{l}\overline{S_{l}}\|_{F}^{2} \\ &+ \sum\limits_{i=1}^{k} \mu_{l,i} (\| d_{v}\|_{2}^{2} -1)\}, \end{aligned} $$

(29)

where μ_l,i is the Lagrange multiplier. A diagonal matrix E_l ∈ R^k×k is constructed, where (E_l)_ii = μ_l,i denotes the diagonal entry, then we rewrite the formula as follows:

$$ \begin{aligned} L(D_{l},\mu)=&\| X_{l}-D_{l}S_{l} \|_{F}^{2} +\alpha \| D_{l}\overline{S_{l}}\|_{F}^{2} \\ &+tr(D^{T}DE)-tr(E_{l}). \end{aligned} $$

(30)

Let the derivative $\frac {\partial L(D_{L},\mu )}{\partial D_{L}}$ be zero, the closed-form solution for D_l is obtained as follows:

$$ \begin{aligned} D^{*}=X_{l}{S_{l}^{T}}(S_{l}{S_{l}^{T}} +\gamma\overline{S_{l}}\overline{S_{l}}^{T} +E_{l})^{-1}. \end{aligned} $$

(31)

E_l is discarded based on the work in [60], which can reduce the computational complexity and decrase the computation cost. Notely, $S_{l}{S_{l}^{T}}+\gamma \overline {S_{l}}\overline {S_{l}}^{T}$ can not be ensured to be invertible, so $S_{l}{S_{l}^{T}}+\gamma \overline {S_{l}}\overline {S_{l}}^{T}$ may produce the singular issue. Therefore, similarly as [61], a regularization term 𝜃I (𝜃 = 10e^− 4 is a small number) is embedded into $S_{l}{S_{l}^{T}}+\gamma \overline {S_{l}}\overline {S_{l}}^{T}$, which can avoid the singular problem and achieve a stable performance. □

Proof Proof of Theorem 2

The Lagrange function of this constrained problem (17) is,

$$ \begin{aligned} \mathcal{L}(P)=& \tau \|P_{l}X_{l}-S_{l}\|_{F}^{2}+\tau\|P_{l}\overline{X_{l}}\|_{F}^{2} \\ &-{\sum}_{i=1}^{N} p_{l}\xi_{l}-{\sum}_{i=1}^{N} q_{l}\{[M_{l}\cdot P_{l}X_{l}+\delta_{l}]{Y_{l}^{T}}-1+\xi_{l}\}, \end{aligned} $$

(32)

where q_l > 0, p_l > 0 are Lagrange multiplier. Let the derivative $\frac {\partial {\mathscr{L}}(P)}{\partial P}$ be zero, we have the expression of the closed-form solution for P as follows:

$$ \begin{aligned} P^{*}=[S_{l}{X_{l}^{T}}+\frac{1}{2\tau}{\sum}_{i=1}^{N} q_{l}(M_{l}X_{l}){Y_{l}^{T}}](X_{l}{X_{l}^{T}}+\overline{X_{l}}\overline{X_{l}}^{T}+\theta I)^{-1}, \end{aligned} $$

(33)

where 𝜃I is a regularization term, and 𝜃 = 10e^− 4. In fact, the samples’ number may smaller than the dimension of feature space; therefore, it is necessary to add regularization term 𝜃I into the formula to avoid the problem of singularity similar as [61]. For example, the inverse of $X_{l}{X_{l}^{T}}$ may be singular. □

Proof Proof of Theorem 3

Variables M_l, δ_l and ξ_l are optimized by the Lagrangian function, and then the dual form of the optimization problem in (20) is obtained. Therefore, α_l > 0 and η_l > 0 are introduced as the Lagrange multipliers. By introducing the Lagrangian function into the objective function in (20), we can rewrite the objective function in (20) as follows: □

$$ \begin{aligned} \mathcal{L}(M,\xi,\eta)=& \frac{1}{2}\|M_{l}\|_{2}^{2} +C_{l}\sum\limits_{i=1}^{N}\xi_{l}-\sum\limits_{i=1}^{N}\alpha_{l}\xi_{l} \\ &-\sum\limits_{i=1}^{N} \eta_{l}\{[M_{l}\cdot P_{l}X_{l} + \delta_{l}]{Y_{l}^{T}} -1+\xi_{l}\},\\ s.t.&\ \forall \ \ \eta_{l}>0,\ \alpha_{l}>0. \end{aligned}$$

(34)

A saddle point in the Lagrangian is the minimum value of the variables M_l and ξ_l, however, it is the maximum value for the dual form. In order to obtain the minimum value of the variables, we require as follows:

$$ \frac{\partial(\mathcal{L})}{\partial \xi_{l}}=C_{l}-\alpha_{l}-\eta_{l}=0,$$

(35)

$$ \frac{\partial(\mathcal{L})}{\partial \delta_{l}}=\sum\limits_{i=1}^{N} \eta_{l}{Y_{l}^{T}}=0.$$

(36)

Similarly, for M_l we require,

$$ \frac{\partial(\mathcal{L})}{\partial M_{l}}= M_{l}-\sum\limits_{i=1}^{N} \eta_{l}P_{l}X_{l}\cdot {Y_{l}^{T}}=0,$$

(37)

The results are presented as follows:

$$ M_{l}=\sum\limits_{i=1}^{N} \eta_{l}P_{l}X_{l}\cdot {Y_{l}^{T}}.$$

(38)

By incorporating (35), (36) and (38) into (34), we have the following optimization function:

$$ \begin{aligned} \underset{\eta_{l}}{\min}\frac{1}{2}\sum\limits_{i=1}^{N} \sum\limits_{j=1}^{N}\eta_{l}\eta_{j}{Y_{l}^{T}}{Y_{j}^{T}}(X_{l}\cdot X_{j})+\sum\limits_{i=1}^{N} \eta_{l},\\ s.t.\sum\limits_{i=1}^{N} \eta_{l}{Y_{l}^{T}}=0,\ \ \ 0<\eta_{l}<C_{l}. \end{aligned} $$

(39)

The dual complementarity condition satisfying the KKT condition is:

$$ \begin{aligned} \eta_{l}^{*}({Y_{l}^{T}}(M_{l}P_{l}X_{l}+\delta_{l})-1+\xi_{l}^{*})=0. \end{aligned} $$

(40)

According to the dual complementarity condition of this KKT condition, we can get:

$$ \begin{aligned} &\eta_{l}^{*}=0\Rightarrow {Y_{l}^{T}}(M_{l}P_{l}X_{l}+\delta_{l})\geq 1,\\ &0<\eta_{l}^{*}<C_{l}\Rightarrow {Y_{l}^{T}}(M_{l}P_{l}X_{l}+\delta_{l})=1,\\ &\eta_{l}^{*}=C_{l}\Rightarrow {Y_{l}^{T}}(M_{l}P_{l}X_{l}+\delta_{l})\leq1. \end{aligned} $$

(41)