The use of data-derived label hierarchies in multi-label classification

Abstract

Instead of traditional (multi-class) learning approaches that assume label independency, multi-label learning approaches must deal with the existing label dependencies and relations. Many approaches try to model these dependencies in the process of learning and integrate them in the final predictive model, without making a clear difference between the learning process and the process of modeling the label dependencies. Also, the label relations incorporated in the learned model are not directly visible and can not be (re)used in conjunction with other learning approaches. In this paper, we investigate the use of label hierarchies in multi-label classification, constructed in a data-driven manner. We first consider flat label sets and construct label hierarchies from the label sets that appear in the annotations of the training data by using a hierarchical clustering approach. The obtained hierarchies are then used in conjunction with hierarchical multi-label classification (HMC) approaches (two local model approaches for HMC, based on SVMs and PCTs, and two global model approaches, based on PCTs for HMC and ensembles thereof). The experimental results reveal that the use of the data-derived label hierarchy can significantly improve the performance of single predictive models in multi-label classification as compared to the use of a flat label set, while this is not preserved for the ensemble models.

Appendices

Appendix A: Evaluation measures

In this section, we present the measures that are used to evaluate the predictive performance of the compared methods in our experiments. In the definitions below, $\mathcal {Y}_{i}$ denotes the set of true labels of example x _i and h(x _i ) denotes the set of predicted labels for the same examples. All definitions refer to the multi-label setting.

1.1 A.1 Example based measures

Hamming loss evaluates how many times an example-label pair is misclassified, i.e., label not belonging to the example is predicted or a label belonging to the example is not predicted. The smaller the value of h a m m i n g_l o s s(h), the better the performance. The performance is perfect when h a m m i n g_l o s s(h)=0. This metric is defined as:

$$ hamming\_loss(h)=\frac{1}{N}\sum\limits^{N}_{i=1}\frac{1}{Q}\left|h(\mathbf{x_{i}}){\Delta} \mathcal{Y}_{i}\right| $$

(7)

where Δ stands for the symmetric difference between two sets, N is the number of examples and Q is the total number of possible class labels.

Accuracy for a single example x _i is defined by the Jaccard similarity coefficients between the label sets h(x _i ) and $\mathcal {Y}_{i}$. Accuracy is micro-averaged across all examples.

$$ accuracy(h)=\frac{1}{N}\sum\limits^{N}_{i=1}\frac{\left|h(\mathbf{x_{i}})\bigcap \mathcal{Y}_{i}\right|}{\left|h(\mathbf{x_{i}})\bigcup \mathcal{Y}_{i}\right|} $$

(8)

Precision is defined as:

$$ precision(h)=\frac{1}{N}\sum\limits^{N}_{i=1}\frac{\left|h(\mathbf{x_{i}})\bigcap \mathcal{Y}_{i}\right|}{\left|h(\mathbf{x_{i}})\right|} $$

(9)

Recall is defined as:

$$ recall(h)=\frac{1}{N}\sum\limits^{N}_{i=1}\frac{\left|h(\mathbf{x_{i}})\bigcap \mathcal{Y}_{i}\right|}{\left|\mathcal{Y}_{i}\right|} $$

(10)

F ₁ score is the harmonic mean between precision and recall and is defined as:

$$ F_{1}=\frac{1}{N}\sum\limits^{N}_{i=1}\frac{2 \times \left|h(\mathbf{x_{i}}) \cap \mathcal{Y}_{i}\right|}{\left|h(\mathbf{x_{i}})\right| + \left|\mathcal{Y}_{i}\right|} $$

(11)

F ₁ is an example based metric and its value is an average over all examples in the dataset. F ₁ reaches its best value at 1 and worst score at 0.

Subset accuracy or classification accuracy is defined as follows:

$$ subset\_accuracy(h)=\frac{1}{N}\sum\limits^{N}_{i=1}I(h(\mathbf{x_{i}})=\mathcal{Y}_{i}) $$

(12)

where I(t r u e) = 1 and I(f a l s e) = 0. This is a very strict evaluation measure as it requires the predicted set of labels to be an exact match of the true set of labels.

1.2 A.2 Label based measures

Macro precision (precision averaged across all labels) is defined as:

$$ macro\_precision=\frac{1}{Q}\sum\limits^{Q}_{j=1}\frac{tp_{j}}{tp_{j} + fp_{j}} $$

(13)

where t p _j , f p _j are the number of true positives and false positives for the label λ _j considered as a binary class.

Macro recall (recall averaged across all labels) is defined as:

$$ macro\_recall=\frac{1}{Q}\sum\limits^{Q}_{j=1}\frac{tp_{j}}{tp_{j} + fn_{j}} $$

(14)

where t p _j , f p _j are defined as for the macro precision and f n _j is the number of false negatives for the label λ _j considered as a binary class.

Macro F ₁ is the harmonic mean between precision and recall, where the average is calculated per label and then averaged across all labels. If p _j and r _j are the precision and recall for all λ _j ∈h(x _i ) from $\lambda _{j} \in \mathcal {Y}_{i}$, the macro F ₁ is

$$ macro\_F_{1}=\frac{1}{Q}\sum\limits^{Q}_{j=1}\frac{2\times p_{j} \times r_{j}}{p_{j} + r_{j}} $$

(15)

Micro precision (precision averaged over all the example/label pairs) is defined as:

$$ micro\_precision=\frac{{\sum}^{Q}_{j=1}{tp_{j}}}{{\sum}^{Q}_{j=1}{tp_{j}} + {\sum}^{Q}_{j=1}{fp_{j}}} $$

(16)

where t p _j , f p _j are defined as for macro precision.

Micro recall (recall averaged over all the example/label pairs) is defined as:

$$ micro\_recall=\frac{{\sum}^{Q}_{j=1}{tp_{j}}}{{\sum}^{Q}_{j=1}{tp_{j}} + {\sum}^{Q}_{j=1}{fn_{j}}} $$

(17)

where t p _j and f n _j are defined as for macro recall.

Micro F ₁ is the harmonic mean between micro precision and micro recall. Micro F ₁ is defined as:

$$ micro\_F_{1}=\frac{2 \times micro\_precision \times micro\_recall}{micro\_precision + micro\_recall} $$

(18)

1.3 A.3 Ranking based measures

One error evaluates how many times the top-ranked label is not in the set of relevant labels of the example. The metric o n e_e r r o r(f) takes values between 0 and 1. The smaller the value of o n e_e r r o r(f), the better the performance. This evaluation metric is defined as:

$$ one\_error(f)=\frac{1}{N}\sum\limits^{N}_{i=1}\left[\!\!\!\left[\hspace{1 mm}\left[\arg\max_{\lambda \in \mathcal{Y}} f(\mathbf{x_{i}}, \lambda)\right] \notin \mathcal{Y}_{i} \hspace{1 mm}\right]\!\!\!\right] $$

(19)

where $\lambda \in \mathcal {L} = \left \{\lambda _{1}, \lambda _{2}, ..., \lambda _{Q}\right \}$ and [ [π] ] equals 1 if π holds and 0 otherwise for any predicate π. Note that, for single-label classification problems, the One Error is identical to ordinary classification error.

Coverage evaluates how far, on average, we need to go down the list of ranked labels in order to cover all the relevant labels of the example. The smaller the value of c o v e r a g e(f), the better the performance.

$$ coverage(f)=\frac{1}{N}\sum\limits^{N}_{i=1}\max_{\lambda \in \mathcal{Y}_{i}} rank_{f}(\mathbf{x_{i}}, \lambda) - 1 $$

(20)

where r a n k _f (x _i ,λ) denotes the position of the label λ in the ranking. It maps the outputs of f(x _i ,λ) for any $\lambda \in \mathcal {L}$ to {λ ₁,λ ₂,...,λ _Q } so that f(x _i ,λ _m )>f(x _i ,λ _n ) implies r a n k _f (x _i ,λ _m )<r a n k _f (x _i ,λ _n ). The smallest possible value for c o v e r a g e(f) is l _c , i.e., the label cardinality of the given dataset.

Ranking loss evaluates the average fraction of label pairs that are reversely ordered for the particular example given by:

$$ ranking\ loss(f)=\frac{1}{N}\sum\limits^{N}_{i=1}\frac{\left| D_{i}\right|}{\left| \mathcal{Y}_{i}\right|\left|\bar{\mathcal{Y}_{i}}\right|} $$

(21)

where $D_{i} = \{(\lambda _{m}, \lambda _{n}) | f(\mathbf {x_{i}}, \lambda _{m}) \leq f(\mathbf {x_{i}}, \lambda _{n}), (\lambda _{m}, \lambda _{n}) \in \mathcal {Y}_{i} \times \bar {\mathcal {Y}_{i}}\}$, while $\bar {\mathcal {Y}}$ denotes the complementary set of $\mathcal {Y}$ in $\mathcal {L}$. The smaller the value of r a n k i n g_l o s s(f), the better the performance, so the performance is perfect when r a n k i n g_l o s s(f)=0.

Average Precision is the average fraction of labels ranked above an actual label $\lambda \in \mathcal {Y}_{i}$ that actually are in $\mathcal {Y}_{i}$. The performance is perfect when a v g_p r e c i s i o n(f)=1; the larger the value of a v g_p r e c i s i o n(f), the better the performance. This metric is defined as:

$$ avg\_precision(f)=\frac{1}{N}\sum\limits^{N}_{i=1}\frac{1}{\left| \mathcal{Y}_{i}\right|}\sum\limits_{\lambda \in \mathcal{Y}_{i}}\frac{\left| \mathcal{L}_{i}\right|}{rank_{f}(\mathbf{x_{i}}, \lambda)} $$

(22)

where $\mathcal {L}_{i}=\{\lambda ^{\prime }| rank_{f}(\mathbf {x_{i}}, \lambda ^{\prime }) \leq rank_{f}(\mathbf {x_{i}}, \lambda ), \lambda ^{\prime } \in \mathcal {Y}_{i}\}$ and r a n k _f (x _i ,λ) is defined as in coverage above.

1.4 B.1 Results on the example-based evaluation measures

Table 4 The performance of the multi-label classification approaches in terms of the examples-based evaluation measures

Table 5 The performance of the multi-label classification approaches in terms of the examples-based evaluation measures

Fig. 6

The critical diagrams for the example-based evaluation measures: The results from the Nemenyi post-hoc test at 0.05 significance level on all the datasets

1.5 B.2 Results on the label-based evaluation measures

The big difference between the micro-based and macro-based evaluation measures appears due to the averaging strategy of the obtained predictions. It is more emphasized on the large datasets that have highly unbalanced number of examples per label. Namely, the averaging in the micro-based measures is made across the predictions per example for all labels, while the averaging in the macro-based measures is made across the predictions per label for all examples, which means that for macro-based measures the labels with small number of examples are equally important as the labels with large number of examples.

Table 6 The performance of the multi-label classification approaches in terms of the label-based evaluation measures

Table 7 The performance of the multi-label classification approaches in terms of the label-based evaluation measures

Fig. 7

The critical diagrams for the label-based evaluation measures: The results from the Nemenyi post-hoc test at 0.05 significance level on all the datasets

1.6 B.3 Results on the ranking-based evaluation measures

Fig. 8

The critical diagrams for the ranking-based evaluation measures: The results from the Nemenyi post-hoc test at 0.05 significance level on all the datasets

Table 8 The performance of the multi-label classification approaches in terms of the ranking-based evaluation measures

Appendix B: Complete results from the experimental evaluation

In this section, we present the complete results from the experimental evaluation. We present the results based on the evaluation measures. Tables 4, 5, 6, 7 and 8 give the performance of the compared methods on each of the datasets measured in terms of the example based, label based and ranking based evaluation measures. The first column of the tables lists the dataset, while the remaining columns show the performance of each method for every dataset. The best results per dataset are shown in boldface. For the bookmarks dataset, HOMER did not manage to construct a predictive model within one week under the available resources. The corresponding entries in the tables with the results are marked with DNF (Did Not Finish).

To assess whether the overall differences in performance across the different approaches are statistically significant, we also employed the corrected Friedman test (Friedman 1940) and the post-hoc Nemenyi test (Nemenyi 1963) as recommended by Demšar (2006). We present the results from the Nemenyi post-hoc test with average rank diagrams (Demšar 2006). These are given in Figs. 6, 7 and 8. A critical diagram contains an enumerated axis on which the average ranks of the algorithms are drawn. The algorithms are depicted along the axis in such a manner that the best ranking ones are at the right-most side of the diagram. The lines for the average ranks of the algorithms that do not differ significantly (at the significance level of p=0.05) are connected with a line. For the bookmarks dataset, we penalize HOMER that does not finish by assigning it the lowest value (i.e., the lowest rank value) for each evaluation measure.