Ensemble of classifier chains and Credal C4.5 for solving multi-label classification

Abstract

In this work, we have considered the ensemble of classifier chains (ECC) algorithm in order to solve the multi-label classification (MLC) task. It starts from binary relevance algorithm (BR), a simple and direct approach to MLC that has been shown to provide good results in practice. Nevertheless, unlike BR, ECC aims to exploit the correlations between labels. ECC uses an algorithm of traditional supervised classification in order to approach the binary problems. Within this field, Credal C4.5 (CC4.5) is a new version of the well-known C4.5 algorithm that uses imprecise probabilities in order to estimate the probability distribution of the class variable. This new version of C4.5 algorithm has been shown to provide better performance when noisy datasets are classified. In MLC, the intrinsic noise might be higher than in traditional supervised classification. The reason is very simple: in MLC, there are multiple labels, whereas in traditional classification there is just a class variable. Thus, there is more probability of error for an instance. For the previous reasons, the performance of ECC with CC4.5 as base classifier is studied in this work. We have carried out an extensive experimental analysis with several multi-label datasets, different noise levels and a large number of evaluation metrics for MLC. This experimental study has shown that, generally, ECC has better performance with CC4.5 as base classifier than using C4.5. The higher is the label noise level introduced in the data, the more significative is this improvement. Therefore, it is probably suitable to use imprecise probabilities in Decision Trees within MLC.

Appendices

Appendix A: Evaluation measures

In this Appendix, the measures used in order to compare the algorithms of the experiments are explained. We suppose that we have a test set of N instances $\left\{ \mathbf{x }_\mathbf{1 }, \ldots , \mathbf{x }_\mathbf{N }\right\}$. The same notation of Sect. 2 is used.

Example-based measures

Example-based metrics [20, 22, 36] evaluate the predictions made in each one of the test instances.

• Subset accuracy It is a very strict metric that measures the proportion of examples whose predicted set of labels are exactly its set of relevant tags:

  $$\begin{aligned} \text {Subset}\_\text {Accuracy}(h) = \frac{1}{N}\sum _{i=1}^{N}I(h(\mathbf{x }_\mathbf{i }) = \mathbf{L }_i) \end{aligned}$$

  (21)

• Hamming loss It measures how many times an example-label is classified incorrectly, i.e, how many times an irrelevant label is predicted or a relevant label is not predicted on average:

  $$\begin{aligned} \text {Hamming}\_\text {Loss}(h) = \frac{1}{N}\sum _{i=1}^{N}\frac{1}{q}\left| h(\mathbf{x }_i)\triangle \mathbf{L }_i\right| \end{aligned}$$

  (22)

  where $\triangle$ denote the symmetric difference between two sets, i.e the number of elements that belongs to one set and not to the other one.

• Accuracy The example-based accuracy is defined by the average Jaccard similarity coefficients between the predicted label sets and the relevant label sets over the examples. Formally:

  $$\begin{aligned} \text {Accuracy}(h) = \frac{1}{N}\sum _{i=1}^{N}\frac{\left| h(\mathbf{x }_i) \cap \mathbf{L }_i\right| }{\left| h(\mathbf{x }_i) \cup \mathbf{L }_i\right| } \end{aligned}$$

  (23)

• Precision It is defined by the average of the proportion of real labels of the instances that are predicted:

  $$\begin{aligned} \text {Precision}(h) = \frac{1}{N}\sum _{i=1}^{N}\frac{\left| h(\mathbf{x }_i) \cap \mathbf{L }_i\right| }{\left| \mathbf{L }_i\right| } \end{aligned}$$

  (24)

• Recall It measures the average of the predicted labels that are really relevant for the instances:

  $$\begin{aligned} \text {Recall}(h) = \frac{1}{N}\sum _{i=1}^{N}\frac{\left| h(\mathbf{x }_i) \cap \mathbf{L }_i\right| }{\left| h(\mathbf{x }_i)\right| } \end{aligned}$$

  (25)

• F1-score It is the harmonic mean between Precision and Recall:

  $$\begin{aligned} F1(h) = \frac{1}{N}\sum _{i=1}^{N}\frac{ 2 \times \left| h(\mathbf{x }_i) \cap \mathbf{L }_i\right| }{\left| h(\mathbf{x }_i)\right| + \left| \mathbf{L }_i\right| } \end{aligned}$$

  (26)

Label-based measures

These measures [43] consider each label as a binary class, which contains the value 1 for an instance if it has associated that label and 0 otherwise.

• Macro Precision The Macro Precision is the Precision averaged across all labels:

  $$\begin{aligned} \text {Macro}\_\text {Precision} = \frac{1}{q}\sum _{j = 1}^{q}\frac{tp_j}{tp_j + fp_j} \end{aligned}$$

  (27)

  being $tp_j$ and $fp_j$ the number of true positives and false positives respectively for the label j, $1 \le j \le q$.

• Macro-Recall It is the Recall averaged across all labels:

  $$\begin{aligned} \text {Macro}\_\text {Recall} = \frac{1}{q}\sum _{j = 1}^{q}\frac{tp_j}{tp_j + fn_j} \end{aligned}$$

  (28)

  where $fn_j$ is the number of false negatives for the jth label.

• Macro F1 It measures the harmonic mean between Precision and Recall, calculated for each label and averaging over all labels.

  $$\begin{aligned} \text {Macro}\_F1 = \frac{1}{q}\sum _{j = 1}^{q}\frac{2 \times r_j \times p_j}{r_j + p_j} \end{aligned}$$

  (29)

  being $p_j$ and $r_j$ the Precision and Recall for the jth label respectively.

• Micro Precision It measures the Precision averaged over all example/label pairs

  $$\begin{aligned} \text {Micro}\_\text {Precision} = \frac{\sum _{j = 1}^{q}tp_j}{\sum _{j = 1}^{q}tp_j + \sum _{j = 1}^{q}fp_j} \end{aligned}$$

  (30)

• Micro-Recall It is the Recall averaged over all example/label pairs

  $$\begin{aligned} \text {Micro}\_\text {Recall} = \frac{\sum _{j = 1}^{q}tp_j}{\sum _{j = 1}^{q}tp_j + \sum _{j = 1}^{q}fn_j} \end{aligned}$$

  (31)

• Micro-F1 Micro-F1 is simply the harmonic mean between Micro Precision and Micro-Recall

  $$\begin{aligned}&\text {Micro}\_F1 \nonumber \\&= \frac{2 \times \text {Micro}\_\text {Precision} \times \text {Micro}\_\text {Recall}}{\text {Micro}\_\text {Precision} + \text {Micro}\_\text {Recall}} \end{aligned}$$

  (32)

Ranking-based measures

Ranking-based metrics [20, 22, 36, 43] principally measure the real-valuated function returned by the algorithm with its corresponding ranking function.

• One-Error It evaluates how many times on average the top-ranked label is not relevant for an instance. It is defined as:

  $$\begin{aligned} \text {One}\_\text {Error}(f) = \frac{1}{N}\sum _{i = 1}^{N}I\left( \arg \max _{l \in \mathbf{L }}{f(\mathbf{x }_i, l}) \notin \mathbf{L }_i\right) \nonumber \\ \end{aligned}$$

  (33)

  where I is the indicator function defined above.

• Coverage Coverage measures the number of steps, on average, that it is necessary to do going down the rank of labels to cover all the relevant labels of the labels.

  $$\begin{aligned} \text {Coverage}(f) = \frac{1}{N}\sum _{i = 1}^{N}\max _{l \in \mathbf{L }_i}{\mathrm{rank}\_f_{\mathbf{x }_i}(l)} - 1 \end{aligned}$$

  (34)

• Ranking Loss It evaluates, on average, the proportion of label pairs that are reversely ordered for the particular instance. Formally: Let $\mathbf{Z }_i = \{(l_n,l_m) \mid f(\mathbf{x }_i, lm) \le f(\mathbf{x }_i, ln), lm \in \mathbf{L }_i, ln \in \overline{\mathbf{L }}_i \}$, being $\overline{\mathbf{L }}_i$ the complementary set of $\mathbf{L }_i$. The ranking loss is defined as

  $$\begin{aligned} \text {Ranking}\_\text {Loss}(f) = \frac{1}{N}\sum _{i = 1}^{N}\frac{\left| \mathbf{Z }_i\right| }{\left| \mathbf{L }_i\right| \left| \overline{\mathbf{L }}_i\right| } \end{aligned}$$

  (35)

• Average precision It is the average proportion of labels ranked above a relevant label. Let denote $\varLambda _i = \left\{ l^{'} \mid \mathrm{rank}\_f_{\mathbf{x }_i}(l^{'}) \le \mathrm{rank}\_f_{\mathbf{x }_i}(l), l^{'} \in \mathbf{L }_i \right\}$. Average precision is defined as:

  $$\begin{aligned} \text {Average}\_\text {Precision}(f) = \frac{1}{N}\sum _{i = 1}^{N}\frac{\left| \varLambda _i\right| }{\mathrm{rank}\_f_{\mathbf{x }_i}(l)} \end{aligned}$$

  (36)

  For One-Error, Coverage and Ranking Loss the performance is better if the value is lower. However, a higher value of Average Precision implies a better performance.

Appendix B: Complete results from the experimental study

The complete results from the experimentation are shown in this section. In concrete, we present the results for each dataset used in the experiments for each metric. The results of the example-based classification metrics are presented in Tables 5, 6, 7, 8, 9 and 10. The results of the label-based classification metrics are presented in Tables 11, 12, 13, 14, 15 and 16. The results of the example-based ranking metrics are presented in Tables 17, 18, 19 and 20. In those tables, the best result for each dataset, algorithm and noise level is marked in bold.

Table 5 Subset Accuracy results with $0\%$, $5\%$ and $10\%$ of noise

Table 6 Hamming Loss results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 7 Accuracy results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 8 Precision results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 9 Recall results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 10 F1 results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 11 Micro Precision results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 12 Macro Precision results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 13 Micro-Recall results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 14 Macro-Recall results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 15 Micro F1 results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 16 Macro F1 results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 17 Ranking Loss results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 18 One-Error results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 19 Coverage results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise

Table 20 Average Precision results of each method obtained for each dataset with $0\%$, $5\%$ and $10\%$ of noise