Skip to main content
SpringerLink
Log in

PLVI-CE: a multi-label active learning algorithm with simultaneously considering uncertainty and diversity

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In multi-label learning, each instance simultaneously associates with multiple labels, which means that labeling such instances is quite costly. Active learning, as an important machine learning paradigm, learns the classification model by querying merely a small portion of data with important information, by means of which, the labeling cost can be greatly reduced during the training process and an accurate and robust classification model could be obtained. Therefore, multi-label active learning (MLAL) has garnered increasing attentions. The primary challenge in MLAL lies in designing an effective query strategy to measure uniform information about unlabeled instances throughout all labels. In this study, we propose a query strategy named predicted label vectors inconsistency and cross entropy measure (PLVI-CE) that considers both uncertainty and diversity measures. In PLVI-CE, the uncertainty is measured by the inconsistency between two predicted label vectors from the same unlabeled instance, and the diversity is assessed by the average discrepancy in posterior probabilities between each unlabeled instance and all instances in the labeled set. Furthermore, in this study, we try to adopt label-weighted extreme learning machine (LW-ELM) as the base classifier in the MLAL framework with considering its following advantages: (1) LW-ELM has a low computational cost, (2) LW-ELM has strong generalization performance, and (3) LW-ELM can be directly used to classify multi-label data with class imbalance distributions, hence providing approximately unbiased instance querying during MLAL. Experimental results on 12 benchmark multi-label datasets indicate the effectiveness and superiority of the proposed PLVI-CE algorithm in comparison with several current state-of-the-art MLAL algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from http://meka.sourceforge.net/#datasets.

Code Availability

Code is available at: https://github.com/ML-YanGu/PLVI-CE.

References

  1. Ren P, Xiao Y, Chang X, Huang P-Y, Li Z, Gupta BB, Chen X, Wang X (2021) A survey of deep active learning. ACM Comput Surv (CSUR) 54(9):1–40

    Article  Google Scholar 

  2. Yu H, Yang X, Zheng S, Sun C (2018) Active learning from imbalanced data: a solution of online weighted extreme learning machine. IEEE Trans Neural Netw Learn Syst 30(4):1088–1103

    Article  Google Scholar 

  3. Gui X, Lu X, Yu G (2021) Cost-effective batch-mode multi-label active learning. Neurocomputing 463:355–367

    Article  Google Scholar 

  4. Chakraborty S, Balasubramanian V, Panchanathan S (2014) Adaptive batch mode active learning. IEEE Trans Neural Netw Learn Syst 26(8):1747–1760

    Article  MathSciNet  Google Scholar 

  5. Yu H, Sun C, Yang W, Yang X, Zuo X (2015) AL-ELM: one uncertainty-based active learning algorithm using extreme learning machine. Neurocomput 166:140–150

    Article  Google Scholar 

  6. Min F, Zhang S-M, Ciucci D, Wang M (2020) Three-way active learning through clustering selection. Int J Mach Learn Cybern 11(5):1033–1046

    Article  Google Scholar 

  7. Zhang Y, Er MJ (2016) Sequential active learning using meta-cognitive extreme learning machine. Neurocomputing 173:835–844

    Article  Google Scholar 

  8. Nguyen DH, Patrick JD (2014) Supervised machine learning and active learning in classification of radiology reports. J Am Med Inform Assoc 21(5):893–901

    Article  Google Scholar 

  9. Gao N, Huang S-J, Chen S (2016) Multi-label active learning by model guided distribution matching. Front Comput Sci 10(5):845–855

    Article  Google Scholar 

  10. Jing X-Y, Wu F, Li Z, Hu R, Zhang D (2016) Multi-label dictionary learning for image annotation. IEEE Trans Image Process 25(6):2712–2725

    Article  MathSciNet  MATH  Google Scholar 

  11. Liu Y, Wen K, Gao Q, Gao X, Nie F (2018) SVM based multi-label learning with missing labels for image annotation. Pattern Recognit 78:307–317

    Article  Google Scholar 

  12. Reyes O, Ventura S (2018) Evolutionary strategy to perform batch-mode active learning on multi-label data. ACM Trans Int Syst Technol (TIST) 9(4):1–26

    Article  Google Scholar 

  13. Min X-Y, Qian K, Zhang B-W, Song G, Min F (2022) Multi-label active learning through serial-parallel neural networks. Knowl-Based Syst 251:109226

    Article  Google Scholar 

  14. Ye C, Wu J, Sheng VS, Zhao P, Cui Z (2015) Multi-label active learning with label correlation for image classification. In: 2015 IEEE international conference on image processing (ICIP). IEEE, pp 3437–3441

  15. Wu J, Zhao S, Sheng VS, Zhao P, Cui Z (2016) Multi-label active learning for image classification with asymmetrical conditional dependence. In: 2016 IEEE international conference on multimedia and expo (ICME). IEEE, pp 1–6

  16. Wu J, Zhao S, Sheng VS, Zhang J, Ye C, Zhao P, Cui Z (2017) Weak-labeled active learning with conditional label dependence for multilabel image classification. IEEE Trans Multimed 19(6):1156–1169

    Article  Google Scholar 

  17. Yang B, Sun J-T, Wang T, Chen Z (2009) Effective multi-label active learning for text classification. In: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining, pp 917–926

  18. Kang X, Shi X, Wu Y, Ren F (2020) Active learning with complementary sampling for instructing class-biased multi-label text emotion classification. IEEE Trans Affect Comput 14(1):523–536

    Article  Google Scholar 

  19. Shi W, Liu X, Yu Q (2017) Correlation-aware multi-label active learning for web service tag recommendation. In: 2017 IEEE international conference on web services (ICWS). IEEE, pp 229–236

  20. Carrillo D, López VF, Moreno MN (2013) Multi-label classification for recommender systems. Trends Pract Appl Agents Multiagent Syst 181–188

  21. El-Hasnony IM, Elzeki OM, Alshehri A, Salem H (2022) Multi-label active learning-based machine learning model for heart disease prediction. Sensors 22(3):1184

    Article  Google Scholar 

  22. Wu J, Zhu W, Jiang Y, Sun G, Gao, Y (2018) Predicting protein functions of bacteria genomes via multi-instance multi-label active learning. In: 2018 IEEE 3rd international conference on integrated circuits and microsystems (ICICM). IEEE, pp 302–307

  23. Li X, Guo Y (2013) Active learning with multi-label SVM classification. In: IjCAI. Citeseer, pp 1479–1485

  24. Wu J, Sheng VS, Zhang J, Zhao P, Cui Z (2014) Multi-label active learning for image classification. In: 2014 IEEE international conference on image processing (ICIP). IEEE, pp 5227–5231

  25. Huang S-J, Chen S, Zhou Z-H (2015) Multi-label active learning: query type matters. In: Twenty-fourth international joint conference on artificial intelligence (IjCAI). pp 946–952

  26. Huang, S-J, Zhou, Z-H (2013) Active query driven by uncertainty and diversity for incremental multi-label learning. In: 2013 IEEE 13th international conference on data mining. IEEE, pp 1079–1084

  27. Reyes O, Morell C, Ventura S (2018) Effective active learning strategy for multi-label learning. Neurocomputing 273:494–508

    Article  Google Scholar 

  28. Du B, Wang Z, Zhang L, Zhang L, Tao D (2017) Robust and discriminative labeling for multi-label active learning based on maximum correntropy criterion. IEEE Trans on Image Process 26(4):1694–1707

    Article  MathSciNet  MATH  Google Scholar 

  29. Yu G, Chen X, Domeniconi C, Wang J, Li Z, Zhang Z, Zhang X (2020) Cmal: cost-effective multi-label active learning by querying subexamples. IEEE Trans on Knowl Data Eng 34(5):2091–2105

    Article  Google Scholar 

  30. Wu J, Sheng VS, Zhang J, Li H, Dadakova T, Swisher CL, Cui Z, Zhao P (2020) Multi-label active learning algorithms for image classification: overview and future promise. ACM Comput Surv (CSUR) 53(2):1–35

    Article  Google Scholar 

  31. Wu J, Ye C, Sheng VS, Zhang J, Zhao P, Cui Z (2017) Active learning with label correlation exploration for multi-label image classification. IET Comput Vis 11(7):577–584

    Article  Google Scholar 

  32. Freund Y, Seung HS, Shamir E, Tishby N (1997) Selective sampling using the query by committee algorithm. Mach Learn 28(2):133–168

    Article  MATH  Google Scholar 

  33. Zhang M-L, Li Y-K, Yang H, Liu X-Y (2020) Towards class-imbalance aware multi-label learning. IEEE Trans Cybern 52(6):4459–4471

    Article  Google Scholar 

  34. Tarekegn AN, Giacobini M, Michalak K (2021) A review of methods for imbalanced multi-label classification. Pattern Recog 118:107965

    Article  Google Scholar 

  35. Yu H, Sun C, Yang X, Zheng S, Wang Q, Xi X (2018) Lw-elm: a fast and flexible cost-sensitive learning framework for classifying imbalanced data. IEEE Access 6:28488–28500

    Article  Google Scholar 

  36. Suter BW (1990) The multilayer perceptron as an approximation to a bayes optimal discriminant function. IEEE Trans Neural Netw 1(4):291

  37. Wan EA (1990) Neural network classification: a bayesian interpretation. IEEE Trans Neural Netw 1(4):303–305

  38. Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks (IEEE Cat. No. 04CH37541). IEEE, vol 2, pp 985–990

  39. Huang G-B, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122

    Article  Google Scholar 

  40. Huang G-B, Zhou H, Ding X, Zhang R (2011) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B (Cybernetics) 42(2):513–529

  41. Zong W, Huang G-B, Chen Y (2013) Weighted extreme learning machine for imbalance learning. Neurocomputing 101:229–242

    Article  Google Scholar 

  42. Chen S, Wang R, Lu J, Wang X (2022) Stable matching-based two-way selection in multi-label active learning with imbalanced data. Inform Sci 610:281–299

    Article  Google Scholar 

  43. Kang F, Jin R, Sukthankar R (2006) Correlated label propagation with application to multi-label learning. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06). IEEE, vol 2, pp 1719–1726

  44. Wang B, Tu Z, Tsotsos JK (2013) Dynamic label propagation for semi-supervised multi-class multi-label classification. In: Proceedings of the IEEE international conference on computer vision, pp 425–432

  45. Minaev G, Visa A, Piché R (2017) Comprehensive survey of similarity measures for ranked based location fingerprinting algorithm. In: 2017 international conference on indoor positioning and indoor navigation (IPIN). IEEE, pp 1–4

  46. Zhang M-L, Zhou Z-H (2013) A review on multi-label learning algorithms. IEEE Trans Knowl Data Eng 26(8):1819–1837

    Article  Google Scholar 

  47. Gibaja E, Ventura S (2014) Multi-label learning: a review of the state of the art and ongoing research. Wiley Interdiscip Rev: Data Min Knowl Discov 4(6):411–444

    Google Scholar 

  48. Wang M, Feng T, Shan Z, Min F (2022) Attribute and label distribution driven multi-label active learning. Appl Int 52(10):11131–11146

    Article  Google Scholar 

  49. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  50. García S, Fernández A, Luengo J, Herrera F (2010) Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inform Sci 180(10):2044–2064

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Natural Science Foundation of Jiangsu Province of China under (Grant No.BK20191457), Postgraduate Research & Practice Innovation Program of Jiangsu Province of China under (Grant No. SJCX22_1901), and National Natural Science Foundation of China under (Grant No. 62176107).

Author information

Authors and Affiliations

  1. School of Computer, Jiangsu University of Science and Technology, Zhen Jiang, 212100, China

    Yan Gu, Jicong Duan, Hualong Yu, Xibei Yang & Shang Gao

Authors
  1. Yan Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jicong Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hualong Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xibei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yan Gu: Methods, Investigation, Codes, Experiments, Writing-original draft; Jicong Duan: Investigation, Formal analysis, Data analysis; Hualong Yu: Conceptualization, Data analysis, Writing-review and editing, Funding acquisition, Supervision; Xibei Yang: Resources, Data analysis, Writing-review and editing; Shang Gao: Formal analysis, Data analysis.

Corresponding author

Correspondence to Hualong Yu.

Ethics declarations

Ethics approval

The contents in this article conform to the aims and interests of Applied Intelligence Journal. We also expect to declare that the paper is our original work and it hasnt been submitted to any other journals or conferences.

Conflict of interest

The authors declared that they have no known conflicts of interest to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, Y., Duan, J., Yu, H. et al. PLVI-CE: a multi-label active learning algorithm with simultaneously considering uncertainty and diversity. Appl Intell 53, 27844–27864 (2023). https://doi.org/10.1007/s10489-023-05008-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05008-2

Keywords

Search

Navigation