Skip to main content
SpringerLink
Log in

Learning from group supervision: the impact of supervision deficiency on multi-label learning

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Multi-label learning studies the problem where one instance is associated with multiple labels. Weakly supervised multi-label learning has attracted considerable research attention because of the annotation difficulty. Majority of the studies on weakly supervised multi-label learning assume that one group of weak annotations is available for each instance; however, none of these studies considers multiple groups of weak annotations that can be easily acquired through crowdsourcing. Recent studies on crowdsourced multi-label learning observed that the current query strategies do not agree well with human habits and that data cannot be collected as expected. Therefore, this study aims to design a new query strategy in accordance with human behavior patterns to obtain multiple groups of weak annotations. Further, a learning algorithm is proposed based on neural networks for such type of data. In addition, this study qualitatively and empirically analyzes factors in the proposed query strategy that may impact further learning and provides insights to obtain better query strategy with respect to future crowdsourcing in case of multi-label data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Zhou Z-H, Zhang M-L. Multi-label learning. In: Encyclopedia of Machine Learning and Data Mining. Berlin: Springer, 2016. 875–881

    Google Scholar 

  2. Cabral R S, de la Torre F, Costeira J P, et al. Matrix completion for weakly-supervised multi-label image classification. IEEE Trans Pattern Anal Mach Intell, 2015, 37: 121–135

    Article  Google Scholar 

  3. Chen M, Zheng A X, Weinberger K Q. Fast image tagging. In: Proceedings of the 30th International Conference on Machine Learning, 2013. 1274–1282

  4. Chalkidis I, Fergadiotis M, Malakasiotis P, et al. Large-scale multi-label text classification on EU legislation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 6314–6322

  5. Nam J, Kim J, Menciaa E L, et al. Large-scale multi-label text classification — revisiting neural networks. In: Proceedings of the 25th European Conference on Machine Learning, 2014. 437–452

  6. Zhou J, Chen L, Guo Z. iATC-NRAKEL: an efficient multi-label classifier for recognizing anatomical therapeutic chemical classes of drugs. Bioinform, 2020, 36: 1391–1396

    Google Scholar 

  7. Zhang J, Zhang Z, Wang Z, et al. Ontological function annotation of long non-coding RNAs through hierarchical multi-label classification. Bioinform, 2018, 34: 1750–1757

    Article  Google Scholar 

  8. Zhou Z H. A brief introduction to weakly supervised learning. Natl Sci Rev, 2018, 5: 44–53

    Article  Google Scholar 

  9. Xu M, Jin R, Zhou Z. Speedup matrix completion with side information: application to multi-label learning. In: Proceedings of Advances in Neural Information Processing Systems 26, 2013. 2301–2309

  10. Sun Y, Zhang Y, Zhou Z. Multi-label learning with weak label. In: Proceedings of the 24th Conference on Artificial Intelligence, 2010

  11. Bucak S S, Jin R, Jain A K. Multi-label learning with incomplete class assignments. In: Proceedings of the 24th Conference on Computer Vision and Pattern Recognition, 2011. 2801–2808

  12. Xie M, Huang S. Partial multi-label learning. In: Proceedings of the 32nd Conference on Artificial Intelligence, 2018. 4302–4309

  13. Yu G, Chen X, Domeniconi C, et al. Feature-induced partial multi-label learning. In: Proceedings of the 2018 International Conference on Data Mining, 2018. 1398–1403

  14. Estelles-Arolas E, González-Ladrón-de-Guevara F. Towards an integrated crowdsourcing definition. J Inf Sci, 2012, 38: 189–200

    Article  Google Scholar 

  15. Li S, Jiang Y, Chawla N V, et al. Multi-label learning from crowds. IEEE Trans Knowl Data Eng, 2019, 31: 1369–1382

    Article  Google Scholar 

  16. Li S, Jiang Y. Multi-label crowdsourcing learning with incomplete annotations. In: Proceedings of the 15th Pacific Rim International Conference on Artificial Intelligence, 2018. 232–245

  17. Quiroga R Q, Pedreira C. How do we see art: an eye-tracker study. Front Hum Neurosci, 2011, 5: 98

    Article  Google Scholar 

  18. Group N N. How People Read Online: The Eyetracking Evidence. 2nd ed. Technical Report, 2020

  19. Kingma D P, Ba J. Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, 2015

  20. Srivastava N, Hinton G E, Krizhevsky A, et al. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res, 2014, 15: 1929–1958

    MathSciNet  MATH  Google Scholar 

  21. Boutell M R, Luo J, Shen X, et al. Learning multi-label scene classification. Pattern Recogn, 2004, 37: 1757–1771

    Article  Google Scholar 

  22. Tsoumakas G, Vlahavas I P. Random k-labelsets: an ensemble method for multilabel classification. In: Proceedings of the 18th European Conference on Machine Learning, 2007. 406–417

  23. Read J, Pfahringer B, Holmes G, et al. Classifier chains for multi-label classification. In: Proceedings of the 20th European Conference on Machine Learning, 2009. 254–269

  24. Elisseeff A, Weston J. A kernel method for multi-labelled classification. In: Proceedings of Advances in Neural Information Processing Systems 14, 2001. 681–687

  25. Bhatia K, Jain H, Kar P, et al. Sparse local embeddings for extreme multi-label classification. In: Proceedings of Advances in Neural Information Processing Systems 28, 2015. 730–738

  26. Zhang M L, Zhou Z H. A review on multi-label learning algorithms. IEEE Trans Knowl Data Eng, 2014, 26: 1819–1837

    Article  Google Scholar 

  27. Hsu D J, Kakade S M, Langford J, et al. Multi-label prediction via compressed sensing. In: Proceedings of Advances in Neural Information Processing Systems 22, 2009. 772–780

  28. Tai F, Lin H T. Multilabel classification with principal label space transformation. Neural Computation, 2012, 24: 2508–2542

    Article  MathSciNet  Google Scholar 

  29. Bi W, Kwok J T. Efficient multi-label classification with many labels. In: Proceedings of the 30th International Conference on Machine Learning, 2013. 405–413

  30. Ubaru S, Mazumdar A. Multilabel classification with group testing and codes. In: Proceedings of the 34th International Conference on Machine Learning, 2017. 3492–3501

  31. Goldberg A B, Zhu X, Recht B, et al. Transduction with matrix completion: three birds with one stone. In: Proceedings of Advances in Neural Information Processing Systems 23, 2010. 757–765

  32. Bi W, Kwok J T. Multilabel classification with label correlations and missing labels. In: Proceedings of the 28th Conference on Artificial Intelligence, 2014. 1680–1686

  33. Xu L, Wang Z, Shen Z, et al. Learning low-rank label correlations for multi-label classification with missing labels. In: Proceedings of the 2014 International Conference on Data Mining, 2014. 1067–1072

  34. Jing L, Yang L, Yu J, et al. Semi-supervised low-rank mapping learning for multi-label classification. In: Proceedings of the 28th Conference on Computer Vision and Pattern Recognition, 2015. 1483–1491

  35. Wu B, Lyu S, Ghanem B. ML-MG: multi-label learning with missing labels using a mixed graph. In: Proceedings of the 2015 International Conference on Computer Vision, 2015. 4157–4165

  36. Ferng C, Lin H. Multi-label classification with error-correcting codes. In: Proceedings of the 3rd Asian Conference on Machine Learning, 2011. 281–295

  37. Lv J, Xu M, Feng L, et al. Progressive identification of true labels for partial-label learning. 2020. ArXiv:2002.08053

  38. Bottou L. On-line learning and stochastic approximations. In: Online Learning in Neural Networks. Cambridge: Cambridge University Press, 1998. 9–42

    MATH  Google Scholar 

  39. Andrew G, Gao J. Scalable training of L1-regularized log-linear models. In: Proceedings of the 24th International Conference on Machine Learning, 2007. 33–40

  40. Liu L, Dietterich T G. Learnability of the superset label learning problem. In: Proceedings of the 31st International Conference on Machine Learning, 2014. 1629–1637

  41. Zhang M-L, Zhou Z-H. Multilabel neural networks with applications to functional genomics and text categorization. IEEE Trans Knowl Data Eng, 2006, 18: 1338–1351

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia, 4072, Australia

    Miao Xu

  2. RIKEN Center for Advanced Intelligence Project, Tokyo, 103-0027, Japan

    Miao Xu

  3. National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Lan-Zhe Guo

Authors
  1. Miao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lan-Zhe Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Miao Xu or Lan-Zhe Guo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, M., Guo, LZ. Learning from group supervision: the impact of supervision deficiency on multi-label learning. Sci. China Inf. Sci. 64, 130101 (2021). https://doi.org/10.1007/s11432-020-3132-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-020-3132-4

Keywords

Search

Navigation