Skip to main content

Advertisement

Springer Nature Link
Log in

Imbalanced multi-instance multi-label learning via tensor product-based semantic fusion

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

With powerful expressiveness of multi-instance multi-label learning (MIML) for objects with multiple semantics and its great flexibility for complex object structures, MIML has been widely applied to various applications. In practical MML tasks, the naturally skewed label distribution and label interdependence bring up the label imbalance issue and decrease model performance, which is rarely studied. To solve these problems, we propose an imbalanced multi-instance multi-label learning method via tensor product-based semantic fusion (IMIML-TPSF) to deal with label interdependence and label distribution imbalance simultaneously. Specifically, to reduce the effect of label interdependence, it models similarity between the query object and object sets of different label classes for similarity-structural features. To alleviate disturbance caused by the imbalanced label distribution, it establishes the ensemble model for imbalanced distribution features. Subsequently, IMIML-TPSF fuses two types of features by tensor product and generates the new feature vector, which can preserve the original and interactive feature information for each bag. Based on such features with rich semantics, it trains the robust generalized linear classification model and further captures label interdependence. Extensive experimental results on several datasets validate the effectiveness of IMIML-TPSF against state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Yang S J, Jiang Y, Zhou Z H. Multi-instance multi-label learning with weak label. In: Proceedings of the 23rd International Joint Conference on Artificial Intelligence. 2013, 1862–1868

    MATH  Google Scholar 

  2. Chen A, Dhingra B. Hierarchical multi-instance multi-label learning for detecting propaganda techniques. In: Proceedings of the 8th Workshop on Representation Learning for NLP. 2023, 155–163

    MATH  Google Scholar 

  3. Julia, Da Silva E. A deep learning system to perform multi-instance multi-label event classification in video game footage. Universidade Federal de Uberlandia, Dissertation, 2022

    Book  MATH  Google Scholar 

  4. Pan Z, Wang B, Zhang R, Wang S, Li Y, Li Y. MIML-GAN: a GAN-based algorithm for multi-instance multi-label learning on overlapping signal waveform recognition. IEEE Transactions on Signal Processing, 2023, 71: 859–872

    Article  MathSciNet  MATH  Google Scholar 

  5. Loukas C, Sgouros N P. Multi-instance multi-label learning for surgical image annotation. The International Journal of Medical Robotics and Computer Assisted Surgery, 2020, 16(2): e2058

    MATH  Google Scholar 

  6. Lai Q, Zhou J, Gan Y, Vong C M, Chen C L P. Single-stage broad multi-instance multi-label learning (BMIML) with diverse inter-correlations and its application to medical image classification. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024, 8(1): 828–839

    Article  Google Scholar 

  7. Zhou Z H, Zhang M L, Huang S J, Li Y F. Multi-instance multi-label learning. Artificial Intelligence, 2012, 176(1): 2291–2320

    Article  MathSciNet  MATH  Google Scholar 

  8. Wu J S, Huang S J, Zhou Z H. Genome-wide protein function prediction through multi-instance multi-label learning. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2014, 11(5): 891–902

    Article  MATH  Google Scholar 

  9. Liu J, Tang X, Cui S, Guan X. Predicting the function of rice proteins through multi-instance multi-label learning based on multiple features fusion. Briefings in Bioinformatics, 2022, 23(3): bbac095

    Article  Google Scholar 

  10. Li Y F, Hu J H, Jiang Y, Zhou Z H. Towards discovering what patterns trigger what labels. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence. 2012, 1012–1018

    MATH  Google Scholar 

  11. Yang M, Tang W T, Min F. Multi-instance multi-label learning based on parallel attention and local label manifold correlation. In: Proceedings of the 9th IEEE International Conference on Data Science and Advanced Analytics. 2022, 1–10

    MATH  Google Scholar 

  12. Qiu S, Wang M, Yang Y, Yu G, Wang J, Yan Z, Domeniconi C, Guo M. Meta multi-instance multi-label learning by heterogeneous network fusion. Information Fusion, 2023, 94: 272–283

    Article  MATH  Google Scholar 

  13. Su C, Yan Z, Yu G. Cost-effective multi-instance multilabel active learning. International Journal of Intelligent Systems, 2021, 36(12): 7177–7203

    Article  Google Scholar 

  14. Yu G, Xing Y, Wang J, Domeniconi C, Zhang X. Multiview multi-instance multilabel active learning. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(9): 4311–4321

    Article  MATH  Google Scholar 

  15. Sánchez J, Perronnin F, Mensink T, Verbeek J. Image classification with the fisher vector: theory and practice. International Journal of Computer Vision, 2013, 105(3): 222–245

    Article  MathSciNet  MATH  Google Scholar 

  16. Newton J. Statistical analysis of finite mixture distributions. Journal of the International Biometric Society, 1986, 42(3): 679–680

    MathSciNet  MATH  Google Scholar 

  17. Ma Z, Chen S. A similarity-based framework for classification task. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(5): 5438–5443

    MathSciNet  MATH  Google Scholar 

  18. Poladi I, Ishwardas H. Review paper on error correcting output code based on multiclass classification. International Journal of Scientific Research, 2013, 2(2): 134–136

    Article  MATH  Google Scholar 

  19. Zadeh A, Chen M, Poria S, Cambria E, Morency L P. Tensor fusion network for multimodal sentiment analysis. In: Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 1103–1114

    MATH  Google Scholar 

  20. Liu J, Ji S, Ye J. SLEP: Sparse Learning with Efficient Projections. Phoenix City: Arizona State University, 2009

    MATH  Google Scholar 

  21. Zhou Z L, Zhang M L. Multi-instance multi-label learning with application to scene classification. In: Proceedings of the 20th Annual Conference on Neural Information Processing Systems. 2006, 1609–1616

    MATH  Google Scholar 

  22. Zhang M L, Zhou Z H. M3MIML: a maximum margin method for multi-instance multi-label learning. In: Proceedings of the 8th IEEE International Conference on Data Mining. 2008, 688–697

    MATH  Google Scholar 

  23. Maron O, Ratan A L. Multiple-instance learning for natural scene classification. In: Proceedings of the 15th International Conference on Machine Learning. 1998, 341–349

    MATH  Google Scholar 

  24. Andrews S, Tsochantaridis I, Hofmann T. Support vector machines for multiple-instance learning. In: Proceedings of the 15th International Conference on Neural Information Processing Systems. 2002, 577–584

    MATH  Google Scholar 

  25. Briggs F, Fern X Z, Raich R. Rank-loss support instance machines for MIML instance annotation. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2012, 534–542

    MATH  Google Scholar 

  26. Frey P W, Slate D J. Letter recognition using Holland-style adaptive classifiers. Machine Learning, 1991, 6(2): 161–182

    Article  MATH  Google Scholar 

  27. Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick C L. Microsoft COCO: common objects in context. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 740–755

    Google Scholar 

  28. Luo T, Zhang W, Qiu S, Yang Y, Yi D, Wang G, Ye J, Wang J. Functional annotation of human protein coding isoforms via non-convex multi-instance learning. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2017, 345–354

    Chapter  MATH  Google Scholar 

  29. Panwar B, Menon R, Eksi R, Li H D, Omenn G S, Guan Y. Genome-wide functional annotation of human protein-coding splice variants using multiple instance learning. Journal of Proteome Research, 2016, 15(6): 1747–1753

    Article  Google Scholar 

  30. The ENCODE Project Consortium. An integrated encyclopedia of DNA elements in the human genome. Nature, 2012, 489(7414): 57–74

    Article  MATH  Google Scholar 

  31. Charte F, Rivera A, Del Jesus M J, Herrera F. A first approach to deal with imbalance in multi-label datasets. In: Proceedings of the 8th International Conference on Hybrid Artificial Intelligent Systems. 2013, 150–160

    MATH  Google Scholar 

  32. Charte F, Rivera A, Del Jesus M J, Herrera F. Concurrence among imbalanced labels and its influence on multilabel resampling algorithms. In: Proceedings of the 9th International Conference on Hybrid Artificial Intelligence Systems. 2014, 110–121

    Google Scholar 

  33. Liu Z, Wei P, Jiang J, Cao W, Bian J, Chang Y. MESA: boost ensemble imbalanced learning with meta-sampler. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1212

    MATH  Google Scholar 

  34. Shan J, Hou C, Tao H, Zhuge W, Yi D. Randomized multi-label subproblems concatenation via error correcting output codes. Neurocomputing, 2020, 410: 317–327

    Article  MATH  Google Scholar 

  35. Liu X Y, Wang S T, Zhang M L. Transfer synthetic over-sampling for class-imbalance learning with limited minority class data. Frontiers of Computer Science, 2019, 13(5): 996–1009

    Article  MATH  Google Scholar 

  36. Saglam F, Cengiz M A. A novel SMOTE-based resampling technique trough noise detection and the boosting procedure. Expert Systems with Applications, 2022, 200: 117023

    Article  MATH  Google Scholar 

  37. Yu D, Wang L, Chen X, Chen J. Using BiLSTM with attention mechanism to automatically detect self-admitted technical debt. Frontiers of Computer Science, 2021, 15(4): 154208

    Article  Google Scholar 

  38. Liu J, Feng R, Chen P, Wang X, Ni Y. Dynamic loss reweighting method based on cumulative classification scores for long-tailed remote sensing image classification. Remote Sensing, 2023, 15(2): 394

    Article  MATH  Google Scholar 

  39. Ji Z, Ni J, Liu X, Pang Y. Teachers cooperation: team-knowledge distillation for multiple cross-domain few-shot learning. Frontiers of Computer Science, 2023, 17(2): 172312

    Article  Google Scholar 

  40. Wu Y, Dong G, Liang L, Zhao Y, Zhang K. Group-wise co-salient object detection via multi-view self-labeling novel class discovery. Frontiers of Computer Science, 2024, 18(2): 182709

    Article  Google Scholar 

  41. Guo W, Zhuang F Z, Zhang X, Tong Y Q, Dong J. A comprehensive survey of federated transfer learning: challenges, methods and applications. Frontiers of Computer Science, 2024, 18(6): 186356

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 62376281 and 62036013), and the NSF for Huxiang Young Talents Program of Hunan Province (2021RC3070).

Author information

Authors and Affiliations

  1. College of Science, National University of Defense Technology, Changsha, 410073, China

    Xinyue Zhang & Tingjin Luo

Authors
  1. Xinyue Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Tingjin Luo
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Tingjin Luo.

Ethics declarations

Competing interests The authors declare that they have no competing interests or financial conflicts to disclose.

Additional information

Xinyue Zhang received the BS in School of Mathematics and Statistics, Anhui Normal University, China in 2022. She is currently a master candidate at the National University of Defense Technology, China. Her research interests include machine learning, big data analysis, and computer vision.

Tingjin Luo received the BS, Master, and PhD degrees from the National University of Defense Technology (NUDT), China. He is currently an Associate Professor with the College of Science of NUDT. He has authored more than 40 papers in journals and conferences, such as IEEE TPAMI, IEEE TKDE, IEEE TCYB, IEEE TIP, and ACM KDD. He has been a Program Committee member of several conferences including ICML, IJCAI, AAAI, and ICLR etc. His research interests include machine learning, multi-media analysis, data mining and computer vision.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Luo, T. Imbalanced multi-instance multi-label learning via tensor product-based semantic fusion. Front. Comput. Sci. 19, 198346 (2025). https://doi.org/10.1007/s11704-024-40192-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-024-40192-5

Keywords