Imbalanced Multi-instance Multi-label Learning via Coding Ensemble and Adaptive Thresholds
Pages 5413 - 5422
Abstract
Multi-instance multi-label learning (MIML), which deals with objects with complex structures and multiple semantics, plays a crucial role in various fields. In practice, the naturally skewed label distribution and label dependence contribute to the issue of label imbalance in MIML, which is crucial but rarely studied. Most existing MIML methods often produce biased models due to the ignorance of inter-class variations in imbalanced data. To address this issue, we propose a novel imbalanced multi-instance multi-label learning method named IMIMLC, based on the error-correcting coding ensemble and an adaptive threshold strategy. Specifically, we design a feature embedding method to extract the structural information of each object via Fisher vectors and eliminate inexact supervision. Subsequently, to alleviate the disturbance caused by the imbalanced distribution, a novel ensemble model is constructed by concatenating the error-correcting codes of randomly selected subtasks. Meanwhile, IMIMLC trains binary base classifiers on small-scale data blocks partitioned by our codes to enhance their diversity and then learns more reliable results to improve model robustness for the imbalance issue. Furthermore, IMIMLC adaptively learns thresholds for each individual label by margin maximization, preventing inaccurate predictions caused by the semantic discrepancy across many labels and their unbalanced ratios. Finally, extensive experimental results on various datasets validate the effectiveness of IMIMLC against state-of-the-art approaches.
References
[1]
Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. 2004. Learning Multi-label Scene Classification. Pattern Recognition, Vol. 37, 9 (2004), 1757--1771.
[2]
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, Vol. 16 (2002), 321--357.
[3]
Nitesh V. Chawla, Aleksandar Lazarevic, Lawrence O. Hall, and Kevin W. Bowyer. 2003. SMOTEBoost: Improving Prediction of the Minority Class in Boosting. In Proceedings of the Knowledge Discovery in Databases (PKDD). Springer, Berlin, Heidelberg, 107--119.
[4]
Anni Chen and Bhuwan Dhingra. 2023. Hierarchical Multi-Instance Multi-Label Learning for Detecting Propaganda Techniques. In Proceedings of the Workshop on Representation Learning for NLP (RepL4NLP). Association for Computational Linguistics, Toronto, Canada, 155--163.
[5]
Mengyuan Ding, Shanshan Zhang, and Jian Yang. 2021. Improving pedestrian detection from a long-tailed domain perspective. In Proceedings of the ACM International Conference on Multimedia (ACM MM). Association for Computing Machinery, Chengdu, China, 2918--2926.
[6]
Jian Guan, Jiabei Liu, Jianguo Sun, Pengming Feng, Shuai Tong, and Wenwu Wang. 2020. Meta Metric Learning for Highly Imbalanced Aerial Scene Classification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, Barcelona, Spain, 4047--4051.
[7]
Jingjing Guo, Qian Wang, Yiting Li, and Pengkun Liu. 2020. Faccade defects classification from imbalanced dataset using meta learning-based convolutional neural network. Computer-Aided Civil and Infrastructure Engineering, Vol. 35, 12 (2020), 1403--1418.
[8]
Angélica Guzmán-Ponce, José Salvador Sánchez, Rosa Maria Valdovinos, and José Raymundo Marcial-Romero. 2021. DBIG-US: A two-stage under-sampling algorithm to face the class imbalance problem. Expert Systems with Applications, Vol. 168 (2021), 1--12.
[9]
Hui Huang, Shihao Wu, Minglun Gong, Daniel Cohen-Or, Uri Ascher, and Hao Zhang. 2013. Edge-aware point set resampling. ACM Transactions on Graphics, Vol. 32, 9 (2013), 1--12.
[10]
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. 2020. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Virtual, 7610--7619.
[11]
Julia and Etienne Da Silva. 2022. A deep learning system to perform multi-instance multi-label event classification in video game footage. Ph.,D. Dissertation. Universidade Federal de Uberlândia.
[12]
William Karush. 2014. Minima of Functions of Several Variables with Inequalities as Side Conditions. In Traces and Emergence of Nonlinear Programming, Giorgio Giorgi and Tinne Hoff Kjeldsen (Eds.). Springer, Basel, 217--245.
[13]
Michał Koziarski, Michał Wo'zniak, and Bartosz Krawczyk. 2020. Combined Cleaning and Resampling algorithm for multi-class imbalanced data with label noise. Knowledge-Based Systems, Vol. 204 (2020), 1--18.
[14]
Harold W. Kuhn and Albert W. Tucker. 2013. Nonlinear programming. In Traces and emergence of nonlinear programming. Springer, Basel, 247--258.
[15]
Qi Lai, Jianhang Zhou, Yanfen Gan, Chiman Vong, and C.L. Philip Chen. 2024. Single-Stage Broad Multi-Instance Multi-Label Learning (BMIML) with Diverse Inter-Correlations and its application to medical image classification. IEEE Transactions on Emerging Topics in Computational Intelligence, Vol. 8, 1 (2024), 828--839.
[16]
Yufeng Li, Juhua Hu, Yuang Jiang, and Zhihua Zhou. 2012. Towards Discovering What Patterns Trigger What Labels. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). AAAI, Toronto, Canada, 1012--1018.
[17]
Jiahang Liu, Ruilei Feng, Peng Chen, Xiaozhen Wang, and Yue Ni. 2023. Dynamic Loss Reweighting Method Based on Cumulative Classification Scores for Long-Tailed Remote Sensing Image Classification. Remote Sensing, Vol. 15, 394 (2023), 1--26.
[18]
Jing Liu, Xinghua Tang, Shuanglong Cui, and Xiao Guan. 2022. Predicting the Function of Rice Proteins Through Multi-instance Multi-label Learning Based on Multiple Features Fusion. Briefings in Bioinformatics, Vol. 23, 3 (2022), 1--11.
[19]
Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tieyan Liu. 2020. Self-paced Ensemble for Highly Imbalanced Massive Data Classification. In Proceedings of the IEEE International Conference on Data Engineering (ICDE). IEEE, Dallas, Texas, USA, 841--852.
[20]
Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. 2020. MESA: Boost Ensemble Imbalanced Learning with MEta-SAmpler. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), Vol. 33. MIT Press, Virtual, 14463--14474.
[21]
Constantinos Loukas and Nicholas P Sgouros. 2020. Multi-instance multi-label learning for surgical image annotation. The International Journal of Medical Robotics and Computer Assisted Surgery, Vol. 16, 2 (2020), 1--12.
[22]
Tingjin Luo, Weizhong Zhang, Shuang Qiu, Yang Yang, Dongyun Yi, Guangtao Wang, Jieping Ye, and Jie Wang. 2017. Functional annotation of human protein coding isoforms via non-convex multi-instance learning. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (KDD). Association for Computing Machinery, Halifax, Nova Scotia, Canada, 345--354.
[23]
Nuno Moniz and Vitor Cerqueira. 2021. Automated imbalanced classification via meta-learning. Expert Systems with Applications, Vol. 178 (2021), 1--14.
[24]
Zesi Pan, Bo Wang, Ruibin Zhang, Shafei Wang, Yunjie Li, and Yan Li. 2023. MIML-GAN: A GAN-based algorithm for multi-instance multi-label learning on overlapping signal waveform recognition. IEEE Transactions on Signal Processing, Vol. 71 (2023), 859--872.
[25]
Zhiliang Peng, Wei Huang, Zonghao Guo, Xiaosong Zhang, Jianbin Jiao, and Qixiang Ye. 2021. Long-tailed distribution adaptation. In Proceedings of the ACM International Conference on Multimedia (ACM MM). Association for Computing Machinery, Chengdu, China, 3275--3282.
[26]
Mohsen Pirizadeh, Nafiseh Alemohammad, Mohammad Manthouri, and Meysam Pirizadeh. 2021. A new machine learning ensemble model for class imbalance problem of screening enhanced oil recovery methods. Journal of Petroleum Science and Engineering, Vol. 198 (2021), 1--22.
[27]
John Platt. 1998. Sequential minimal optimization: A fast algorithm for training support vector machines. Technical Report MSR-TR-98--14. Microsoft.
[28]
Irfan Poladi and Hitesh Ishwardas. 2012. Review Paper on Error Correcting Output Code Based on Multiclass Classification. International Journal of Scientific Research, Vol. 2, 2 (2012), 134--136.
[29]
R. Tyrrell Rockafellar. 1993. Lagrange Multipliers and Optimality. SIAM Review, Vol. 35, 2 (1993), 183--238.
[30]
Fatih Sauglam and Mehmet Ali Cengiz. 2022. A novel SMOTE-based resampling technique trough noise detection and the boosting procedure. Expert Systems with Applications, Vol. 200 (2022), 1--12.
[31]
Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. 2013. Image Classification with the Fisher Vector: Theory and Practice. International Journal of Computer Vision, Vol. 105, 3 (2013), 222--245.
[32]
Jincheng Shan, Chenping Hou, Hong Tao, Wenzhang Zhuge, and Dongyun Yi. 2020. Randomized multi-label subproblems concatenation via error correcting output codes. Neurocomputing, Vol. 410 (2020), 317--327.
[33]
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019. Meta-Weight-Net: Learning an Explicit Mapping for Sample Weighting. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), Vol. 32. MIT Press, Vancouver, Canada, 1--12.
[34]
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2022. Learning from Noisy Labels with Deep Neural Networks: A Survey. IEEE Transactions on Neural Networks and Learning Systems, Vol. 34, 11 (2022), 8135--8153.
[35]
Cong Su, Zhongmin Yan, and Guoxian Yu. 2021. Cost-effective Multi-instance Multi-label Active Learning. International Journal of Intelligent Systems, Vol. 36, 12 (2021), 7177--7203.
[36]
Adane Nega Tarekegn, Mario Giacobini, and Krzysztof Michalak. 2021. A review of methods for imbalanced multi-label classification. Pattern Recognition, Vol. 118 (2021), 1--12.
[37]
Jiansheng Wu, Shengjun Huang, and Zhihua Zhou. 2014. Genome-wide Protein Function Prediction Through Multi-instance Multi-label Learning. IEEE/ACM Transactions on Computational Biology and Bioinformatics, Vol. 11, 5 (2014), 891--902.
[38]
Jialian Wu, Liangchen Song, Tiancai Wang, Qian Zhang, and Junsong Yuan. 2020. Forest R-CNN: Large-vocabulary long-tailed object detection and instance segmentation. In Proceedings of the ACM international conference on multimedia (ACM MM). Association for Computing Machinery, Seattle, United States, 1570--1578.
[39]
Xinshun Xu, Yuan Jiang, Xiangyang Xue, and Zhihua Zhou. 2012. Semi-supervised multi-instance multi-label learning for video annotation task. In Proceedings of the ACM international conference on Multimedia (ACM MM). Association for Computing Machinery, Nara, Japan, 737--740.
[40]
Xinshun Xu, Xiangyang Xue, and Zhihua Zhou. 2011. Ensemble multi-instance multi-label learning approach for video annotation task. In Proceedings of the ACM international conference on Multimedia (ACM MM). Association for Computing Machinery, Scottsdale Arizona, USA, 1153--1156.
[41]
Mei Yang, Wentao Tang, and Fan Min. 2022. Multi-instance Multi-label Learning Based on Parallel Attention and Local Label Manifold Correlation. In Proceedings of the IEEE International Conference on Data Science and Advanced Analytics (DSAA). IEEE, Virtual, 1--10.
[42]
Shujun Yang, Yuan Jiang, and Zhihua Zhou. 2013. Multi-instance Multi-label Learning with Weak Label. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Morgan Kaufmann, Beijing, China, 1862--1868.
[43]
Zhili Zhang and Minling Zhang. 2006. Multi-instance Multi-label Learning with Application to Scene Classification. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), Vol. 19. MIT Press, Vancouver, Canada, 1609--1616.
[44]
Hao Zhou, Jun Zhang, Tingjin Luo, Yazhou Yang, and Jun Lei. 2023. Debiased Scene Graph Generation for Dual Imbalance Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, 4 (2023), 4274--4288.
[45]
Zhihua Zhou. 2017. A brief introduction to weakly supervised learning. National Science Review, Vol. 5, 1 (2017), 44--53.
[46]
Zhihua Zhou. 2022. Open-environment machine learning. National Science Review, Vol. 9, 8 (2022), 1--11.
[47]
Zhihua Zhou. 2022. Rehearsal: learning from prediction to decision. Frontiers of Computer Science, Vol. 16 (2022), 2095--2236.
[48]
Zhihua Zhou, Minling Zhang, Shengjun Huang, and Yufeng Li. 2012. Multi-instance Multi-label Learning. Artificial Intelligence, Vol. 176, 1 (2012), 2291--2320.
Index Terms
- Imbalanced Multi-instance Multi-label Learning via Coding Ensemble and Adaptive Thresholds
Recommendations
Imbalanced multi-instance multi-label learning via tensor product-based semantic fusion
AbstractWith powerful expressiveness of multi-instance multi-label learning (MIML) for objects with multiple semantics and its great flexibility for complex object structures, MIML has been widely applied to various applications. In practical MML tasks, ...
A multi-instance multi-label learning algorithm based on instance correlations
Existing multi-instance multi-label learning algorithms generally assume that instances in a bag are independent of each other, which is difficult to be guaranteed in practical applications. A novel multi-instance multi-label learning algorithm is ...
Bayesian multi-instance multi-label learning using Gaussian process prior
Multi-instance multi-label learning (MIML) is a newly proposed framework, in which the multi-label problems are investigated by representing each sample with multiple feature vectors named instances. In this framework, the multi-label learning task ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- the National Science Foundation of China Grant
- the NSF for Huxiang Young Talents Program of Hunan Province under Grant
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 136Total Downloads
- Downloads (Last 12 months)136
- Downloads (Last 6 weeks)93
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in