Abstract
In this paper, a class of classifier fusion methods are compared to verify the impact of the use of some prior information about individual classifiers during fusion of probabilistic classifier outputs. In particular, we compare two versions (i.e., uninformed and informed versions) of a performance-agnostic fusion of probabilistic classifier outputs from Masakuna et al. (2020) (called Yayambo). Yayambo is iterative and treated black-box classifiers. For this paper, cases where prior information, i.e., performances of individual classifiers in the form of accuracy is taken into account for fusion of classifier outputs, are considered. Then we discuss the relevance of prior information for combination of probabilistic classifier outputs. The experiments have demonstrated that classifier fusion methods, for both informed and uninformed fusion methods, achieve different performances, i.e., the differences are significant in general (using the p-value and the effect size (Gail & Richard, 2012)). Surprisingly, in some particular cases and under the same experimental conditions, the two versions of Yayambo achieve similar results (using the \(p-\)value). This means that one might not need to carefully, for some situations, select a classifier fusion method. We consider 12 classifier fusion methods (5 uninformed and 7 informed), use 8 data sets and apply different experimental settings to address our research question.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We built the PCA model using all of the training data.
References
Masakuna, J. F., Simukai, U. W., Kroon, S. (2020). Performance-Agnostic Fusion of Probabilistic Classifier Outputs. 2020 IEEE 23rd International Conference on Information Fusion (FUSION), 1–8.
Schapire, R. E. (2003). The Boosting Approach to Machine Learning: An Overview. Nonlinear estimation and classification. 149–171.
Finkelstein, A. C. W., Gabbay, D., Hunter A., Kramer, J., Nuseibeh, B. (1994). Inconsistency Handling in Multiperspective Specifications. IEEE Trans Softw Eng., 20(8), 569–578.
Castanedo, F. (2013) A Review of Data Fusion Techniques. Sci World J.
Kittler, J. (1998). Combining Classifiers: A Theoretical Framework. Pattern Analysis and Applications., 1, 18–27.
Kuncheva, L. I. (2002). Switching between Selection and Fusion in Combining Classifiers: An Experiment. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics). 32(2), 146–156.
Soofi, A. A., & Awan, A. (2017). Classification Techniques in Machine Learning: Applications and Issues. J. Basic Appl. Sci. 13, 459–465.
Zhang, L., Wang, S., & Liu, B. (2018). Deep Learning for Sentiment Analysis: A Survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery., 8(4), e1253.
Chen, N., & Blostein, D. (2007). A Survey of Document Image Classification: Problem Statement. Classifier Architecture and Performance Evaluation International Journal of Document Analysis and Recognition (IJDAR)., 10, 1–16.
Alurkar, A. A., Ranade, S. B., Joshi, S. V., Ranade, S. S., Sonewar, P. A., Mahalle, P. N., Deshpande, A. V. (2017). A Proposed Data Science Approach for Email Spam Classification using Machine Learning Techniques. Internet of Things Business Models, Users, and Networks. IEEE 1–5
Langville, A. N., & Meyer, C. D. (2012). Who’s# 1?: the Science of Rating and Ranking. Princeton University Press.
Utete, S. W., Barshan, B., & Ayrulu, B. (1999). Voting as Validation in Robot Programming. Int J Robot Res., 18(4), 401–413.
Lynch, K. M., Schwartz, I. B., Yang, P., & Freeman, R. A. (2008). Decentralized Environmental Modeling by Mobile Sensor Networks. IEEE Transactions on Robotics, 24(3), 710–724.
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016). A Survey of Transfer Learning. Journal of Big data, 3(1), 1–40.
Mangai, U. G., Samanta, S., Das, S., & Chowdhury, P. R. (2010). A survey of Decision Fusion and Feature Fusion Strategies for Pattern Classification. IETE Technical Review, 27(4), 293–307.
Wolpert, D. H. (1996). The Lack of a Priori Distinctions Between Learning Algorithms. Neural Comput, 8(7), 1341–1390.
Keynes, J. M. (1921). Chapter IV: The Principle of Indifference. A Treatise on Probability, 4, 41–64.
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998). On Combining Classifiers. IEEE Trans. Pattern Analysis. Mach. Intell., 20(3), 226–239.
Emerson, P. (2013). The Original Borda Count and Partial Voting. Soc Choice Welf, 40(2), 353–358.
Kim, H. C., Ghahramani, Z. (2012). Bayesian Classifier Combination. Artificial Intelligence and Statistics, PMLR pp 619–627.
Fei, L., Xia, J., Feng, Y., & Liu, L. (2019). A Novel Method to Determine Basic Probability Assignment in Dempster-Shafer Theory and its Application in Multi-Sensor Information Fusion. Int J Distrib Sens Networks., 15(7), 1550147719865876.
Asuncion, A., Newman, D. (2007). UCI Machine Learning Repository
Nene, S. A., Nayar, S. K., Murase, H. (1996). Columbia Object Image Library (coil-100), Computer Vision Laboratory, Computer Science Department, Columbia University, TR CUCS-005-96 Feb.
Ruta, D., & Gabrys, B. (2000). An Overview of Classifier Fusion Methods. Comput. Info. syst., 7(1), 1–10.
Domingos, P. (2000). Bayesian Averaging of Classifiers and the Overfitting Problem ICML, 747, 223–230.
Deng, X., Liu, Q., Deng, Y., & Mahadevan, S. (2000). An Improved Method to Construct Basic Probability Assignment Based on the Confusion Matrix for Classification Problem. Information Sciences, 340, 250–261.
Smets, P., Kennes, R. (2008). The Transferable Belief Model, Classic Works of the Dempster-Shafer Theory of Belief Functions, pp 693–736.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V. et al. (2011). Scikit-learn: Machine Learning in Python, J Mach Learn Res 12, 2825–2830. Oct
Fan, R. E., Chang, K. W., Hsieh, C. J., Wang, X. R., & Lin, C. J. (2008). LIBLINEAR: A Library for Large Linear Classification. J. Mach. Lear. Res. 9, 1871–1874.
Ho, Y. C., & Pepyne, D. L. (2002). Simple Explanation of the No-Free-Lunch Theorem and its Implications. J. Optim. Theory Appl. 115, 549–570.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning
Fan, J., Upadhye, S., & Worster, A. (2006). Understanding Receiver Operating Characteristic (ROC) Curves. Canadian J Emerg Med, 8(1), 19–20.
Gail, M. S., & Richard, F. (2012). Using Effect Size-or Why the P Value Is Not Enough. J. Grad. Med. Educ. 4(3), 279–282.
Wilcoxon, F. (1992). Individual Comparisons by Ranking Methods, Breakthroughs in Statistics pp 196–202
Cohen, J. (1962). The Statistical Power of Abnormal-Social Psychological Research: A Review. J Abnorm Soc Psychol, 65(3), 145.
Brossart, D. F., Laird, V. C., & Armstrong, T. W. (2018). Interpreting Kendall’s Tau and Tau-U for Single-Case Experimental Designs. Cogent Psychology, 5(1), 1518687.
Bisong, E. (2019). Introduction to Scikit-learn, Building Machine Learning and Deep Learning Models on Google Cloud Platform, pp 215–229
Masakuna, J. F. (2020). Active Strategies for Coordination of Solitary Robots. Stellenbosch: Stellenbosch University.
Masakuna, J. F., Kafunda, P. K., Kayembe, M. L. (2022). On the Theoretical Convergence and Error Sensitivity Analysis of Yayambo for Fusion of Probabilistic Classifier Outputs, The 25th International Conference on Information Fusion. p 01–08
Funding
No funding was granted to this research.
Author information
Authors and Affiliations
Contributions
In this manuscript authors have conducted a statistical analysis between uninformed and informed classifier fusion methods to verify whether the consideration of prior information about individual classifiers is invaluable during the process of fusion. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare that there is no conflict of interest.
Ethical Conduct
The manuscript is only submitted to the Journal of Classification. The submitted work is original and is not published elsewhere in any form or language (partially or in full).
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
MASAKUNA, J.F., Kafunda, P.K. Do Prior Information on Performance of Individual Classifiers for Fusion of Probabilistic Classifier Outputs Matter?. J Classif 40, 468–487 (2023). https://doi.org/10.1007/s00357-023-09444-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00357-023-09444-0