Abstract
In practical machine learning scenarios, there may be multiple predictors available for the same task. Ensemble learning combines these predictors to obtain a predictor with higher generalization performance. Weighted averaging is one of the most basic methods, which can be generalized by a formulation of an exponential mixture model. In this formulation, the weight optimization in ensemble learning is represented as selecting the predictors to be used by concentrating the weights and maintaining the diversity of the predictors to be used by distributing the weights. It has been theoretically shown that if the balance between these two factors is adjusted to be equal, the generalization performance improves. However, having an equal balance may not always be optimal, as there could be better alternatives. In this paper, we propose a method to obtain a predictor with higher generalization performance by adjusting the balance between selecting predictors and maintaining their diversity. Numerical experiments showed that when there is a large amount of training data and an unbiased label distribution, adjusting the balance can result in improved generalization performance.
This work was supported in part by the Japan Society for the Promotion of Science through Grants-in-Aid for Scientific Research (C) (23K11111).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning, vol. 4. Springer, New York (2006)
Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)
Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)
David, H.W.: Stacked generalization. Neural Netw. 5(2), 241–259 (1992)
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)
Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat., 1189–1232 (2001)
Hastie, T., Tibshirani, R., Friedman, J.H., Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, vol. 2. Springer, New York (2009). https://doi.org/10.1007/978-0-387-84858-7
Ke, G., et al.: LightGBM: a highly efficient gradient boosting decision tree. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Schapire, R.E.: The strength of weak learnability. Mach. Learn. 5(2), 197–227 (1990)
Uchida, M., Maehara, Y., Shioya, H.: Unsupervised weight parameter estimation method for ensemble learning. J. Mathe. Model. Algorithms 10(4), 307–322 (2011)
Uchida, M.: Tight lower bound of generalization error in ensemble learning. In: 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS) and 15th International Symposium on Advanced Intelligent Systems (ISIS), pp. 1130–1133 (2014). https://doi.org/10.1109/SCIS-ISIS.2014.7044723
Ueda, N., Nakano, R.: Generalization error of ensemble estimators. In: Proceedings of International Conference on Neural Networks 1996 (ICNN 1996), vol. 3, pp. 90–95 (1996)
Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. SIGKDD Explor. 15(2), 49–60 (2013). https://doi.org/10.1145/2641190.2641198
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sugiyama, K., Uchida, M. (2023). Balancing Selection and Diversity in Ensemble Learning with Exponential Mixture Model. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial Neural Networks and Machine Learning – ICANN 2023. ICANN 2023. Lecture Notes in Computer Science, vol 14256. Springer, Cham. https://doi.org/10.1007/978-3-031-44213-1_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-44213-1_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44212-4
Online ISBN: 978-3-031-44213-1
eBook Packages: Computer ScienceComputer Science (R0)