Abstract
A common approach to aggregate classification estimates in an ensemble of decision trees is to either use voting or to average the probabilities for each class. The latter takes uncertainty into account, but not the reliability of the uncertainty estimates (so to say, the “uncertainty about the uncertainty”). More generally, much remains unknown about how to best combine probabilistic estimates from multiple sources. In this paper, we investigate a number of alternative prediction methods. Our methods are inspired by the theories of probability, belief functions and reliable classification, as well as a principle that we call evidence accumulation. Our experiments on a variety of data sets are based on random decision trees which guarantees a high diversity in the predictions to be combined. Somewhat unexpectedly, we found that taking the average over the probabilities is actually hard to beat. However, evidence accumulation showed consistently better results on all but very small leafs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Downloaded from the UCI Machine Learning Repository http://archive.ics.uci.edu and OpenML https://www.openml.org/.
- 2.
Our code is publicly available at https://github.com/olfub/RDT-Uncertainty.
References
Bostrom, H.: Estimating class probabilities in random forests. In: 6th International Conference on Machine Learning and Applications, pp. 211–216 (2007)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Costa, V.S., Farias, A.D.S., Bedregal, B., Santiago, R.H., de P. Canuto, A.M.: Combining multiple algorithms in classifier ensembles using generalized mixture functions. Neurocomputing 313, 402–414 (2018)
Denœux, T.: The cautious rule of combination for belief functions and some extensions. In: 9th International Conference on Information Fusion, pp. 1–8 (2006)
Dietterich, T.G.: Ensemble methods in machine learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45014-9_1
Fan, W., Wang, H., Yu, P.S., Ma, S.: Is random model better? On its accuracy and efficiency. In: 3rd IEEE International Conference on Data Mining (2003)
Farias, A.D.S., Santiago, R.H.N., Bedregal, B.: Some properties of generalized mixture functions. In: IEEE International Conference on Fuzzy Systems, pp. 288–293 (2016)
Kulessa, M., Loza Mencía, E.: Dynamic classifier chain with random decision trees. In: Soldatova, L., Vanschoren, J., Papadopoulos, G., Ceci, M. (eds.) DS 2018. LNCS (LNAI), vol. 11198, pp. 33–50. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01771-2_3
Lu, Y.: Knowledge integration in a multiple classifier system. Appl. Intell. 6(2), 75–86 (1996)
Nguyen, T.T., Pham, X.C., Liew, A.W.C., Pedrycz, W.: Aggregation of classifiers: a justifiable information granularity approach. IEEE Trans. Cybern. 49(6), 2168–2177 (2018)
Nguyen, V.L., Destercke, S., Masson, M.H., Hüllermeier, E.: Reliable multi-class classification based on pairwise epistemic and aleatoric uncertainty. In: International Joint Conference on Artificial Intelligence, pp. 5089–5095 (2018)
Provost, F., Domingos, P.: Tree induction for probability-based ranking. Mach. Learn. 52(3), 199–215 (2003)
Raza, M., Gondal, I., Green, D., Coppel, R.L.: Classifier fusion using dempster-shafer theory of evidence to predict breast cancer tumors. In: IEEE Region 10 International Conference TENCON, pp. 1–4 (2006)
Ruta, D., Gabrys, B.: An overview of classifier fusion methods. Comput. Inf. Syst. 7, 1–10 (2000)
Shafer, G.: A Mathematical Theory of Evidence, vol. 42. Princeton University Press (1976)
Shaker, M.H., Hüllermeier, E.: Aleatoric and epistemic uncertainty with random forests. In: Berthold, M.R., Feelders, A., Krempl, G. (eds.) IDA 2020. LNCS, vol. 12080, pp. 444–456. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-44584-3_35
Smets, P.: Belief functions: the disjunctive rule of combination and the generalized Bayesian theorem. Int. J. Approximate Reasoning 9(1), 1–35 (1993)
Wolpert, D.: Stacked generalization. Neural Netw. 5(2), 241–259 (1992)
Yuksel, S.E., Wilson, J.N., Gader, P.D.: Twenty years of mixture of experts. IEEE Trans. Neural Netw. Learn. Syst. 23(8), 1177–1193 (2012)
Zhou, S., Mentch, L.: Trees, forests, chickens, and eggs: when and why to prune trees in a random forest. arXiv preprint arXiv:2103.16700 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Busch, F., Kulessa, M., Loza Mencía, E., Blockeel, H. (2021). Combining Predictions Under Uncertainty: The Case of Random Decision Trees. In: Soares, C., Torgo, L. (eds) Discovery Science. DS 2021. Lecture Notes in Computer Science(), vol 12986. Springer, Cham. https://doi.org/10.1007/978-3-030-88942-5_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-88942-5_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88941-8
Online ISBN: 978-3-030-88942-5
eBook Packages: Computer ScienceComputer Science (R0)