Abstract
It has been demonstrated how random subspaces can be used to create a Diversified Random Forest, which in turn can lead to better performance in terms of predictive accuracy. Motivated by the fact that each subsforest is built using a set of features that can overlap with those sets of features in other subforests, we hypothesise that using Replicator Dynamics can perform a collective feature engineering, by allowing subforests with better performance to grow and those with lower performance to shrink. In this paper, we propose a new method to further improve the performance of Diversified Random Forest using Replicator Dynamics which has been used extensively in evolutionary game dynamics. A thorough experimental study on 15 real datasets showed favourable results, demonstrating the potential of the proposed method. Some experiments reported a boost in predictive accuracy of over 10 % consistently, evidencing the effectiveness of the iterative feature engineering achieved through the Replicator Dynamics procedure.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adeva, J.J.G., Beresi, U., Calvo, R.: Accuracy and diversity in ensembles of text categorisers. CLEI Electron. J. 9(1) (2005)
Bache, K., Lichman, M.: Uci machine learning repository (2013)
Bomze, I.M.: Lotka-volterra equation and replicator dynamics: new issues in classification. Biol. Cybern. 72(5), 447–453 (1995)
Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity creation methods: a survey and categorisation. Inf. Fusion 6(1), 5–20 (2005)
Cuzzocrea, A., Francis, S.L., Gaber, M.M.: An information-theoretic approach for setting the optimal number of decision trees in random forests. In: Systems, Man, and Cybernetics (SMC), 2013 IEEE International Conference on, pp. 1013–1019. IEEE (2013)
Fawagreh, K., Gaber, M.M., Elyan, E.: Diversified random forests using random subspaces. In: Intelligent Data Engineering and Automated Learning–IDEAL 2014, pp. 85–92. Springer (2014)
Fawagreh, K., Gaber, M.M., Elyan, E.: Random forests: from early developments to recent advancements. Syst. Sci. Control Eng: Open Access J. |bf 2(1), 602–609 (2014)
Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15, 3133–3181 (2014)
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)
Galstyan, A.: Continuous strategy replicator dynamics for multi-agent Q-learning. Auton. Agents Multi-agent Syst. 26(1), 37–53 (2013)
Hauert, C.: Replicator dynamics of reward & reputation in public goods games. J. Theor. Biol. 267(1), 22–28 (2010)
Hauert, C., De Monte, S., Hofbauer, J., Sigmund, K.: Replicator dynamics for optional public good games. J. Theor. Biol. 218(2), 187–194 (2002)
Hilbe, C.: Local replicator dynamics: a simple link between deterministic and stochastic models of evolutionary game theory. Bull. Math. Biol. 73(9), 2068–2087 (2011)
Hofbauer, J., Sigmund, K.: Evolutionary game dynamics. Bull. Am. Math. Soc. 40(4), 479–519 (2003)
Hutson, V., Schmitt, K.: Permanence and the dynamics of biological systems. Math. Biosci. 111(1), 1–71 (1992)
Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)
Lohmann, G., Bohn, S.: Using replicator dynamics for analyzing fMRI data of the human brain. IEEE Trans. Med. Imag. 21(5), 485–492 (2002)
Maclin, R., Opitz, D.: Popular ensemble methods: an empirical study. arXiv:1106.0257 (2011) (preprint)
Nowak, M.A., Sigmund, K.: Evolutionary dynamics of biological games. Science 303(5659), 793–799 (2004)
Olfati-Saber, R.: Evolutionary dynamics of behavior in social networks. In: Decision and Control, 2007 46th IEEE Conference on, pp. 4051–4056. IEEE (2007)
Polikar, R.: Ensemble based systems in decision making. Circuits Syst. Mag. IEEE 6(3), 21–45 (2006)
Roca, C.P., Cuesta, J.A., Sánchez, A.: Evolutionary game theory: temporal and spatial effects beyond replicator dynamics. Phys. Life Rev. 6(4), 208–249 (2009)
Rokach, L.: Ensemble-based classifiers. Artif. Intell. Rev. 33(1–2), 1–39 (2010)
Tang, E.K., Suganthan, P.N., Yao, X.: An analysis of diversity measures. Mach. Learn. 65(1), 247–271 (2006)
Taylor, P.D., Jonker, L.B.: Evolutionary stable strategies and game dynamics. Math. Biosci. 40(1), 145–156 (1978)
Wolpert, D.H.: Stacked generalization. Neural Netw. 5(2), 241–259 (1992)
Yan, W., Goebel, K.F.: Designing classifier ensembles with constrained performance requirements. In: Defense and Security, pp. 59–68. International Society for Optics and Photonics (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Fawgreh, K., Gaber, M.M., Elyan, E. (2015). A Replicator Dynamics Approach to Collective Feature Engineering in Random Forests. In: Bramer, M., Petridis, M. (eds) Research and Development in Intelligent Systems XXXII. SGAI 2015. Springer, Cham. https://doi.org/10.1007/978-3-319-25032-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-25032-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-25030-4
Online ISBN: 978-3-319-25032-8
eBook Packages: Computer ScienceComputer Science (R0)