Abstract
Ensemble classifiers are strong and robust methods for classification and regression tasks. Considering the balance between runtime and classifier accuracy the learning problem becomes a multi-objective optimization problem. In this work, we propose an evolutionary multi-objective algorithm based on non-dominated sorting that balances runtime and accuracy properties of nearest neighbor classifier ensembles and decision tree ensembles. We identify relevant ensemble parameters with a significant impact on the accuracy and runtime. In the experimental part of this paper, we analyze the behavior on typical classification benchmark problems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This observation could be confirmed in our experiments and is difficult to observe in Fig. 1(a) due to the limited plot resolution.
References
Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. In: Vitányi, P. (ed.) EuroCOLT 1995. LNCS, vol. 904, pp. 23–37. Springer, Heidelberg (1995)
Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. CRC Press, Boca Raton (1984)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 182–197 (2002)
Huang, C.L., Wang, C.J.: A ga-based feature selection and parameters optimizationfor support vector machines. Expert Syst. Appl. 31, 231–240 (2006)
Hu, Q.-H., Yu, D.-R., Wang, M.-Y.: Constructing rough decision forests. In: Ślzak, D., Yao, J.T., Peters, J.F., Ziarko, W.P., Hu, X. (eds.) RSFDGrC 2005. LNCS (LNAI), vol. 3642, pp. 147–156. Springer, Heidelberg (2005)
Ishibuchi, H., Yamamoto, T.: Evolutionary multiobjective optimization for generating an ensemble of fuzzy rule-based classifiers. In: Cantú-Paz, E., et al. (eds.) GECCO 2003. LNCS, vol. 2723, pp. 1077–1088. Springer, Heidelberg (2003)
Abbass, H.A.: Pareto neuro-evolution: Constructing ensemble of neural networks using multi-objective optimization. In: Proceedings of the IEEE Congress on Evolutionary Computation. vol. 3, pp. 2074–2080. IEEE (2003)
Opitz, D.W., Shavlik, J.W.: Actively searching for an effective neural network ensemble. Connect. Sci. 8, 337–354 (1996)
Tumer, K., Oza, N.C.: Input decimated ensembles. Pattern Anal. Appl. 6, 65–77 (2003)
Oei, C.K., Goldberg, D.E., Chang, S.J.: Tournament selection, niching, and the preservation of diversity. Urbana 51, 61801 (1991)
Deb, K., Agrawal, R.B.: Simulated binary crossover for continuous search space. Complex Syst. 9, 1–34 (1994)
Deb, L., et al.: Multi-objective Optimization Using Evolutionary Algorithms, vol. 2012. Wiley, Chichester (2001)
Schwefel, H.P.: Adaptive mechanismen in der biologischen evolution und ihr einfluß auf die evolutionsgeschwindigkeit. Interner Bericht der Arbeitsgruppe Bionik und Evolutionstechnik am Institut für Mess-und Regelungstechnik 215 (1974)
Joines, J.A., Houck, C.R.: On the use of non-stationary penalty functions to solve nonlinear constrained optimization problems with GA’s. In: First IEEE Conference on Evolutionary Computation, pp. 579–584. IEEE (1994)
Fumera, G., Roli, F., Serrau, A.: A theoretical analysis of bagging as a linear combination of classifiers. IEEE Trans. Pattern Anal. Mach. Intel. 30, 1293–1299 (2008)
Parmanto, B., Munro, P.W., Doyle, H.R.: Improving committee diagnosis with resampling techniques. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems, pp. 882–888. MIT Press, Cambridge (1995)
Raviv, Y., Intrator, N.: Bootstrapping with noise: an effective regularization technique. Connet. Sci. 8, 355–372 (1996)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Acknowledgements
We thank the ministry of science and culture of Lower Saxony for supporting us with the graduate schools Safe Automation of Maritime Systems (SAMS) and System Integration of Renewable Energies (SEE).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A Benchmark Problems
A Benchmark Problems
The experiments in this paper are based on the following benchmark problems. MakeClass is a classification data set generated with the scikit-learn [20] method make_classification. MakeClass consists of \(N=13 500\) patterns with three classes and \(d=20\) features, of which four are informative. In 0.002 % of the patterns, labels have been changed. Waveform2 by Breiman et al. [3] consists of \(N=5000\) patterns with \(d=59\) features. The data set is an artificial one consisting of waveform data with three waveforms BreastCancer from the UCI repository comprises \(N=699\) patterns with two labels (cancer and no cancer) and \(d=9\) features. ImageSegmentation from the UCI repository consists of \(N=2310\) with seven classes corresponding to seven images. Each pattern consists of an image region of size \(3 \times 3\) pixels.
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Oehmcke, S., Heinermann, J., Kramer, O. (2015). Analysis of Diversity Methods for Evolutionary Multi-objective Ensemble Classifiers. In: Mora, A., Squillero, G. (eds) Applications of Evolutionary Computation. EvoApplications 2015. Lecture Notes in Computer Science(), vol 9028. Springer, Cham. https://doi.org/10.1007/978-3-319-16549-3_46
Download citation
DOI: https://doi.org/10.1007/978-3-319-16549-3_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16548-6
Online ISBN: 978-3-319-16549-3
eBook Packages: Computer ScienceComputer Science (R0)