Skip to main content
SpringerLink
Log in

Application of a Stochastic Schemata Exploiter for Multi-Objective Hyper-parameter Optimization of Machine Learning

  • Article
  • Published:
The Review of Socionetwork Strategies Aims and scope Submit manuscript

Abstract

The Stochastic Schemata Exploiter (SSE), one of the Evolutionary Algorithms, is designed to find the optimal solution of a function. SSE extracts common schemata from individual sets with high fitness and generates individuals from the common schemata. For hyper-parameter optimization, the initialization method, the schema extraction method, and the new individual generation method, which are characteristic processes in SSE, are extended. In this paper, an SSE-based multi-objective optimization for AutoML is proposed. AutoML gives good results in terms of model accuracy. However, if only model accuracy is considered, the model may be too complex. Such complex models cannot always be allowed because of the long computation time. The proposed method maximizes the stacking model accuracy and minimizes the model complexity simultaneously. When compared with existing methods, SSE has interesting features such as fewer control parameters and faster convergence properties. The visualization method makes the optimization process transparent and helps users understand the process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data Availability

The datasets analyzed during the current study are available in the UCI Machine Learning repository, https://archive.ics.uci.edu/ml/datasets/abalone, https://archive.ics.uci.edu/ml/datasets/wine+quality.

References

  1. Makino, H., Feng, X., & Kita, E. (2020). Stochastic schemata exploiter-based optimization of convolutional neural network. In: IEEE International Conference on Systems, Man, and Cybernetics, pp. 4365–4371. https://doi.org/10.1109/SMC42975.2020.9283473

  2. Makino, H., & Kita, E. (2021). Stochastic schemata exploiter-based AutoML. In: Proceedings of the 2021 International Conference on Data Mining Workshops (ICDMW), pp. 238–245.

  3. Aizawa, A. N. (1994). Evolving SSE: A stochastic schemata exploiter. In: Proceedings of the First IEEE Conference on evolutionary computation. IEEE World Congress on Computational Intelligence, pp. 525–529. https://doi.org/10.1109/ICEC.1994.349895

  4. Aizawa, A. N. (1996). Evolving SSE: A new population-oriented search scheme based on schemata processing. Systems and Computers in Japan, 27(2), 41–52. https://doi.org/10.1002/scj.4690270204

    Article  Google Scholar 

  5. Maruyama, T., & Kita, E. (2007). Extension of stochastic schemata exploiter to real-valued problem. In: Proceedings of Computer Aided Optimum Design in Engineering X, pp. 45–53.

  6. Maruyama, T., & Kita, E. (2007). Investigation of real-valued stochastic schemata exploiter. Information Processing Society of Japan Transactions on Mathematical Modeling and its Applications, 48(SIG19(TOM19)), 10–22.

    Google Scholar 

  7. Kotthoff, L., Thornton, C., Hoos, H. H., Hutter, F., Leyton-Brown, K. (2019). Auto-WEKA: Automatic model selection and hyperparameter optimization in WEKA. In: Automated machine learning: methods, systems, challenges (pp. 81–95). Springer.

  8. Ledell, E., & Poirier, S. (2020). H2O AutoML: Scalable automatic machine learning. In: 7th ICML Workshop on Automated Machine Learning, pp. 1–16.

  9. Maziarz, K., Tan, M., Khorlin, A., Georgiev, M., & Gesmundo, A. (2018). Evolutionary-neural hybrid agents for architecture search. arXiv preprint arXiv:1811.09828.

  10. Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710.

  11. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

  12. Holland, J. H. (1992). Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. MIT Press.

    Book  Google Scholar 

  13. Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2), 182–197. https://doi.org/10.1109/4235.996017

    Article  Google Scholar 

  14. Zhang, Q., & Li, H. (2007). MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 11(6), 712–731. https://doi.org/10.1109/TEVC.2007.892759

    Article  Google Scholar 

  15. Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(1), 281–305.

    Google Scholar 

  16. Mantovani, R. G., Rossi, A. L. D., Vanschoren, J., Bischl, B., & De Carvalho, A. C. P. L. F. (2015). Effectiveness of random search in SVM hyper-parameter tuning. In: 2015 International Joint Conference on Neural Networks, pp. 1–8. https://doi.org/10.1109/IJCNN.2015.7280664

  17. Hansen, N., Muller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11(1), 1–18. https://doi.org/10.1162/106365603321828970

    Article  Google Scholar 

  18. Xia, Y., Liu, C., Li, Y. Y., & Liu, N. (2017). A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring. Expert Systems with Applications, 78, 225–241. https://doi.org/10.1016/j.eswa.2017.02.017

    Article  Google Scholar 

  19. Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. arXiv preprint arXiv:1206.2944.

  20. Feurer, M., & Hutter, F. (2019). Hyperparameter optimization. In: F. Hutter, L. Kotthoff, J. Vanschoren (eds.), Automated machine learning: Methods, systems, challenges, (pp. 3–33). Springer.

  21. Friedrichs, F., & Igel, C. (2005). Evolutionary tuning of multiple SVM parameters. Neurocomputing, 64, 107–117. https://doi.org/10.1016/j.neucom.2004.11.022

    Article  Google Scholar 

  22. Loshchilov, I., & Hutter, F. (2016). CMA-ES for hyperparameter optimization of deep neural networks. arXiv preprint arXiv:1604.07269.

  23. Zhang, L. M. (2019). A new compensatory genetic algorithm-based method for effective compressed multi-function convolutional neural network model selection with multi-objective optimization. arXiv preprint arXiv:1906.11912, 1–13.

  24. Laredo, D., Qin, Y., Schütze, O., & Sun, J.-Q. (2019). Automatic model selection for neural networks. arXiv preprint arXiv:1905.06010, 1–31.

  25. Loni, M., Majd, A., Loni, A., Daneshtalab, M., Sjodin, M., & Troubitsyna, E. (2018). Designing compact convolutional neural network for embedded stereo vision systems. https://doi.org/10.1109/MCSoC2018.2018.00049

  26. Vargas, D. V., & Kotyan, S. (2019). Evolving robust neural architectures to defend from adversarial attacks. arXiv preprint arXiv:1906.11667.

  27. Liu, Q., Li, X., Liu, H., & Guo, Z. (2020). Multi-objective metaheuristics for discrete optimization problems: A review of the state-of-the-art. Applied Soft Computing. https://doi.org/10.1016/j.asoc.2020.106382

    Article  Google Scholar 

  28. Lu, Z., Whalen, I., Boddeti, V., Dhebar, Y., Deb, K., Goodman, E., & Banzhaf, W. (2019). NSGA-Net: Neural architecture search using multi-objective genetic algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419–427.

  29. Hsu, C.-H., Chang, S.-H., Liang, J.-H., Chou, H.-P., Liu, C.-H., Chang, S.-C., Pan, J.-Y., Chen, Y.-T., Wei, W., & Juan, D.-C. (2018). MONAS: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332.

  30. Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K., & Miikkulainen, R. (2019). Evolutionary neural AutoML for deep learning. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 401–409.

  31. Stanley, K. O., & Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2), 99–127. https://doi.org/10.1162/106365602320169811

    Article  Google Scholar 

  32. Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., & Tan, K. C. (2021). A survey on evolutionary neural architecture search. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2021.3100554

    Article  Google Scholar 

  33. Dong, J. D., Cheng, A. C., Juan, D. C., Wei, W., & Sun, M. (2018). PPP-Net: Platform-aware progressive search for pareto-optimal neural architectures. In: International Conference on Learning Representations (ICLR) Workshop 2018, pp. 1–4.

  34. Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In: International Conference on learning and intelligent optimization, pp. 507–523.

  35. Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67. https://doi.org/10.1080/00401706.1970.10488634

    Article  Google Scholar 

  36. Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x

    Article  Google Scholar 

  37. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

    Google Scholar 

  38. Tibshirani, R. (2011). Regression shrinkage and selection via the Lasso: A retrospective. Journal of the Royal Statistical Society: Series B (statistical methodology), 73(3), 273–282. https://doi.org/10.1111/j.1467-9868.2011.00771.x

    Article  Google Scholar 

  39. Feng, X., Zhao, J., & Kita, E. (2019). Genetic algorithm based optimization of deep neural network ensemble for personal identification in pedestrians behaviors. In: Proceedings of the 2019 International Conference on Data Mining Workshops (ICDMW), pp. 318–325. https://doi.org/10.1109/ICDMW.2019.00054

  40. Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural architecture search. In: F. Hutter, L. Kotthoff, J. Vanschoren (eds.), Automated machine learning: Methods, systems, challenges, (pp. 63–77). Springer.

    Chapter  Google Scholar 

  41. Zoph, B., & Le, Q. V. (2017). Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.

  42. Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019). Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, 33, 4780–4789. https://doi.org/10.1609/aaai.v33i01.33014780

  43. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, pp. 1–14.

  44. Kim, J., Lee, J. K., & Lee, K. M. (2016). Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 1646–1654.

  45. Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017). A survey of deep neural network architectures and their applications. Neurocomputing, 234, 11–26. https://doi.org/10.1016/j.neucom.2016.12.038

    Article  Google Scholar 

  46. Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on machine learning, pp. 807–814.

  47. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1201/9780429469275-8

    Article  Google Scholar 

  48. Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on knowledge discovery and data mining, pp. 785–794.

  49. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T. Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30, 3149–3157.

    Google Scholar 

  50. Blank, J., & Deb, K. (2020). Pymoo: Multi-objective optimization in python. IEEE Access, 8, 89497–89509. https://doi.org/10.1109/ACCESS.2020.2990567

    Article  Google Scholar 

  51. Nash, W. J., Sellers, T. L., Talbot, S. R., Cawthorn, A. J., & Ford, W. B. (1994). The population biology of abalone (Haliotis species) in Tasmania. i. Blacklip abalone (H. rubra) from the north coast and islands of bass strait. Sea fisheries division, technical report 48.

  52. Cortez, P., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009). Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4), 547–553.

    Article  Google Scholar 

Download references

Funding

The authors did not receive support from any organization for the submitted work.

Author information

Authors and Affiliations

  1. Graduate School of Informatics, Nagoya University, Nagoya, Japan

    Hiroya Makino & Eisuke Kita

Authors
  1. Hiroya Makino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eisuke Kita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eisuke Kita.

Ethics declarations

Conflict of Interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Makino, H., Kita, E. Application of a Stochastic Schemata Exploiter for Multi-Objective Hyper-parameter Optimization of Machine Learning. Rev Socionetwork Strat 17, 179–213 (2023). https://doi.org/10.1007/s12626-023-00151-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12626-023-00151-1

Keywords

Search

Navigation