skip to main content
10.1145/3440943.3444722acmconferencesArticle/Chapter ViewAbstractPublication PagesiceaConference Proceedingsconference-collections
research-article

An Effective Multi-Swarm Algorithm for Optimizing Hyperparameters of DNN

Published: 27 September 2021 Publication History

Abstract

Different hyperparameter settings for a deep neural network (DNN) algorithm will come up with different prediction results. One of the most important things is thus in selecting a set of suitable hyperparameters for a DNN so as to increase its accuracy. This can be regarded as a hyperparameter optimization problem for DNN or DNN-based algorithms. Compared with manual, grid search, or random search for parameter settings, metaheuristic algorithms are able to find better hyperparameters for DNNs. To improve the accuracy of a prediction model based on DNN, an improved version of multi-swarm particle swarm optimization (MSPSO) is presented in this paper. Moreover, data provided by Taipei Rapid Transit Corporation will be used to evaluate the performance of the proposed algorithm in predicting the number of passengers for the Taipei metro station. The simulation results show that the proposed algorithm can be used to find better hyperparameters for DNN. This means that the proposed algorithm can provide a more accurate result than other machine learning algorithms, DNN, and PSO with DNN in terms of the prediction accuracy.

References

[1]
Mohammdreza Afshin, Alireza Sadeghian, and Kaamran Raahemifar. 2007. On efficient tuning of LS-SVM hyper-parameters in short-term load forecasting: A comparative study. In Proceedings of the IEEE Power Engineering Society General Meeting. 1--6.
[2]
Yun Bai, Chuan Li, Zhenzhong Sun, and Haibin Chen. 2017. Deep neural network for manufacturing quality prediction. In Proceedings of the IEEE Prognostics and System Health Management Conference. 1--5.
[3]
James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13 (2012), 281--305.
[4]
Taiwan Central Weather Bureau. 2020. CWB Observation Data Inquire System. [Online] Available: https://e-service.cwb.gov.tw/HistoryDataQuery/index.jsp.
[5]
Dan CireşAn, Ueli Meier, Jonathan Masci, and Jürgen Schmidhuber. 2012. Multi-column deep neural network for traffic sign classification. Neural Networks 32 (2012), 333--338.
[6]
Taipei Rapid Transit Corporation. 2019. The Ridership Data of Taipei Mass Rapid Transit. [Online] Available: https://www.metro.taipei/cp.aspx?n=FF31501BEBDD0136.
[7]
Giovanni Lucca França da Silva, Thales Levi Azevedo Valente, Aristófanes Corrêa Silva, Anselmo Cardoso de Paiva, and Marcelo Gattass. 2018. Convolutional neural network-based PSO for lung nodule false positive reduction on CT images. Computer Methods and Programs in Biomedicine 162 (2018), 109--118.
[8]
Katherine Ellis, Suneeta Godbole, Simon Marshall, Gert Lanckriet, John Staudenmayer, and Jacqueline Kerr. 2014. Identifying active travel behaviors in challenging environments using GPS, accelerometers, and machine learning algorithms. Public Health 2, 36 (2014), 1--8.
[9]
Wisam Elmasry, Akhan Akbulut, and Abdul Halim Zaim. 2020. Evolving deep learning architectures for network intrusion detection using a double PSO meta-heuristic. Computer Networks 168 (2020), 107042.
[10]
Yasser Ganjisaffar, Thomas Debeauvais, Sara Javanmardi, Rich Caruana, and Cristina Videira Lopes. 2011. Distributed tuning of machine learning algorithms using MapReduce clusters. In Proceedings of the Workshop on Large Scale Data Mining: Theory and Applications. 1--8.
[11]
GitHub. 2020. Mean Absolute Percentage Error. [Online] Available: https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/keras/losses.py#L1231-L1265.
[12]
Erik Jenelius. 2019. Data-driven metro train crowding prediction based on realtime load data. IEEE Transactions on Intelligent Transportation Systems 21, 6 (2019), 2254--2265.
[13]
Nikolas Julio, Ricardo Giesen, and Pedro Lizana. 2016. Real-time prediction of bus travel speeds using traffic shockwaves and machine learning algorithms. Transportation Economics 59 (2016), 250--257.
[14]
Woo-Young Lee, Seung-Min Park, and Kwee-Bo Sim. 2018. Optimal hyperparameter tuning of convolutional neural networks based on the parameter-setting-free harmony search algorithm. Optik 172 (2018), 359--367.
[15]
J. Jing Liang and Ponnuthurai Nagaratnam Suganthan. 2005. Dynamic multi-swarm particle swarm optimizer. In Proceedings of the IEEE Swarm Intelligence Symposium. 124--129.
[16]
Lijuan Liu and Rung-Ching Chen. 2017. A novel passenger flow prediction model using deep learning methods. Transportation Research Part C: Emerging Technologies 84 (2017), 74--91.
[17]
Zhi Liu, Sanya Liu, Lin Liu, Jianwen Sun, Xian Peng, and Tai Wang. 2016. Senti-ment recognition of online course reviews using multi-swarm optimization-based selected features. Neurocomputing 185 (2016), 11--20.
[18]
Pablo Ribalta Lorenzo, Jakub Nalepa, Michal Kawulok, Luciano Sanchez Ramos, and José Ranilla Pastor. 2017. Particle swarm optimization for hyper-parameter selection in deep neural networks. In Proceedings of the Genetic and Evolutionary Computation Conference. 481--488.
[19]
Zuriani Mustaffa, Mohd Herwan Sulaiman, Yuhanis Yusof, and Syafiq Fauzi Kamarulzaman. 2017. A novel hybrid metaheuristic algorithm for short term load forecasting. International Journal of Simulation: Systems 17, 41 (2017), 1--6.
[20]
Noboru Nakafuri and Yoshio Miki. 2017. Prediction model of number of bus users based on influence of sensible temperature and discomfort index. In Proceedings of the IEEE International Conference on Intelligent Transportation Engineering. 155--158.
[21]
Buren Qi, Mengfei Wu, and Lin Zhang. 2017. A DNN-based object detection system on mobile cloud computing. In Proceedings of the IEEE International Symposium on Communications and Information Technologies. 1--6.
[22]
Chenye Qiu. 2020. A Multi-swarm Particle Swarm Optimization with an Adaptive Regrouping Strategy for Feature Selection. In Proceedings of the IEEE International Conference on Robotics and Automation Sciences. 130--136.
[23]
Basheer Qolomany, Majdi Maabreh, Ala Al-Fuqaha, Ajay Gupta, and Driss Ben-haddou. 2017. Parameters optimization of deep learning models using particle swarm optimization. In Proceedings of the IEEE International Wireless Communications and Mobile Computing Conference. 1285--1290.
[24]
Tatsuki Serizawa and Hamido Fujita. 2020. Optimization of convolutional neural network using the linearly decreasing weight particle swarm optimization. arXiv preprint arXiv:2001.05670, [Online] Available: https://https://arxiv.org/abs/2001.05670.
[25]
Foo Chong Soon, Hui Ying Khaw, Joon Huang Chuah, and Jeevan Kanesan. 2018. Hyper-parameters optimization of deep CNN architecture for vehicle logo recognition. IET Intelligent Transport Systems 12, 8 (2018), 939--946.
[26]
Dhruv Srivastava, Yasha Singh, and Anita Sahoo. 2019. Auto tuning of RNN hyper-parameters using cuckoo search algorithm. In Proceedings of the IEEE International Conference on Contemporary Computing. 1--5.
[27]
Wichai Treethidtaphat, Wasan Pattara-Atikom, and Sippakorn Khaimook. 2017. Bus arrival time prediction at any distance of bus route using deep neural network model. In Proceedings of the IEEE International Conference on Intelligent Transportation Systems. 988--992.
[28]
Chun-Wei Tsai, Chien-Hui Hsia, Shuang-Jie Yang, Shih-Jui Liu, and Zhi-Yan Fang. 2020. Optimizing hyperparameters of deep learning in predicting bus passengers based on simulated annealing. Applied Soft Computing 88 (2020), 1--9.
[29]
Eva Tuba, Lazar Mrkela, and Milan Tuba. 2016. Support vector machine parameter tuning using firefly algorithm. In Proceedings of the IEEE International Conference Radioelektronika. 413--418.
[30]
Hao Wang, GaoJun Liu, Jianyong Duan, and Lei Zhang. 2017. Detecting transportation modes using deep neural network. Transactions on Information and Systems 100, 5 (2017), 1132--1135.
[31]
Xuewen Xia, Ling Gui, and Zhi-Hui Zhan. 2018. A multi-swarm particle swarm optimization algorithm based on dynamical topology and purposeful detecting. Applied Soft Computing 67 (2018), 126--140.
[32]
Fei Ye. 2017. Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data. PloS One 12, 12 (2017), 1--36.

Cited By

View all
  • (2024)Reduced-order modelling for real-time physics-based variation simulation enhanced with adaptive sampling and optimized interpolationThe International Journal of Advanced Manufacturing Technology10.1007/s00170-024-13493-z132:7-8(3709-3734)Online publication date: 16-Apr-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ACM ICEA '20: Proceedings of the 2020 ACM International Conference on Intelligent Computing and its Emerging Applications
December 2020
219 pages
ISBN:9781450383042
DOI:10.1145/3440943
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Deep learning
  2. and hyperparameter optimization
  3. metaheuristic algorithm

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • MOST108-2221-E-005-021-MY3

Conference

ACM ICEA '20
Sponsor:

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)1
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Reduced-order modelling for real-time physics-based variation simulation enhanced with adaptive sampling and optimized interpolationThe International Journal of Advanced Manufacturing Technology10.1007/s00170-024-13493-z132:7-8(3709-3734)Online publication date: 16-Apr-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media