research-article

An Empirical Study of the Impact of Hyperparameter Tuning and Model Optimization on the Performance Properties of Deep Neural Networks

Authors:
Lizhi Liao

Concordia University, Montréal, Québec, Canada

Concordia University, Montréal, Québec, Canada

0000-0001-9920-5855
View Profile

,
Heng Li

Polytechnique Montréal, Montréal, Québec, Canada

Polytechnique Montréal, Montréal, Québec, Canada

0000-0001-5441-6763
View Profile

,
Weiyi Shang

Concordia University, Montréal, Québec, Canada

Concordia University, Montréal, Québec, Canada
View Profile

,
Lei Ma

University of Alberta, Edmonton, Alberta, Canada

University of Alberta, Edmonton, Alberta, Canada
View Profile

ACM Transactions on Software Engineering and Methodology Volume 31 Issue 3Article No.: 53pp 1–40https://doi.org/10.1145/3506695

Published:09 April 2022Publication History

ACM Transactions on Software Engineering and Methodology

Abstract

Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification, Resnet-50, CNN text classification, and LSTM sentiment classification, to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

REFERENCES

[1] 2019. Hyperparameters in Deep Learning. https://towardsdatascience.com/hyperparameters-in-deep-learning-927f7b2084dd. Last accessed 10/10/2020.Google Scholar
[2] 2020. Keras Code Examples. https://github.com/keras-team/keras-io/tree/master/examples. Last accessed 10/16/2020.Google Scholar
[3] 2020. Pruning in Keras Example. https://www.tensorflow.org/model_optimization/guide/pruning/pruning_with_keras.Google Scholar
[4] 2020. Tensorflow Model Optimization. https://www.tensorflow.org/model_optimization.Google Scholar
[5] Abadi Martín, Agarwal Ashish, Barham Paul, Brevdo Eugene, al. Yuan Yu, et2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/. Software available from tensorflow.org.Google Scholar
[6] Anwar Sajid, Hwang Kyuyeon, and Sung Wonyong. 2017. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC) 13, 3 (2017), 1–18.Google ScholarDigital Library
[7] Ashok Anubhav, Rhinehart Nicholas, Beainy Fares, and Kitani Kris M.. 2018. N2N learning: Network to network compression via policy gradient reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference Track Proceedings. OpenReview.net.Google Scholar
[8] Ayinde Babajide O. and Zurada Jacek M.. 2018. Building efficient ConvNets using redundant feature pruning. arXiv preprint arXiv:1802.07653 (2018).Google Scholar
[9] Sharfuddin Abdullah Aziz, Tihami Md. Nafis, and Islam Md. Saiful. 2018. A deep recurrent neural network with BiLSTM model for sentiment classification. In 2018 International Conference on Bangla Speech and Language Processing (ICBSLP). 1–4.Google ScholarCross Ref
[10] Baker Bowen, Gupta Otkrist, Raskar Ramesh, and Naik Nikhil. 2018. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Workshop Track Proceedings. OpenReview.net.Google Scholar
[11] Bergstra James and Bengio Yoshua. 2012. Random search for hyper-parameter optimization. The Journal of Machine Learning Research 13, 1 (2012), 281–305.Google ScholarDigital Library
[12] Bergstra James, Yamins Daniel, and Cox David D.. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16–21 June 2013(JMLR Workshop and Conference Proceedings, Vol. 28). JMLR.org, 115–123.Google Scholar
[13] Bergstra James S., Bardenet Rémi, Bengio Yoshua, and Kégl Balázs. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems. 2546–2554.Google Scholar
[14] Bianco Simone, Cadene Remi, Celona Luigi, and Napoletano Paolo. 2018. Benchmark analysis of representative deep neural network architectures. IEEE Access 6 (2018), 64270–64277.Google ScholarCross Ref
[15] Bisong Ekaba. 2019. Google AutoML: Cloud vision. In Building Machine Learning and Deep Learning Models on Google Cloud Platform. Springer, 581–598.Google ScholarCross Ref
[16] Bonferroni Carlo. 1936. Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze 8 (1936), 3–62.Google Scholar
[17] Cai Han, Zhu Ligeng, and Han Song. 2018. ProxylessNAS: Direct neural architecture search on target task and hardware. CoRR abs/1812.00332 (2018).Google Scholar
[18] Cai Jingjing, Li Jianping, Li Wei, and Wang Ji. 2018. Deeplearning model used in text classification. In 2018 15th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP). 123–126.Google ScholarCross Ref
[19] Canziani Alfredo, Paszke Adam, and Culurciello Eugenio. 2016. An analysis of deep neural network models for practical applications. arXiv preprint arXiv:1605.07678 (2016).Google Scholar
[20] Chen Tse-Hsun, Shang Weiyi, Hassan Ahmed E., Nasser Mohamed, and Flora Parminder. 2016. CacheOptimizer: Helping developers configure caching frameworks for hibernate-based database-centric web applications. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 666–677.Google ScholarDigital Library
[21] Cheng Yu, Wang Duo, Zhou Pan, and Zhang Tao. 2017. A survey of model compression and acceleration for deep neural networks. CoRR abs/1710.09282 (2017).Google Scholar
[22] Choi Yoojin, El-Khamy Mostafa, and Lee Jungwon. 2020. Universal deep neural network compression. IEEE Journal of Selected Topics in Signal Processing (2020).Google ScholarCross Ref
[23] Chollet François et al. 2015. Keras. https://keras.io.Google Scholar
[24] Choukroun Yoni, Kravchik Eli, Yang Fan, and Kisilev Pavel. 2019. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27–28, 2019. IEEE, 3009–3018.Google ScholarCross Ref
[25] Cliff Norman. 2014. Ordinal Methods for Behavioral Data Analysis. Psychology Press.Google ScholarCross Ref
[26] Couto Mauricio Farias, Peternelli Luiz Alexandre, and Barbosa Márcio Henrique Pereira. 2013. Classification of the coefficients of variation for sugarcane crops. Ciência Rural 43 (2013), 957–961.Google ScholarCross Ref
[27] Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2–7, 2019, Volume 1 (Long and Short Papers), Burstein Jill, Doran Christy, and Solorio Thamar (Eds.). Association for Computational Linguistics, 4171–4186.Google Scholar
[28] Emmerich Michael T. M., Deutz André H., and Klinkenberg Jan Willem. 2011. Hypervolume-based expected improvement: Monotonicity properties and exact computation. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2011, New Orleans, LA, USA, 5–8 June, 2011. IEEE, 2147–2154.Google ScholarCross Ref
[29] Faes Livia, Wagner Siegfried K., Fu Dun Jack, Liu Xiaoxuan, Korot Edward, Ledsam Joseph R., Back Trevor, Chopra Reena, Pontikos Nikolas, Kern Christoph, et al. 2019. Automated deep learning design for medical image classification by health-care professionals with no coding experience: A feasibility study. The Lancet Digital Health 1, 5 (2019), e232–e242.Google ScholarCross Ref
[30] Falkner Stefan, Klein Aaron, and Hutter Frank. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018(Proceedings of Machine Learning Research, Vol. 80), Dy Jennifer G. and Krause Andreas (Eds.). PMLR, 1436–1445.Google Scholar
[31] Fawcett Chris and Hoos Holger H.. 2016. Analysing differences between algorithm configurations through ablation. J. Heuristics 22, 4 (2016), 431–458.Google ScholarDigital Library
[32] Feurer Matthias, Eggensperger Katharina, Falkner Stefan, Lindauer Marius, and Hutter Frank. 2020. Auto-Sklearn 2.0. arXiv:2007.04074 [cs.LG] (2020).Google Scholar
[33] Gao Yanjie, Liu Yu, Zhang Hongyu, Li Zhengxian, Zhu Yonghao, Lin Haoxiang, and Yang Mao. 2020. Estimating GPU Memory Consumption of Deep Learning Models. Technical Report MSR-TR-2020-20. Microsoft.Google ScholarDigital Library
[34] Gramacy Robert B., Taddy Matt, and Wild Stefan M.. 2013. Variable selection and sensitivity analysis using dynamic trees, with an application to computer code performance tuning. The Annals of Applied Statistics (2013), 51–80.Google Scholar
[35] Guo Jia and Potkonjak Miodrag. 2017. Pruning ConvNets online for efficient specialist models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 113–120.Google ScholarCross Ref
[36] Hale Jeff. 2019. Which Deep Learning Framework is Growing Fastest? https://towardsdatascience.com/which-deep-learning-framework-is-growing-fastest-3f77f14aa318.Google Scholar
[37] Han Song, Mao Huizi, and Dally William J.. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149 (2015).Google Scholar
[38] Han Song, Pool Jeff, Tran John, and Dally William J.. 2015. Learning both weights and connections for efficient neural networks. CoRR abs/1506.02626 (2015).Google Scholar
[39] Han Seong-Hyeon and Lee Kwang-Yeob. 2017. Implementation of image classification CNN using multi thread GPU. In 2017 International SoC Design Conference (ISOCC). 296–297.Google ScholarCross Ref
[40] He Yang, Kang Guoliang, Dong Xuanyi, Fu Yanwei, and Yang Yi. 2018. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018).Google Scholar
[41] Hsu Chi-Hung, Chang Shu-Huan, Juan Da-Cheng, Pan Jia-Yu, Chen Yu-Ting, Wei Wei, and Chang Shih-Chieh. 2018. MONAS: Multi-objective neural architecture search using reinforcement learning. CoRR abs/1806.10332 (2018).Google Scholar
[42] Hutter Frank, Hoos Holger H., and Leyton-Brown Kevin. 2013. Identifying key algorithm parameters and instance features using forward selection. In Learning and Intelligent Optimization - 7th International Conference, LION 7, Catania, Italy, January 7–11, 2013, Revised Selected Papers(Lecture Notes in Computer Science, Vol. 7997), Nicosia Giuseppe and Pardalos Panos M. (Eds.). Springer, 364–381.Google ScholarDigital Library
[43] Hutter Frank, Hoos Holger H., and Leyton-Brown Kevin. 2014. An efficient approach for assessing hyperparameter importance. In Proceedings of the 31st International Conference on Machine Learning, ICML 2014, Beijing, China, 21–26 June 2014(JMLR Workshop and Conference Proceedings, Vol. 32). JMLR.org, 754–762.Google Scholar
[44] Jiang Jing, Han Fei, Ling Qinghua, Wang Jie, Li Tiange, and Han Henry. 2020. Efficient network architecture search via multiobjective particle swarm optimization based on decomposition. Neural Networks 123 (2020), 305–316.Google ScholarDigital Library
[45] Jin Haifeng, Song Qingquan, and Hu Xia. 2019. Auto-Keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4–8, 2019, Teredesai Ankur, Kumar Vipin, Li Ying, Rosales Rómer, Terzi Evimaria, and Karypis George (Eds.). ACM, 1946–1956.Google ScholarDigital Library
[46] Prost Julie. 2020. Hands on Hyperparameter Tuning with Keras Tuner. https://www.sicara.ai/blog/hyperparameter-tuning-keras-tuner.Google Scholar
[47] Kim Yong-Deok, Park Eunhyeok, Yoo Sungjoo, Choi Taelim, Yang Lu, and Shin Dongjun. 2015. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 (2015).Google Scholar
[48] Knowles Joshua D.. 2006. ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans. Evol. Comput. 10, 1 (2006), 50–66.Google ScholarDigital Library
[49] Kotthoff Lars, Thornton Chris, Hoos Holger H., Hutter Frank, and Leyton-Brown Kevin. 2017. Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA. J. Mach. Learn. Res. 18 (2017), 25:1–25:5.Google Scholar
[50] Krizhevsky Alex, Hinton Geoffrey, et al. 2009. Learning multiple layers of features from tiny images. (2009).Google Scholar
[51] Krizhevsky Alex, Sutskever Ilya, and Hinton Geoffrey E.. 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90.Google ScholarDigital Library
[52] LeCun Yann, Cortes Corinna, and Burges CJ. 2010. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2 (2010).Google Scholar
[53] Li Da, Chen Xinbo, Becchi Michela, and Zong Ziliang. 2016. Evaluating the energy efficiency of deep convolutional neural networks on CPUs and GPUs. In 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom)(BDCloud-SocialCom-SustainCom). IEEE, 477–484.Google ScholarCross Ref
[54] Li Hao, Kadav Asim, Durdanovic Igor, Samet Hanan, and Graf Hans Peter. 2016. Pruning filters for efficient ConvNets. arXiv preprint arXiv:1608.08710 (2016).Google Scholar
[55] Liao Lizhi, Chen Jinfu, Li Heng, Zeng Yi, Shang Weiyi, Guo Jianmei, Sporea Catalin, Toma Andrei, and Sajedi Sarah. 2020. Using black-box performance models to detect performance regressions under varying workloads: An empirical study. Empirical Software Engineering 25, 5 (2020), 4130–4160.Google ScholarDigital Library
[56] Lindauer Marius and Hutter Frank. 2019. Best practices for scientific research on neural architecture search. CoRR abs/1909.02453 (2019).Google Scholar
[57] Liu Jia, Gong Maoguo, Miao Qiguang, Wang Xiaogang, and Li Hao. 2018. Structure learning for deep neural networks based on multiobjective optimization. IEEE Trans. Neural Networks Learn. Syst. 29, 6 (2018), 2450–2463.Google ScholarCross Ref
[58] Loni Mohammad, Sinaei Sima, Zoljodi Ali, Daneshtalab Masoud, and Sjödin Mikael. 2020. DeepMaker: A multi-objective optimization framework for deep neural networks in embedded systems. Microprocess. Microsystems 73 (2020), 102989.Google ScholarDigital Library
[59] Luan Yuandong and Lin Shaofu. 2019. Research on text classification based on CNN and LSTM. In 2019 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA). 352–355.Google ScholarCross Ref
[60] Ma Yuexin, Zhu Xinge, Zhang Sibo, Yang Ruigang, Wang Wenping, and Manocha Dinesh. 2019. TrafficPredict: Trajectory prediction for heterogeneous traffic-agents. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 6120–6127.Google ScholarDigital Library
[61] Maas Andrew L., Daly Raymond E., Pham Peter T., Huang Dan, Ng Andrew Y., and Potts Christopher. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, 142–150.Google ScholarDigital Library
[62] Mehrotra Abhinav, Ramos Alberto Gil C. P., Bhattacharya Sourav, Dudziak Lukasz, Vipperla Ravichander, Chau Thomas C. P., Abdelfattah Mohamed S., Ishtiaq Samin, and Lane Nicholas Donald. 2021. NAS-Bench-ASR: Reproducible neural architecture search for speech recognition. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3–7, 2021. OpenReview.net.Google Scholar
[63] Mnih Volodymyr, Kavukcuoglu Koray, Silver David, Graves Alex, Antonoglou Ioannis, Wierstra Daan, and Riedmiller Martin. 2013. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).Google Scholar
[64] Molchanov Pavlo, Tyree Stephen, Karras Tero, Aila Timo, and Kautz Jan. 2016. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 (2016).Google Scholar
[65] Nachar Nadim et al. 2008. The Mann-Whitney U: A test for assessing whether two independent samples come from the same distribution. Tutorials in Quantitative Methods for Psychology 4, 1 (2008), 13–20.Google ScholarCross Ref
[66] O’Malley Tom, Bursztein Elie, Long James, Chollet François, Jin Haifeng, Invernizzi Luca, et al. 2019. Keras Tuner. https://github.com/keras-team/keras-tuner.Google Scholar
[67] Paria Biswajit, Kandasamy Kirthevasan, and Póczos Barnabás. 2019. A flexible framework for multi-objective Bayesian optimization using random scalarizations. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22–25, 2019(Proceedings of Machine Learning Research, Vol. 115), Globerson Amir and Silva Ricardo (Eds.). AUAI Press, 766–776.Google Scholar
[68] Pei Kexin, Cao Yinzhi, Yang Junfeng, and Jana Suman. 2017. DeepXplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles. 1–18.Google ScholarDigital Library
[69] Qaisar Saeed Mian. 2020. Sentiment analysis of IMDb movie reviews using long short-term memory. In 2020 2nd International Conference on Computer and Information Sciences (ICCIS). 1–4.Google ScholarCross Ref
[70] Rapaport Elad, Shriki Oren, and Puzis Rami. 2019. EEGNAS: Neural architecture search for electroencephalography data analysis and decoding. In Human Brain and Artificial Intelligence - First International Workshop, HBAI 2019, Held in Conjunction with IJCAI 2019, Macao, China, August 12, 2019, Revised Selected Papers(Communications in Computer and Information Science, Vol. 1072), Zeng An, Pan Dan, Hao Tianyong, Zhang Daoqiang, Shi Yiyu, and Song Xiaowei (Eds.). Springer, 3–20.Google Scholar
[71] Reimers Nils and Gurevych Iryna. 2017. Optimal hyperparameters for deep LSTM-networks for sequence labeling tasks. arXiv preprint arXiv:1707.06799 (2017).Google Scholar
[72] Ren Pengzhen, Xiao Yun, Chang Xiaojun, Huang Po-Yao, Li Zhihui, Chen Xiaojiang, and Wang Xin. 2020. A comprehensive survey of neural architecture search: Challenges and solutions. CoRR abs/2006.02903 (2020).Google Scholar
[73] Saranya C. and Manikandan G.. 2013. A study on normalization techniques for privacy preserving data mining. International Journal of Engineering and Technology (IJET) 5, 3 (2013), 2701–2704.Google Scholar
[74] Scott Macfarlane T. U., Scott Tannath J., and Kelly Vincent G.. 2016. The validity and reliability of global positioning systems in team sport: A brief review. The Journal of Strength & Conditioning Research 30, 5 (2016), 1470–1490.Google ScholarCross Ref
[75] Snoek Jasper, Larochelle Hugo, and Adams Ryan P.. 2012. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems. 2951–2959.Google ScholarDigital Library
[76] Srivastava Nitish, Hinton Geoffrey, Krizhevsky Alex, Sutskever Ilya, and Salakhutdinov Ruslan. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 56 (2014), 1929–1958.Google ScholarDigital Library
[77] Stapleton James H.. 2007. Models for Probability and Statistical Inference: Theory and Applications, Vol. 652, John Wiley & Sons.Google ScholarCross Ref
[78] Stock Pierre, Fan Angela, Graham Benjamin, Grave Edouard, Gribonval Rémi, Jégou Hervé, and Joulin Armand. 2021. Training with quantization noise for extreme model compression. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3–7, 2021.Google Scholar
[79] Sze Vivienne, Chen Yu-Hsin, Yang Tien-Ju, and Emer Joel S.. 2017. Efficient processing of deep neural networks: A tutorial and survey. Proc. IEEE 105, 12 (2017), 2295–2329.Google ScholarCross Ref
[80] Talbi El-Ghazali. 2019. A unified view of parallel multi-objective evolutionary algorithms. J. Parallel Distributed Comput. 133 (2019), 349–358.Google ScholarCross Ref
[81] Tan Mingxing, Chen Bo, Pang Ruoming, Vasudevan Vijay, Sandler Mark, Howard Andrew, and Le Quoc V.. 2019. MnasNet: Platform-aware neural architecture search for mobile. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16–20, 2019. Computer Vision Foundation/IEEE, 2820–2828.Google ScholarCross Ref
[82] Thornton Chris, Hutter Frank, Hoos Holger H., and Leyton-Brown Kevin. 2013. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11–14, 2013, Dhillon Inderjit S., Koren Yehuda, Ghani Rayid, Senator Ted E., Bradley Paul, Parekh Rajesh, He Jingrui, Grossman Robert L., and Uthurusamy Ramasamy (Eds.). ACM, 847–855.Google ScholarDigital Library
[83] Tung Frederick and Mori Greg. 2018. Clip-Q: Deep network compression learning by in-parallel pruning-quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7873–7882.Google ScholarCross Ref
[84] Vo An Tien, Tran Hai Son, and Le Thai Hoang. 2017. Advertisement image classification using convolutional neural network. In 2017 9th International Conference on Knowledge and Systems Engineering (KSE). 197–202.Google ScholarCross Ref
[85] Wang Yequan, Huang Minlie, Zhu Xiaoyan, and Zhao Li. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 606–615.Google ScholarCross Ref
[86] Wong Jenna, Manderson Travis, Abrahamowicz Michal, Buckeridge David L., and Tamblyn Robyn. 2019. Can hyperparameter tuning improve the performance of a super learner?: A case study. Epidemiology (Cambridge, Mass.) 30, 4 (2019), 521.Google ScholarCross Ref
[87] Yao Shuochao, Zhao Yiran, Shao Huajie, Liu ShengZhong, Liu Dongxin, Su Lu, and Abdelzaher Tarek. 2018. FastDeepIoT: Towards understanding and optimizing neural network execution time on mobile and embedded devices. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 278–291.Google ScholarDigital Library
[88] Yao Zhewei, Dong Zhen, Zheng Zhangcheng, Gholami Amir, Yu Jiali, Tan Eric, Wang Leyuan, Huang Qijing, Wang Yida, Mahoney Michael W., and Keutzer Kurt. 2021. HAWQ-V3: Dyadic neural network quantization. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18–24 July 2021, Virtual Event(Proceedings of Machine Learning Research, Vol. 139). PMLR, 11875–11886.Google Scholar
[89] Ye Shaokai, Feng Xiaoyu, Zhang Tianyun, Ma Xiaolong, Lin Sheng, Li Zhengang, Xu Kaidi, Wen Wujie, Liu Sijia, Tang Jian, et al. 2019. Progressive DNN compression: A key to achieve ultra-high weight pruning and quantization rates using ADMM. arXiv preprint arXiv:1903.09769 (2019).Google Scholar
[90] Yoon Min, Yun Yeboon, and Nakayama Hirotaka. 2009. Sequential Approximate Multiobjective Optimization Using Computational Intelligence. Springer.Google ScholarCross Ref
[91] Zahisham Zharfan, Lee Chin Poo, and Lim Kian Ming. 2020. Food recognition with ResNet-50. In 2020 IEEE 2nd International Conference on Artificial Intelligence in Engineering and Technology (IICAIET). 1–5.Google ScholarCross Ref
[92] Zelenski Julie, Huffman K., Schwarz K., and Stepp Marty. 2012. Huffman encoding and data compression.Google Scholar
[93] Zhang Qingfu and Li Hui. 2007. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Trans. Evol. Comput. 11, 6 (2007), 712–731.Google ScholarDigital Library
[94] Zhang Richard and Golovin Daniel. 2020. Random hypervolume scalarizations for provable multi-objective black box optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event(Proceedings of Machine Learning Research, Vol. 119). PMLR, 11096–11105.Google Scholar

Index Terms

An Empirical Study of the Impact of Hyperparameter Tuning and Model Optimization on the Performance Properties of Deep Neural Networks
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Hyperparameter tuning of convolutional neural networks for building construction image classification
Abstract
Deep Learning models have important applications in image processing. However, one of the challenges in this field is the definition of hyperparameters. Thus, the objective of this work is to propose a rigorous methodology for hyperparameter ...
Read More
Hyperparameter optimization of pre-trained convolutional neural networks using adolescent identity search algorithm
Abstract
Convolutional neural networks (CNNs) are widely used deep learning (DL) models for image classification. The selected hyperparameters for training convolutional neural network (CNN) models have a significant effect on the performance. Therefore, ...
Read More
Hyperparameter optimization of deep neural network using univariate dynamic encoding algorithm for searches
Abstract
This paper proposes a method to find the hyperparameter tuning for a deep neural network by using a univariate dynamic encoding algorithm for searches. Optimizing hyperparameters for such a neural network is difficult because the ...
Highlights
- An optimization method for hyper-parameters for a deep neural network.
- ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Transactions on Software Engineering and Methodology Volume 31, Issue 3
July 2022
912 pages
ISSN:1049-331X
EISSN:1557-7392
DOI:10.1145/3514181
Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 9 April 2022
- Online AM: 31 January 2022
- Accepted: 1 December 2021
- Revised: 1 October 2021
- Received: 1 December 2020
Published in tosem Volume 31, Issue 3

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Deep neural network
hyperparameter tuning
DNN model optimization
DNN model performance
Qualifiers
- research-article
- Refereed
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 10
  Total Citations
  View Citations
- 1,927
  Total Downloads
- Downloads (Last 12 months)883
- Downloads (Last 6 weeks)97
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

HTML Format

View this article in HTML Format .

View HTML Format

An Empirical Study of the Impact of Hyperparameter Tuning and Model Optimization on the Performance Properties of Deep Neural Networks

ACM Transactions on Software Engineering and Methodology

Abstract

REFERENCES

Cited By

Index Terms

Recommendations

Hyperparameter tuning of convolutional neural networks for building construction image classification

Hyperparameter optimization of pre-trained convolutional neural networks using adolescent identity search algorithm

Hyperparameter optimization of deep neural network using univariate dynamic encoding algorithm for searches