Abstract
Rapid development of deep ANNs made the number of hyperparameters constantly grow. As a sequence various aspects of ANNs, such as inference time, efficient resources utilization, losses and even training time were strongly influenced. In general methods of hyperparameters tuning are used for adapting well-known ANN models to new tasks or to tasks in similar areas without pre-training or for synthesis of new particular architectures for particular problems. In this article we compare different types of hyperparameters tuning like CoDeepNEAT, Naive Evolution, Tree-Parzen estimation, structured annealing with MorphNet post-tuning. We apply these methods to particular network architectures for image processing and HRM signal estimation. The process of adaptation this technology to big networks requires a lot of computational resources, so it’s necessary to use parallel implementations. It can be done by utilizing HPC with hybrid computational nodes. Also we propose new type of tool based on Microsoft NNI. It is used for tuners comparison, convergence analysis, and runs different tuners in parallel mode on cluster nodes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bergstra, J.S., et al.: Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems (2011). https://doi.org/10.5555/2986459.2986743
Amado, N., Gama, J., Silva, F.: Parallel implementation of decision tree learning algorithms. In: Brazdil, P., Jorge, A. (eds.) EPIA 2001. LNCS (LNAI), vol. 2258, pp. 6–13. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45329-6_4
Devlin, J., et al.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Real, E., et al.: Large-scale evolution of image classifiers. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70. JMLR. org (2017). https://doi.org/10.5555/3305890.3305981
Neural Network Intelligence, April 2020. https://github.com/microsoft/nni
Miikkulainen, R., Liang, J., Meyerson, E., et al.: Evolving deep neural networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312. Elseiver (2019). https://doi.org/10.1016/B978-0-12-815480-9.00015-3
Gordon, A., et al.: MorphNet: fast & simple resource-constrained structure learning of deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/CVPR.2018.00171
CoDeepNEAT implementation base, April 2020. https://github.com/sash-a/CoDeepNEAT
Polus cluster specifications, April 2020. http://hpc.cs.msu.su/polus
Voevodin, V.l., et al.: Supercomputer Lomonosov-2: large scale, deep monitoring and fine analytics for the user community. Supercomput. Front. Innov 6(2), 4–11 (2019). https://doi.org/10.14529/jsfi190201
CTE cluster configuration, April 2020. https://www.bsc.es/user-support/power.php#ToC-systemoverview
Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings, Edinburgh, UK, pp. 958–963 (2003). https://doi.org/10.1109/ICDAR.2003.1227801
Bobbia, S., Macwan, R., Benezeth, Y., Mansouri, A., Dubois, J.: Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recogn. Lett. (2017). https://doi.org/10.1016/j.patrec.2017.10.017
Khamitov, K., Popova, N.: Research of measurements of ANN hyperparameters tuning on HPC clusters with POWER8. In: Russian Supercomputing Days 2019: Proceedings of International Conference, 23–24 September 2019, Moscow, pp. 176–184 (2019)
Poon, A., Narayanan, D.: MorphNet: Towards Faster and Smaller Neural Networks. Google AI Perception (2019)
Szegedy, C., et al.: Going deeper with convolution (2014)
Sergeev, A., Balso, M.D.: Meet Horovod: Uber’s Open Source Distributed Deep Learning Framework for TensorFlow. Uber Engineering Blog (2017). https://eng.uber.com/horovod
Patarasuk, P., Yuan, X.: Bandwidth optimal all-reduce algorithms for clusters of workstations (2009). https://doi.org/10.1016/j.jpdc.2008.09.002
Acknowledgements
Reported study was funded by RFBR according to research project №20-07-01053. This research was carried out using shared research facilities equipment of HPC computing resources at Lomonosov Moscow State University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Khamitov, K., Popova, N., Konkov, Y., Castillo, T. (2020). Tuning ANNs Hyperparameters and Neural Architecture Search Using HPC. In: Voevodin, V., Sobolev, S. (eds) Supercomputing. RuSCDays 2020. Communications in Computer and Information Science, vol 1331. Springer, Cham. https://doi.org/10.1007/978-3-030-64616-5_46
Download citation
DOI: https://doi.org/10.1007/978-3-030-64616-5_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64615-8
Online ISBN: 978-3-030-64616-5
eBook Packages: Computer ScienceComputer Science (R0)