Georgia Channing
ABSTRACT
Neural architecture search (NAS), which automates the design of neural network (NN) architectures for scientific datasets, requires significant computational resources and time — often on the order of days or weeks of GPU hours and training time. We design the Analytics for Neural Network (A4NN) workflow, a composable workflow that significantly reduces the time and resources required to design accurate and efficient NN architectures. We introduce a parametric fitness prediction strategy and distribute training across multiple accelerators to decrease the aggregated NN training time. A4NN rigorously record neural architecture histories, model states, and metadata to reproduce the search for near-optimal NNs. We demonstrate A4NN’s ability to reduce training time and resource consumption on a dataset generated by an X-ray Free Electron Laser (XFEL) experiment simulation. When deploying A4NN, we decrease training time by up to 37% and epochs required by up to 38%.
Supplemental Material
Available for Download
- Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. 2017. Accelerating Neural Architecture Search Using Performance Prediction. In Proceedings of the NIPS Workshop on Meta-Learning. arxiv:1705.10823 [cs.LG]Google Scholar
- Prasanna Balaprakash, Romain Egele, Misha Salim, Stefan Wild, Venkatram Vishwanath, Fangfang Xia, Tom Brettin, and Rick Stevens. 2019. Scalable reinforcement-learning-based neural architecture search for cancer deep learning research. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis.Google ScholarDigital Library
- Han Cai, Ligeng Zhu, and Song Han. 2018. Proxylessnas: Direct Neural Architecture Search on Target Task and Hardware. arXiv preprint arXiv:1812.00332 (2018).Google Scholar
- Ryan Chard, Zhuozhao Li, Kyle Chard, Logan Ward, Yadu Babuji, Anna Woodard, Steven Tuecke, Ben Blaiszik, Michael Franklin, and Ian Foster. 2019. DLHub: Model and Data Serving for Science. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS). 283–292.Google ScholarCross Ref
- Shanyu Chen, Zhipeng He, Xinyin Han, Xiaoyu He, Ruilin Li, Haidong Zhu, Dan Zhao, Chuangchuang Dai, Yu Zhang, Zhonghua Lu, Xuebin Chi, and Beifang Niu. 2019. How Big Data and High-performance Computing Drive Brain Science. Genomics, Proteomics & Bioinformatics 17, 4 (2019), 381–392. Big Data in Brain Science.Google ScholarCross Ref
- Anshul Choudhary, John F. Lindner, Elliott G. Holliday, Scott T. Miller, Sudeshna Sinha, and William L. Ditto. 2020. Physics-enhanced Neural Networks Learn Order and Chaos. Phys. Rev. E 101 (Jun 2020), 062207. Issue 6. https://link.aps.org/doi/10.1103/PhysRevE.101.062207Google ScholarCross Ref
- Juan-Pablo Correa-Baena, Kedar Hippalgaonkar, Jeroen van Duren, Shaffiq Jaffer, Vijay R. Chandrasekhar, Vladan Stevanovic, Cyrus Wadia, Supratik Guha, and Tonio Buonassisi. 2018. Accelerating Materials Development via Automation, Machine Learning, and High-Performance Computing. Joule 2, 8 (2018), 1410–1420.Google ScholarCross Ref
- Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. In Proceedings of the 24th International Conference on Artificial Intelligence (Buenos Aires, Argentina) (IJCAI’15). AAAI Press, 3460–3468.Google ScholarDigital Library
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010.11929 (2020).Google Scholar
- Francisco Erivaldo Fernandes Junior and Gary G. Yen. 2019. Particle Swarm Optimization of Deep Neural Networks Architectures for Image Classification. Swarm and Evolutionary Computation 49 (2019), 62–74.Google ScholarCross Ref
- A. Gertych, Z. Swiderska-Chadaj, and Z Ma. 2019. Convolutional Neural Networks Can Accurately Distinguish Four Histologic Growth Patterns of Lung Adenocarcinoma in Digital Slides. Scientific Reports 9, 1483 (2019).Google Scholar
- Jie Hou, Badri Adhikari, and Cheng Jianlin. 2018. DeepSF: Deep Convolutional Neural Network for Mapping Protein Sequences to Folds. Bioinformatics 34:8 (2018), 1295–1303.Google ScholarCross Ref
- Haifeng Jin, Qingquan Song, and Xia Hu. 2019. Auto-keras: An efficient neural architecture search system. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2019).Google ScholarDigital Library
- Travis Johnston, Steven R. Young, David Hughes, Robert M. Patton, and Devin White. 2017. Optimizing Convolutional Neural Networks for Cloud Detection. In Proceedings of the Machine Learning on HPC Environments (MLHPC) (Denver, CO, USA). Article 4, 9 pages.Google ScholarDigital Library
- A. Kamilaris and F. Prenafeta-Boldú. 2018. A Review of the Use of Convolutional Neural Networks in Agriculture. Journal of Agricultural Science 156:3 (2018), 312–322.Google ScholarCross Ref
- Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabás Póczos, and Eric P. Xing. 2018. Neural Architecture Search with Bayesian Optimisation and Optimal Transport. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. NeurIPS, 2020–2029.Google ScholarDigital Library
- M F Kasim, D Watson-Parris, L Deaconu, S Oliver, P Hatfield, D H Froula, G Gregori, M Jarvis, S Khatiwala, J Korenaga, and et al.2021. Building high accuracy emulators for scientific simulations with deep neural architecture search. Machine Learning: Science and Technology 3, 1 (2021), 015013.Google ScholarCross Ref
- Ariel Keller Rorabaugh, Silvina Caíno-Lores, Travis Johnston, and Michela Taufer. 2022. Building High-Throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine. IEEE Transactions on Parallel and Distributed Systems 33, 11 (2022), 2913–2926.Google Scholar
- Ariel Keller Rorabaugh, Silvina Caíno-Lores, Michael R. Wyatt II, Travis Johnston, and Michela Taufer. 2021. Architecture Descriptions and High Frequency Accuracy and Loss Data of Random Neural Networks Trained on Image Datasets.Google Scholar
- Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. 2017. Learning Curve Prediction with Bayesian Neural Networks. In International Conference on Learning Representations. https://openreview.net/forum?id=S11KBYclxGoogle Scholar
- Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. arxiv:1603.06560 [cs.LG]Google Scholar
- Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A Research Platform for Distributed Model Selection and Training. arXiv preprint arXiv:1807.05118 (2018).Google Scholar
- Chaoyue Liu and Mikhail Belkin. 2018. Accelerating SGD with Momentum for Over-parameterized Learning. arXiv preprint arXiv:1810.13395 (2018).Google Scholar
- Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2018. Progressive Neural Architecture Search. In Proceedings of the European Conference on Computer Vision (ECCV).Google ScholarDigital Library
- Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. 2018. Hierarchical Representations for Efficient Architecture Search. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. 2019. NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm. arxiv:1810.03522 [cs.CV]Google Scholar
- Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2018. Neural Architecture Optimization. In Advances in Neural Information Processing Systems 31. NeurIPS.Google Scholar
- Zhiping Mao, Ameya D. Jagtap, and George Em Karniadakis. 2020. Physics-informed Neural Networks for High-speed Flows. Computer Methods in Applied Mechanics and Engineering 360 (2020), 112789. https://www.sciencedirect.com/science/article/pii/S0045782519306814Google ScholarCross Ref
- Paula Olaya, Silvina Caino-Lores, Vanessa Lama, Ria Patel, Ariel Keller Rorabaugh, Osamu Miyashita, Florence Tama, and Michela Taufer. 2022. Identifying structural properties of proteins from X-ray free electron laser diffraction patterns. 2022 IEEE 18th International Conference on e-Science (e-Science) (2022).Google ScholarCross Ref
- Gyunam Park and Minseok Song. 2020. Predicting performances in business processes using deep neural networks. Decision Support Systems 129 (2020), 113191.Google ScholarDigital Library
- Ria Patel, Ariel Keller Rorabaugh, Paula Olaya, Silvina Caino-Lores, Georgia Channing, Catherine Schuman, Osamu Miyashita, Florence Tama, and Michela Taufer. 2022. A methodology to generate efficient neural networks for classification of scientific datasets. 2022 IEEE 18th International Conference on e-Science (e-Science) (2022).Google ScholarCross Ref
- Robert M Patton, J Travis Johnston, Steven R Young, Catherine D Schuman, Don D March, Thomas E Potok, Derek C Rose, Seung-Hwan Lim, Thomas P Karnowski, and Maxim A Ziatdinov. 2018. 167-PFlops Deep Learning for Electron Microscopy: From Learning Physics to Atomic Manipulation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). 638–648.Google ScholarDigital Library
- Robert M. Patton, J. Travis Johnston, Steven R. Young, Catherine D. Schuman, Thomas E. Potok, Derek C. Rose, Seung-Hwan Lim, Junghoon Chae, Le Hou, Shahira Abousamra, Dimitris Samaras, and Joel Saltz. 2019. Exascale Deep Learning to Accelerate Cancer Research. In 2019 IEEE International Conference on Big Data (Big Data). 1488–1496.Google ScholarCross Ref
- Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018).Google Scholar
- Rahul Ramesh and Pratik Chaudhari. 2022. Model Zoo: A Growing "Brain" That Learns Continually. arxiv:2106.03027 [cs.LG]Google Scholar
- Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. 2017. Large-scale Evolution of Image Classifiers. In Proceedings of the 34th International Conference on Machine Learning - Volume 70. ICML, 2902–2911.Google Scholar
- Binxin Ru, Clare Lyle, Lisa Schut, Miroslav Fil, Mark van der Wilk, and Yarin Gal. 2021. Speedy Performance Estimation for Neural Architecture Search. https://arxiv.org/abs/2006.04492. arxiv:2006.04492 [stat.ML]Google Scholar
- Max Schwarzer, Bryce Rogan, Yadong Ruan, Zhengming Song, Diana Y. Lee, Allon G. Percus, Viet T. Chau, Bryan A. Moore, Esteban Rougier, Hari S. Viswanathan, and Gowri Srinivasan. 2019. Learning to Fail: Predicting Fracture Evolution in Brittle material models using recurrent graph convolutional neural networks. Computational Materials Science 162 (2019), 322–332. https://www.sciencedirect.com/science/article/pii/S0927025619301223Google ScholarCross Ref
- Jonathan Shlomi, Peter Battaglia, and Jean-Roch Vlimant. 2021. Graph Neural Networks in Particle Physics. Machine Learning: Science and Technology 2, 2 (Jan 2021), 021001. https://doi.org/10.1088/2632-2153/abbf9aGoogle ScholarCross Ref
- Tiberiu Stan, Zachary T. Thompson, and Peter W. Voorhees. 2020. Optimizing Convolutional Neural Networks to Perform Semantic Segmentation on Large Materials Imaging Datasets: X-ray Tomography and Serial Sectioning. Materials Characterization 160 (2020), 110119. https://www.sciencedirect.com/science/article/pii/S1044580319304930Google ScholarCross Ref
- Yanan Sun, Bing Xue, Mengjie Zhang, and Gary G. Yen. 2020. Evolving deep convolutional neural networks for Image Classification. IEEE Transactions on Evolutionary Computation 24, 2 (2020), 394–407.Google ScholarCross Ref
- Tom Viering and Marco Loog. 2021. The Shape of Learning Curves: a Review. arXiv preprint arXiv:2103.10948 (2021).Google Scholar
- Guangyu Robert Yang and Xiao-Jing Wang. 2020. Artificial Neural Networks for Neuroscientists: A Primer. Neuron 107, 6 (2020), 1048–1070.Google ScholarCross Ref
- Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes. arXiv preprint arXiv:1904.00962 (2019).Google Scholar
- Steven R. Young, Derek C. Rose, Travis Johnston, William T. Heller, Thomas P. Karnowski, Thomas E. Potok, Robert M. Patton, Gabriel Perdue, and Jonathan Miller. 2017. Evolving Deep Networks using HPC. Proceedings of the Machine Learning on HPC Environments (2017).Google ScholarDigital Library
- Xiaolong Zheng, Peng Zheng, Liang Zheng, Yang Zhang, and Rui-Zhi Zhang. 2020. Multi-channel Convolutional Neural Networks for Materials Properties Prediction. Computational Materials Science 173 (2020), 109436. https://www.sciencedirect.com/science/article/pii/S0927025619307359Google ScholarCross Ref
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for Scalable Image Recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018).Google ScholarCross Ref
Index Terms
- Composable Workflow for Accelerating Neural Architecture Search Using In Situ Analytics for Protein Classification
Recommendations
Neural Architecture Search Applied to Hybrid Morphological Neural Networks
Intelligent SystemsAbstractThis work addresses a way to train morphological neural network differentially using backpropagation. The proposed algorithm can also learn whether to use erosion or dilation, based on the data being processed. Finally, we apply architecture ...
Fine-Grained Software Vulnerability Detection via Neural Architecture Search
Database Systems for Advanced ApplicationsAbstractVulnerability detection methods based on the deep learning have achieved remarkable performance improvements compared to traditional methods. Current deep learning-based detectors mostly use a single RNN or its variants (i.e., LSTM or GRU) to ...
A Training-free Genetic Neural Architecture Search
ACM ICEA '21: Proceedings of the 2021 ACM International Conference on Intelligent Computing and its Emerging ApplicationsThe so-called neural architecture search (NAS) provides an alternative way to construct a "good neural architecture," which would normally outperform hand-made architectures, for solving complex problems without domain knowledge. However, a critical ...
Comments