skip to main content
10.1145/3377713.3377739acmotherconferencesArticle/Chapter ViewAbstractPublication PagesacaiConference Proceedingsconference-collections
research-article

Evolutionary NetArchitecture Search for Deep Neural Networks Pruning

Published: 07 February 2020 Publication History

Abstract

Network pruning is an architecture search process to determine the state (remove/remain) of neurons in the network. It is a com- binatorial optimization problem, and this combinatorial optimiza- tion problem is NP-hard. Most existing pruning methods prune channels/neurons based on the assumption that they are indepen- dent in network. However, there exists dependency among chan- nels/neurons. We try to solve the combinatorial optimization problem by evolutionary algorithm (EA). However, the traditional EA can't be used directly into deep neural networks (DNNs) because the problem dimension is too high. Attention mechanism (AM) can help us get parameter important score to reduce prob- lem difficulty, making the architecture search process more effective. Therefore, combining EA and AM, we propose an Evolutionary NetArchitecture Search (EvoNAS) method to solve network pruning problem. We demonstrate the effectiveness of our method on common datasets with ResNet, ResNeXt, and VGG. For example, for ResNet on CIFAR-10, EvoNAS reduces 73.40% computing operations and 73.95% parameters with 0.13% test accuracy increasement. Compared with the state-of-the-art methods, EvoNAS increases 30% reduction ratio at least.

References

[1]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. MIT Press, Cambridge, 1097--1105.
[2]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 1--9.
[3]
Cosmin Cernazanu-Glavan and Stefan Holban. 2013. Segmentation of bone structure in X-ray images using convolutional neural network. Adv. Electr. Comput. Eng 13, 1 (2013), 87--94.
[4]
Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference oncomputer vision and pattern recognition. IEEE, Piscataway, NJ, 3431--3440.
[5]
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 580--587.
[6]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. IEEE, Piscataway, NJ, 2961--2969.
[7]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[8]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 770--778.
[9]
Saining Xie, Ross Girshick, Piotr Dollr, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 1492--1500.
[10]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 4700--4708.
[11]
Yann LeCun, John S Denker, and Sara A Solla. 1990. Optimal brain damage. In Advances in neural information processing systems. MIT Press, Cambridge, 598--605.
[12]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
[13]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014).
[14]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).
[15]
Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016).
[16]
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural information processing systems. MIT Press, Cambridge, 1269--1277.
[17]
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. 2015. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067 (2015).
[18]
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. 2015. Accelerating very deep convolutional networks for classification and detection. 38, 10 (2015), 1943--1955.
[19]
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 (2016).
[20]
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision. Springer, Berlin, German, 525--542.
[21]
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. 2016. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 4820--4828.
[22]
Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).
[23]
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. 2018. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018).
[24]
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016).
[25]
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. 2018. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 9194--9203.
[26]
Zehao Huang and Naiyan Wang. 2018. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, Berlin, German, 304--320.
[27]
Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems. MIT Press, Cambridge, 1135--1143.
[28]
Yiwen Guo, Anbang Yao, and Yurong Chen. 2016. Dynamic network surgery for efficient dnns. In Advances In Neural Information Processing Systems. MIT Press, Cambridge, 1379--1387.
[29]
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems. MIT Press, Cambridge, 2074--2082.
[30]
Hao Zhou, Jose M Alvarez, and Fatih Porikli. 2016. Less is more: Towards compact cnns. In European Conference on Computer Vision. Springer, Berlin, German, 662--677.
[31]
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016. Pruning convolutional neural networks for resource efficient transfer learning. arXiv preprint arXiv:1611.06440-3 (2016).
[32]
Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Piscataway, NJ, 1389--1397.
[33]
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Piscataway, NJ, 2736--2744.
[34]
Dipankar Dasgupta and Douglas R McGregor. 1992. Designing applicationspecific neural networks using the structured genetic algorithm. In [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks. IEEE Computer Society, Los Alamitos, CA, 87--96.
[35]
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. 2017. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. ACM, New York, NY, 2902--2911.
[36]
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2019. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. AAAI, Menlo Park, CA, 4780--4789.
[37]
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. 2019. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing. Elsevier, London, 293--312.
[38]
Geoffrey F Miller, Peter M Todd, and Shailesh U Hegde. 1989. Designing Neural Networks using Genetic Algorithms. In ICGA, Vol. 89. 379--384.
[39]
Nils T Siebel, Jonas Botel, and Gerald Sommer. 2009. Efficient neural network pruning during neuro-evolution. In 2009 International Joint Conference on Neural Networks. IEEE, Piscataway, NJ, 2920--2927.
[40]
Kenneth O Stanley and Risto Miikkulainen. 2002. Evolving neural networks through augmenting topologies. Evolutionary computation 10, 2 (2002), 99--127.
[41]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning. ACM, New York, NY, 2048--2057.
[42]
Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 7132--7141.
[43]
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and TatSeng Chua. 2017. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 5659--5667.
[44]
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. 2017. Residual attention network for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 3156--3164.

Cited By

View all
  • (2024)Efficient Pruning of DenseNet via a Surrogate-Model-Assisted Genetic AlgorithmProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654409(295-298)Online publication date: 14-Jul-2024
  • (2024)External archive guided radial-grid multi objective differential evolutionScientific Reports10.1038/s41598-024-76877-x14:1Online publication date: 22-Nov-2024
  • (2024)Evolving filter criteria for randomly initialized network pruning in image classificationNeurocomputing10.1016/j.neucom.2024.127872594(127872)Online publication date: Aug-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ACAI '19: Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence
December 2019
614 pages
ISBN:9781450372619
DOI:10.1145/3377713
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • Chinese Univ. of Hong Kong: Chinese University of Hong Kong

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 February 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Attention mechanism
  2. Convolutional neural networks
  3. Evolutionary algorithm
  4. Network compression
  5. Network pruning

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ACAI 2019

Acceptance Rates

ACAI '19 Paper Acceptance Rate 97 of 203 submissions, 48%;
Overall Acceptance Rate 173 of 395 submissions, 44%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)21
  • Downloads (Last 6 weeks)1
Reflects downloads up to 27 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Efficient Pruning of DenseNet via a Surrogate-Model-Assisted Genetic AlgorithmProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654409(295-298)Online publication date: 14-Jul-2024
  • (2024)External archive guided radial-grid multi objective differential evolutionScientific Reports10.1038/s41598-024-76877-x14:1Online publication date: 22-Nov-2024
  • (2024)Evolving filter criteria for randomly initialized network pruning in image classificationNeurocomputing10.1016/j.neucom.2024.127872594(127872)Online publication date: Aug-2024
  • (2024)Joint filter and channel pruning of convolutional neural networks as a bi-level optimization problemMemetic Computing10.1007/s12293-024-00406-616:1(71-90)Online publication date: 17-Feb-2024
  • (2023)Transforming Large-Size to Lightweight Deep Neural Networks for IoT ApplicationsACM Computing Surveys10.1145/357095555:11(1-35)Online publication date: 9-Feb-2023
  • (2023)Embedding channel pruning within the CNN architecture design using a bi-level evolutionary approachThe Journal of Supercomputing10.1007/s11227-023-05273-579:14(16118-16151)Online publication date: 25-Apr-2023
  • (2022)Methods for Pruning Deep Neural NetworksIEEE Access10.1109/ACCESS.2022.318265910(63280-63300)Online publication date: 2022
  • (2022)An adaptive convergence enhanced evolutionary algorithm for many-objective optimization problemsSwarm and Evolutionary Computation10.1016/j.swevo.2022.10118075(101180)Online publication date: Dec-2022
  • (2022)A Novel Pruning Method Based on Correlation Applied in Full-Connection Layer NeuronsArtificial Intelligence and Security10.1007/978-3-031-06788-4_18(205-215)Online publication date: 15-Jul-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media