Skip to main content
SpringerLink
Log in

DLW-NAS: Differentiable Light-Weight Neural Architecture Search

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

In recent years, the method of automatically constructing convolutional neural networks based on neural architecture search has attracted wide attention, and greatly reduces the manual intervention and the cost of manual design of neural networks. However, most neural architecture search methods focus on the performance of the model, but ignore the complexity of the model, which makes it difficult to deploy this method on devices with limited resources. In this paper, a novel differentiable light-weight architecture search method named DLW-NAS is proposed, which aims to search convolutional neural networks (CNNs) with remarkable performance as well as a small amount of parameters and floating point operations (FLOPs). Concretely, in order to limit the parameters and FLOPs from the source of neural architecture search (NAS), we build a light-weight search space containing effective light-weight operations. Moreover, we design a differentiable NAS strategy with computation complexity constraints. In addition, we propose a neural architecture optimization method, which makes the topology of the searched architecture sparse. The experimental results show that DLW-NAS achieves 2.73% error rate on CIFAR-10, which is comparable to the state-of-the-art (SOTA) methods. However, it only needs 2.3M parameters and 334M FLOPs, which reduces that of the SOTA DARTS by 30% and 36% in parameters and FLOPs, respectively. The searched model on CIFAR-100 uses only 2.47M parameters and 376M FLOPs with an error rate of only 17.12%. On ImageNet, compared with the SOTA MobileNet, DLW-NAS achieves 3.3% better top-1 accuracy with much fewer parameters and FLOPs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Han K, Wang Y, Tian Q, Guo J, Xu C, Xu C. Ghostnet: more features from cheap operations. In: CVPR. 2020. p. 1577–586.

  2. Sun Y, Xue B, Zhang M, Yen GG, Lv J. Automatically designing CNN architectures using the genetic algorithm for image classification. IEEE Trans Cybern. 2020;50(9):3840–54.

    Article  Google Scholar 

  3. Orsic M, Segvic S. Efficient semantic segmentation with pyramidal fusion. Pattern Recognit. 2021;110:107611.

  4. Yang L, Wang H, Zeng Q, Liu Y, Bian G. A hybrid deep segmentation network for fundus vessels via deep-learning framework. Neurocomputing. 2021;448:168–78.

    Article  Google Scholar 

  5. Ebadi N, Jozani M, Choo KKR, Rad P. A memory network information retrieval model for identification of news misinformation. IEEE Transactions on Big Data. 2021.

  6. Mao Y, Zhong G, Wang H, Huang K. MCRN: a new content-based music classification and recommendation network. In: ICONIP, vol. 1332. 2020. p. 771–79.

  7. Zoph B, Le QV. Neural architecture search with reinforcement learning. In: ICLR. 2017.

  8. Liu H, Simonyan K, Yang Y. DARTS: differentiable architecture search. In: ICLR. 2019.

  9. Zhao J, Zhang R, Zhou Z, Chen S, Liu Q. A neural architecture search method based on gradient descent for remaining useful life estimation. Neurocomputing. 2021;438(1).

  10. Mao Y, Zhong G, Wang Y, Deng Z. Differentiable light-weight architecture search. In: IEEE International Conference on Multimedia and Expo (ICME). 2021. p. 1–6. https://doi.org/10.1109/ICME51207.2021.9428132.

  11. Iandola FN, Moskewicz MW, Ashraf K, Han S, Dally WJ, Keutzer K. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size. CoRR abs/1602.07360. 2016.

  12. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. Mobilenets: efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861. 2017.

  13. Zhang X, Zhou X, Lin M, Sun J. Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: CVPR. 2018. p. 6848–6856.

  14. Cai H, Gan C, Wang T, Zhang Z, Han S. Once-for-All: train one network and specialize it for efficient deployment. In: ICLR. 2020.

  15. Wu B, Dai X, Zhang P, Wang Y, Sun F, Wu Y, Tian Y, Vajda P, Jia Y, Keutzer K. FBNet: hardware-aware efficient ConvNet design via differentiable neural architecture search. In: CVPR. 2019. p. 10734–10742.

  16. Cai H, Zhu L, Han S. ProxylessNAS: direct neural architecture search on target task and hardware. In: ICLR. 2019.

  17. Zhou H, Yang M, Wang J, Pan W. BayesNAS: a Bayesian approach for neural architecture search. In: ICML, vol. 97. 2019. p. 7603–613.

  18. Zhang X, Huang Z, Wang N. You only search once: single shot neural architecture search via direct sparse optimization. CoRR abs/1811.01567. 2019.

  19. Weng Y, Zhou T, Liu L, Xia C. Automatic convolutional neural architecture search for image classification under different scenes. IEEE Access. 2019;7:38495–506.

    Article  Google Scholar 

  20. Li X, Wang W, Hu X, Yang J. Selective kernel networks. In: CVPR. 2019. p. 510–19.

  21. Li G, Zhang X, Wang Z, Li Z, Zhang T. STACNAS: towards stable and consistent optimization for differentiable neural architecture search. CoRR abs/1909.11926. 2019.

  22. Wu Y, Liu A, Huang Z, Zhang S, Gool LV. Neural architecture search as sparse supernet. CoRR abs/2007.16112. 2020.

  23. Krizhevsky A. Learning multiple layers of features from tiny images. University of Toronto. 2012.

  24. Deng J, Dong W, Socher R, Li L, Li K, Li F. Imagenet: a large-scale hierarchical image database. In: CVPR. 2009. p. 248–55.

  25. Chen X, Xie L, Wu J, Tian Q. Progressive differentiable architecture search: bridging the depth gap between search and evaluation. In: ICCV. 2019. p. 1294–303

  26. Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: CVPR. 2015. p. 1–9.

  27. Devries T, Taylor GW. Improved regularization of convolutional neural networks with cutout. CoRR abs/1708.04552. 2017

  28. Huang G, Liu Z, vander Maaten L, Weinberger KQ. Densely connected convolutional networks. In: CVPR. 2017. p. 2261–269.

  29. Li L, Talwalkar A. Random search and reproducibility for neural architecture search. In: Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI, Tel Aviv, Israel, July 22–25. AUAI Press; 2019. p. 367–77.

  30. Real E, Aggarwal A, Huang Y, Le QV. Regularized evolution for image classifier architecture search. In: AAAI. 2019. p. 4780–4789.

  31. Liu H, Simonyan K, Vinyals O, Fernando C, Kavukcuoglu K. Hierarchical representations for efficient architecture search. In: 6th International Conference on Learning Representations, ICLR, Vancouver, BC, Canada, April 30 - May 3, Conference Track Proceedings. 2018. OpenReview.net.

  32. Yang Z, Wang Y, Chen X, Shi B, Xu C, Xu C, Tian Q, Xu C. CARS: continuous evolution for efficient neural architecture search. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, Seattle, WA, USA, June 13-19. IEEE; 2020. p. 1826–835.

  33. Baker B, Gupta O, Naik N, Raskar R. Designing neural network architectures using reinforcement learning. In: 5th International Conference on Learning Representations. ICLR; 2017. OpenReview.net.

  34. Zhong Z, Yang Z, Deng B, Yan J, Wu W, Shao J, Liu C. Blockqnn: Efficient block-wise neural network architecture generation. CoRR. 2018.

  35. Pham H, Guan MY, Zoph B, Le QV, Dean J. Efficient neural architecture search via parameter sharing. In: ICML, vol. 80. 2018. p. 4092–4101.

  36. Cai H, Yang J, Zhang W, Han S, Yu Y. Path-level network transformation for efficient architecture search. In: Proceedings of the 35th International Conference on Machine Learning, ICML, Stockholmsmässan, Stockholm, Sweden, July 10–15. PMLR; 2018. p. 677–86.

  37. Liu C, Zoph B, Neumann M, Shlens J, Hua W, Li L, Fei-Fei L, Yuille AL, Huang J, Murphy K. Progressive neural architecture search. In: Computer Vision - ECCV - 15th European Conference, Munich, Germany, September 8–14, Proceedings, Part I. Springer; 2018.

  38. Perez-Rua J, Baccouche M, Pateux S. Efficient progressive neural architecture search. In: British Machine Vision Conference, BMVC, Newcastle, UK, September 3-6. BMVA Press; 2018. p. 150.

  39. Zhang C, Ren M, Urtasun R. Graph hypernetworks for neural architecture search abs/1810.05749. 2018.

  40. Xie S, Zheng H, Liu C, Lin L. SNAS: stochastic neural architecture search. In: ICLR. 2019.

  41. Dong X, Yang Y. Searching for a robust neural architecture in four GPU hours. In: CVPR. 2019. p. 1761–1770.

  42. Xie S, Girshick RB, Dollár P, Tu Z, He K. Aggregated residual transformations for deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, HI, USA, July 21–26. IEEE Computer Society; 2017. p. 5987–995.

  43. Zoph B, Vasudevan V, Shlens J, Le QV. Learning transferable architectures for scalable image recognition. In: CVPR. 2018. p. 8697–8710.

Download references

Funding

This work was partially supported by the National Key Research and Development Program of China under Grant No. 2018AAA0100400, the Natural Science Foundation of Shandong Province under Grants No. ZR2020MF131 and No. ZR2021ZD19, and the Science and Technology Program of Qingdao under Grant No. 21-1-4-ny-19-nsh.

Author information

Authors and Affiliations

  1. Ocean University of China, Qingdao, 266100, China

    Shu Li, Yuxu Mao, Fuchang Zhang, Dong Wang & Guoqiang Zhong

Authors
  1. Shu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuxu Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fuchang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guoqiang Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Dong Wang or Guoqiang Zhong.

Ethics declarations

Ethics Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Conflicts of Interest

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Mao, Y., Zhang, F. et al. DLW-NAS: Differentiable Light-Weight Neural Architecture Search. Cogn Comput 15, 429–439 (2023). https://doi.org/10.1007/s12559-022-10046-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-022-10046-y

Keywords

Search

Navigation