Abstract
The emergence of neural architecture search (NAS) technology has lowered the professional threshold for optimizing model architectures. However, existing NAS methods primarily evaluate performance by fully training a network architecture, which is computationally expensive and slow. This paper proposes a hybrid performance estimation strategy search framework for neural architecture search, which can flexibly adjust the performance evaluation strategy at each stage. In the initial stage, this study uses less accurate but low-cost methods to quickly eliminate suboptimal architectures. As the search progresses, more computationally intensive but accurate evaluation strategies are employed to filter out the optimal network architectures. In the final stage, more precise verification is conducted to ensure that the selected network architecture achieves the best performance in practice. This research can adapt to different precision and speed requirements, providing flexible reduction space ratio strategies aimed at meeting accuracy requirements while maintaining efficiency. Its generalizability and flexibility help address various NAS challenges. Experimental results show that the method proposed in this study performs excellently in multiple benchmark tests, achieving a balance between performance and efficiency. Additionally, by testing on other search spaces, datasets, and tasks, this study demonstrates its good generalization ability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdelfattah, M.S., Mehrotra, A., Dudziak, Ł., Lane, N.D.: Zero-cost proxies for lightweight NAS. arXiv preprint arXiv:2101.08134 (2021)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware. In: International Conference on Learning Representations (2018)
Casale, F.P., Gordon, J., Fusi, N.: Probabilistic neural architecture search. arXiv preprint arXiv:1902.05116 (2019)
Chen, W., Gong, X., Wang, Z.: Neural architecture search on imagenet in four GPU hours: a theoretically inspired perspective. In: International Conference on Learning Representations (2020)
Chen, X., Xie, L., Wu, J., Tian, Q.: Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1294–1303 (2019)
Chu, X., Wang, X., Zhang, B., Lu, S., Wei, X., Yan, J.: DARTS-: robustly stepping out of performance collapse without indicators. In: International Conference on Learning Representations (2020)
Chu, X., Zhang, B., Xu, R.: FairNAS: rethinking evaluation fairness of weight sharing neural architecture search. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12,239–12,248 (2021)
Chu, X., Zhou, T., Zhang, B., Li, J.: Fair DARTS: eliminating unfair advantages in differentiable architecture search. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12360, pp. 465–480. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58555-6_28
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Dong, X., Yang, Y.: NAS-bench-201: extending the scope of reproducible neural architecture search. In: International Conference on Learning Representations (2019)
Dong, X., Yang, Y.: One-shot neural architecture search via self-evaluated template network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3681–3690 (2019)
Dong, X., Yang, Y.: Searching for a robust neural architecture in four GPU hours. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1761–1770 (2019)
Elsken, T., et al.: NASLIB: a modular and flexible neural architecture search library. J. Mach. Learn. Res. 22(200), 1–8 (2021)
Guo, Z., et al.: Single path one-shot neural architecture search with uniform sampling. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 544–560. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_32
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Kramer, O., Kramer, O.: Scikit-learn. In: Machine Learning for Evolution Strategies, pp. 45–53. Springer. Cham (2016). https://doi.org/10.1007/978-3-319-33383-0_5
Krishnakumar, A., White, C., Zela, A., Tu, R., Safari, M., Hutter, F.: NAS-bench-suite-zero: accelerating research on zero cost proxies. Adv. Neural. Inf. Process. Syst. 35, 28037–28051 (2022)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. In: Handbook of Systemic Autoimmune Diseases, vol. 1, no. 4 (2009)
Li, G., Qian, G., Delgadillo, I.C., Muller, M., Thabet, A., Ghanem, B.: SGAS: sequential greedy architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1620–1630 (2020)
Lin, M., et al.: Zen-NAS: a zero-shot NAS for high-performance image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 347–356 (2021)
Liu, C., et al.: Progressive neural architecture search. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 19–35. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_2
Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)
Ma, N., Zhang, X., Zheng, H.-T., Sun, J.: ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 122–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_8
Mellor, J., Turner, J., Storkey, A., Crowley, E.J.: Neural architecture search without training. In: International Conference on Machine Learning, pp. 7588–7598. PMLR (2021)
Moser, B., Raue, F., Hees, J., Dengel, A.: Less is more: proxy datasets in NAS approaches. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1953–1961 (2022)
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: International Conference on Machine Learning, pp. 4095–4104. PMLR (2018)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4780–4789 (2019)
Tanaka, H., Kunin, D., Yamins, D.L., Ganguli, S.: Pruning neural networks without any data by iteratively conserving synaptic flow. Adv. Neural. Inf. Process. Syst. 33, 6377–6389 (2020)
Xie, S., Zheng, H., Liu, C., Lin, L.: SNAS: stochastic neural architecture search. In: International Conference on Learning Representations (2018)
Xu, Y., et al.: PC-DARTS: partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737 (2019)
Yu, K., Ranftl, R., Salzmann, M.: How to train your super-net: an analysis of training heuristics in weight-sharing NAS. arXiv preprint arXiv:2003.04276 (2020)
Zhang, M., Pan, S., Chang, X., Su, S., Hu, J., Haffari, G.R., Yang, B.: Balenas: Differentiable architecture search via the bayesian learning rule. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11,871–11,880 (2022)
Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)
Zheng, X., et al.: Neural architecture search with representation mutual information. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11,912–11,921 (2022)
Zheng, X., et al.: MIGO-NAS: towards fast and generalizable neural architecture search. IEEE Trans. Pattern Anal. Mach. Intell. 43(9), 2936–2952 (2021)
Zheng, X., Ji, R., Tang, L., Zhang, B., Liu, J., Tian, Q.: Multinomial distribution learning for effective neural architecture search. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1304–1313 (2019)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, L. et al. (2024). A Hybrid Performance Estimation Strategy for Optimizing Neural Architecture Search. In: Zheng, H., Glass, D., Mulvenna, M., Liu, J., Wang, H. (eds) Advances in Computational Intelligence Systems. UKCI 2024. Advances in Intelligent Systems and Computing, vol 1462. Springer, Cham. https://doi.org/10.1007/978-3-031-78857-4_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-78857-4_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-78856-7
Online ISBN: 978-3-031-78857-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)