skip to main content
10.1145/3587716.3587725acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlcConference Proceedingsconference-collections
research-article

ZeroLess-DARTS: Improved Differentiable Architecture Search with Refined Search Operation and Early Stopping

Published: 07 September 2023 Publication History

Abstract

Differentiable architecture search (DARTS) method has gained noticeable popularity in neural architecture search (NAS) domain as it reduces the required search time compared to reinforcement learning and evolutionary based NAS algorithms. However, some further studies have indicated that the search algorithm of DARTS may be suboptimal, and its performance may deteriorate over the search process. In this paper, we provided a detailed performance analysis of the DARTS search algorithm (on the CIFAR10 image classification task) from different aspects such as changes in accuracies of derived architectures at each search epoch, the trend of changes in strengths of different operations over successive epochs, and the number of skip connections per normal cells. We propose ZeroLess-DARTS that considerably improves original DARTS performance on the CIFAR10 dataset (Cohen's D = 2.213), by refining the operation space in the search procedure and introducing an early stopping criterion. We show that our approach is generalizable to time series classification tasks by evaluating the performance of our model on one-dimensional ECG signals for WCT (Wide Complex Tachycardia) classification (Cohen's D = 1.706).

References

[1]
Krizhevsky, A., Sutskever, I. and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25 (2012).
[2]
Ren, S., He, K., Girshick, R. and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28 (2015).
[3]
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q. and Macherey, K. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016).
[4]
Fayyazifar, N. An accurate CNN architecture for atrial fibrillation detection using neural architecture search. IEEE, City, 2021.
[5]
Fayyazifar, N., Ahderom, S., Suter, D., Maiorana, A. and Dwivedi, G. Impact of neural architecture design on cardiac abnormality classification using 12-lead ECG signals. IEEE, City, 2020.
[6]
Fayyazifar, N., Dwivedi, G., Suter, D., Ahderom, S., Maiorana, A., Clarkin, O., Balamane, S., Saha, N., King, B. and Green, M. S. A novel convolutional neural network structure for differential diagnosis of wide QRS complex tachycardia. Biomedical Signal Processing and Control, 81 (2023), 104506.
[7]
Mrazova, I. and Kukacka, M. Image classification with growing neural networks. International Journal of Computer Theory and Engineering, 5, 3 (2013), 422.
[8]
Thanh, T. T. P., Vununu, C., Atoev, S., Lee, S.-H. and Kwon, K.-R. Leukemia blood cell image classification using convolutional neural network. International Journal of Computer Theory and Engineering, 10, 2 (2018), 54-58.
[9]
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A. Going deeper with convolutions. City, 2015.
[10]
He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition. City, 2016.
[11]
Bergstra, J., Yamins, D. and Cox, D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. PMLR, City, 2013.
[12]
Bello, I., Zoph, B., Vasudevan, V. and Le, Q. V. Neural optimizer search with reinforcement learning. PMLR, City, 2017.
[13]
Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y. L., Tan, J., Le, Q. V. and Kurakin, A. Large-scale evolution of image classifiers. PMLR, City, 2017.
[14]
Hanxiao Liu, K. S., Yiming Yang DARTS: Differentiable architecture search. International Conference on Learning Representations, ICLR 2019: New Orleans, Louisiana, United States (2018).
[15]
Chen, X., Xie, L., Wu, J. and Tian, Q. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. City, 2019.
[16]
Chu, X., Zhou, T., Zhang, B. and Li, J. Fair darts: Eliminating unfair advantages in differentiable architecture search. Springer, City, 2020.
[17]
Liang, H., Zhang, S., Sun, J., He, X., Huang, W., Zhuang, K. and Li, Z. Darts+: Improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909.06035 (2019).
[18]
Yang, A., Esperança, P. M. and Carlucci, F. M. NAS evaluation is frustratingly hard. arXiv preprint arXiv:1912.12522 (2019).
[19]
Zhou, P., Xiong, C., Socher, R. and Hoi, S. C. H. Theory-inspired path-regularized differential network architecture search. Advances in Neural Information Processing Systems, 33 (2020), 8296-8307.
[20]
Arber Zela, T. E., Saikia, T., Marrakchi, Y., Brox, T. and Hutter, F. Understanding and robustifying differentiable architecture search. City, 2020.
[21]
Lowry, R. Concepts and applications of inferential statistics (2014).
[22]
Cohen, J. Statistical power analysis for the behavioral sciences. Routledge, 2013.
[23]
Sawilowsky, S. S. New effect size rules of thumb. Journal of modern applied statistical methods, 8, 2 (2009), 26.

Index Terms

  1. ZeroLess-DARTS: Improved Differentiable Architecture Search with Refined Search Operation and Early Stopping
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        ICMLC '23: Proceedings of the 2023 15th International Conference on Machine Learning and Computing
        February 2023
        619 pages
        ISBN:9781450398411
        DOI:10.1145/3587716
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 07 September 2023

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Differentiable architecture search
        2. Neural architecture search
        3. early stopping
        4. operation space

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        ICMLC 2023

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • 0
          Total Citations
        • 59
          Total Downloads
        • Downloads (Last 12 months)41
        • Downloads (Last 6 weeks)3
        Reflects downloads up to 20 Jan 2025

        Other Metrics

        Citations

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media