Abstract
Neural networks are increasingly applied to real-life problems by training models for solving prediction problems with known training data. Overfitting is one of the major problems that should be considered in designing neural network models. Among various approaches to treat the overfitting problem, DropOut and DropConnect are approaches to adjust the process of training by temporarily changing the structures of neural network models. In this study, we compare the effect of applying DropOut and DropConnect approaches to a deep neural network. The result of this study will help to understand the effect of applying DropOut and DropConnect to neural networks in terms of loss and the accuracy of the trained model. It is also expected to help to design the structure of deep neural networks by effectively applying DropOut and DropConnect to control overfitting in the training of machine learning models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Shai, S.-S., Shai, B.-D.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, Cambridge (2014)
Domingos, P.: A few useful things to know about machine learning. Commun. ACM 55(10), 78–87 (2012)
Salman, S., Liu, X.: Overfitting Mechanism and Avoidance in Deep Neural Networks (2019)
Ying, X.: An overview of overfitting and its solutions. J. Phys.: Conf. Ser. 1168(022022) (2019)
Cohen, P., Jensen, D.: Overfitting explained. In: Proceedings of the Sixth International Workshop on Artificial Intelligence and Statistics (2000)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Baldi, P., Sadowski, P.: Understanding dropout. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, pp. 2814–2822 (2013)
Duyck, J., Lee, M.H., Lei, E.: Modified dropout for training neural network (2014)
Labach, A., Salehinejad, H., Valaee, S.: Survey of Dropout Methods for Deep Neural Networks (2019)
Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In: Proceedings of the 30th International Conference on International Conference on Machine Learning, vol. 28, pp. 1058–1066 (2013)
Ravichandran, J., Saralajew, S., Villmann, T.: DropConnect for evaluation of classification stability in learning vector quantization. In: Proceedings of European Symposium on Artificial Neural Networks (ESANN 2019), pp. 19–24. Belgium (2019)
THE MNIST DATABASE of handwritten digits. https://yann.lecun.com/exdb/mnist/
Python programming language. https://www.python.org/
Keras: the Python deep learning API. https://keras.io/
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (Ministry of Education) (No. NRF-2017R1D1A1B03034769).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lim, Hi. (2021). A Study on Comparative Analysis of the Effect of Applying DropOut and DropConnect to Deep Neural Network. In: Singh, M., Kang, DK., Lee, JH., Tiwary, U.S., Singh, D., Chung, WY. (eds) Intelligent Human Computer Interaction. IHCI 2020. Lecture Notes in Computer Science(), vol 12615. Springer, Cham. https://doi.org/10.1007/978-3-030-68449-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-68449-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68448-8
Online ISBN: 978-3-030-68449-5
eBook Packages: Computer ScienceComputer Science (R0)