Abstract
Deep learning is a state-of-the-art learning method that is used in fields such as visual object recognition and speech recognition. This learning uses a large number of layers and a huge number of units and connections, so overfitting occurs. Dropout learning is a kind of regularizer that neglects some inputs and hidden units in the learning process with a probability p; then, the neglected inputs and hidden units are combined with the learned network to express the final output. We compared dropout learning and ensemble learning from three viewpoints and found that dropout learning can be regarded as ensemble learning that divides the student network into two groups of hidden units. From this insight, we explored novel dropout learning that divides the student network into more than two groups of hidden units to enhance the benefit of ensemble learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advanced in Neural Information Processing System 25 (2012)
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 119–139 (1997)
Hara, K., Okada, M.: Ensemble learning of linear perceptrons: on-line learning theory. J. Phys. Soc. Jpn. 74(11), 2966–2972 (2005)
Miyoshi, S., Hara, K., Okada, M.: Analysis of ensemble learning using simple perceptron based on online learning theory. Phys. Rev. E 71, 036116 (2005)
Biehl, M., Schwarze, H.: Learning by on-line gradient descent. J. Phys. A Math. Gen. Phys. 28, 643–656 (1995)
Saad, D., Solla, S.A.: On-line learning in soft-committee machines. Phys. Rev. E 52, 4225–4243 (1995)
Wager, S., Wang, S., Liang, P.: Dropout training as adaptive regularization. In: Advance in Neural Information Processing System 26 (2013)
Bishop, C.M.: Pattern Recognition and Machine learning. Springer, New York (2006)
Acknowledgments
The authors thank Professor Masato Okada and Assistant Professor Hideitsu Hino for insightful discussions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Hara, K., Saitoh, D., Kondou, T., Suzuki, S., Shouno, H. (2016). Group Dropout Inspired by Ensemble Learning. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds) Neural Information Processing. ICONIP 2016. Lecture Notes in Computer Science(), vol 9948. Springer, Cham. https://doi.org/10.1007/978-3-319-46672-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-46672-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-46671-2
Online ISBN: 978-3-319-46672-9
eBook Packages: Computer ScienceComputer Science (R0)