Abstract
We found that combining the L2 regularizer with Adam kills up to 60% of filters in ResNet-110 trained on CIFAR-100 as opposed to combining L2 with Momentum. It does not have a significant impact in terms of accuracy though, where both reach similar values. However, we found that this can be a serious issue if the impaired model is used as a pre-trained model for another more complex dataset (e.g. larger number of categories). This situation actually happens in continual learning. In this paper we conduct a study on the impact of dead filters in continual learning when the dataset increases its difficulty over time and more power from the network is required. Furthermore, we propose a new regularization term referred to as survival loss, that complements L2 to avoid filters to die when combined with Adam. We show that the survival loss improves accuracy in a simulated continual learning set-up, with the prospect of higher improvements as more iterations arrive.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:151107289 (2015)
Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12(7), 2121–2159 (2011)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:14126980 (2014)
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. 114(13), 3521–3526 (2017)
Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. University of Toronto (2012)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:171105101 (2017)
Lu, L., Shin, Y., Su, Y., Karniadakis, G.E.: Dying relu and initialization: theory and numerical examples. arXiv preprint arXiv:190306733 (2019)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp 807–814 (2010)
Nesterov, Y.: A method of solving a convex programming problem with convergence rate o(1/k2). Sov. Math. Dokl 27, 269 (1983)
Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)
Reddi, S.J., Kale, S., Kumar S.: On the convergence of adam and beyond. arXiv preprint arXiv:190409237 (2019)
Rosenfeld, J.S., Rosenfeld, A., Belinkov, Y., Shavit, N.: A constructive prediction of the generalization error across scales. arXiv preprint arXiv:190912673 (2019)
Shawn, B., Lapo, F., Thomas, M., Joel, L., Kenneth, O.S., Jeff, C., Cheney, N.: Learning to continually learn. arXiv preprint arXiv:200209571 (2020)
Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:170801547 (2017)
Zhang, M., Lucas, J., Ba, J., Hinton, G.E.: Lookahead optimizer: k steps forward, 1 step back. In: Advances in Neural Information Processing Systems, pp. 9593–9604 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Almazán, E.J., Tovar, J., Calle, A.d.l. (2021). Survival Loss: A Neuron Death Regularizer. In: Bergasa, L.M., Ocaña, M., Barea, R., López-Guillén, E., Revenga, P. (eds) Advances in Physical Agents II. WAF 2020. Advances in Intelligent Systems and Computing, vol 1285. Springer, Cham. https://doi.org/10.1007/978-3-030-62579-5_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-62579-5_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62578-8
Online ISBN: 978-3-030-62579-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)