Abstract
Generative adversarial networks (GANs) have become widespread models for complex density estimation tasks such as image generation or image-to-image synthesis. At the same time, training of GANs can suffer from several problems, either of stability or convergence, sometimes hindering their effective deployment. In this paper we investigate whether we can improve GAN training by endowing the neural network models with more flexible activation functions compared to the commonly used rectified linear unit (or its variants). In particular, we evaluate training a deep convolutional GAN wherein all hidden activation functions are replaced with a version of the kernel activation function (KAF), a recently proposed technique for learning non-parametric nonlinearities during the optimization process. On a thorough empirical evaluation on multiple image generation benchmarks, we show that the resulting architectures learn to generate visually pleasing images in a fraction of the number of the epochs, eventually converging to a better solution, even when we equalize (or even lower) the number of free parameters. Overall, this points to the importance of investigating better and more flexible architectures in the context of GANs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Proceedings of the 2017 International Conference on Machine Learning (ICML), pp. 214–223 (2017)
Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: Proceedings of the 2019 International Conference on Learning Representations (ICLR) (2019)
Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., Bharath, A.A.: Generative adversarial networks: an overview. IEEE Signal Process. Mag. 35(1), 53–65 (2018)
Firmani, D., Merialdo, P., Nieddu, E., Scardapane, S.: In codice ratio: OCR of handwritten Latin documents using deep convolutional networks. In: CEUR Workshop Proceedings, pp. 9–16 (2017)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, pp. 6626–6637 (2017)
Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the 2013 International Conference on Machine Learning (ICML), vol. 30, p. 3 (2013)
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2794–2802 (2017)
Marra, G., Zanca, D., Betti, A., Gori, M.: Learning neuron non-linearities with kernel-based deep neural networks. arXiv preprint arXiv:1807.06302 (2018)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)
Scardapane, S., Van Vaerenbergh, S., Comminiello, D., Totaro, S., Uncini, A.: Recurrent neural networks with flexible gates using kernel activation functions. In: Proceedings of the 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP). pp. 1–6. IEEE (2018)
Scardapane, S., Van Vaerenbergh, S., Totaro, S., Uncini, A.: Kafnets: kernel-based non-parametric activation functions for neural networks. Neural Networks 110, 19–32 (2019)
Song, J., He, T., Gao, L., Xu, X., Hanjalic, A., Shen, H.T.: Binary generative adversarial networks for image retrieval. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: ESRGAN: Enhanced super-resolution generative adversarial networks. In: Proceedings of the 2018 European Conference on Computer Vision (ECCV) (2018)
Wolterink, J.M., Leiner, T., Viergever, M.A., Išgum, I.: Generative adversarial networks for noise reduction in low-dose ct. IEEE Trans. Med. Imaging 36(12), 2536–2545 (2017)
Zhang, X., Trmal, J., Povey, D., Khudanpur, S.: Improving deep neural network acoustic models using generalized maxout networks. In: Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 215–219. IEEE (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Grassucci, E., Scardapane, S., Comminiello, D., Uncini, A. (2021). Flexible Generative Adversarial Networks with Non-parametric Activation Functions. In: Esposito, A., Faundez-Zanuy, M., Morabito, F., Pasero, E. (eds) Progresses in Artificial Intelligence and Neural Systems. Smart Innovation, Systems and Technologies, vol 184. Springer, Singapore. https://doi.org/10.1007/978-981-15-5093-5_7
Download citation
DOI: https://doi.org/10.1007/978-981-15-5093-5_7
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-5092-8
Online ISBN: 978-981-15-5093-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)