Skip to main content
SpringerLink
Log in

Improved face super-resolution generative adversarial networks

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

The face super-resolution method is used for generating high-resolution images from low-resolution ones for better visualization. The super-resolution generative adversarial network (SRGAN) can generate a single super-resolution image with realistic textures, which is a groundbreaking work. Based on SRGAN, we proposed improved face super-resolution generative adversarial networks. The super-resolution image details generated by SRGAN usually have undesirable artifacts. To further improve visual quality, we delve into the key components of the SRGAN network architecture and improve each part to achieve a more powerful SRGAN. First, the SRGAN employs residual blocks as the core of the very deep generator network G. In this paper, we decide to employ dense convolutional network blocks (dense blocks), which connect each layer to every other layer in a feed-forward fashion as our very deep generator networks. Moreover, in the past few years, generative adversarial networks (GANs) have been applied to solve various problems. Despite its superior performance, it is difficult to train. A simple and effective regularization method called spectral normalization GAN is used to solve this problem. We have experimentally confirmed that our proposed method is superior to the other existing method in training stability and visual improvements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Jourabloo, A., Ye, M., Liu, X., Ren, L.: Pose-invariant face alignment with a single CNN. In: The IEEE International Conference on Computer Vision (ICCV) (2017)

  2. Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In IEEE Conference on Computer Vision & Pattern Recognition (CVPR) (2015)

  3. Taigman, Y., Ming, Y., Ranzato, M., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: IEEE Conference on Computer Vision & Pattern Recognition (2014)

  4. Yang, J., Luo, L., Qian, J., Tai, Y., Zhang, F., Xu, Y.: Nuclear norm based matrix regression with applications to face recognition with occlusion and illumination changes. IEEE Trans. Pattern Anal. Mach. Intell. 39(1), 156–171 (2016)

    Article  Google Scholar 

  5. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  6. Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision & Pattern Recognition (2017)

  7. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks (2018). arXiv:1802.05957

  8. Arjovsky, M., Bottou, L.: Towards principled methods for training generative adversarial networks (2017). arXiv:1701.04862

  9. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan (2017). arXiv:1701.07875

  10. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)

  11. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015)

  12. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: IEEE International Conference on Computer Vision (2017)

  13. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407. Springer (2016)

  14. He, K., Zhang, X., Ren, S., Jian, S.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision & Pattern Recognition (2016)

  15. Gao, H., Zhuang, L., Maaten, L.V.D., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision & Pattern Recognition (2017)

  16. Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Asian Conference on Computer Vision, pp. 111–126. Springer (2014)

  17. Yang, J., Wright, J., Huang, T.S., Yi, M.: Image super-resolution as sparse representation of raw image patches. In: IEEE Conference on Computer Vision & Pattern Recognition (2008)

  18. Gao, X., Zhang, K., Tao, D., Li, X.: Image super-resolution with sparse neighbor embedding. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 21(7), 3194 (2012)

    MathSciNet  MATH  Google Scholar 

  19. Salvador, J., Perez-Pellitero, E.: Naive bayes super-resolution forest. In: IEEE International Conference on Computer Vision (2016)

  20. Shi, W., Caballero, J., Huszr, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Computer Vision & Pattern Recognition (2016)

  21. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision & Pattern Recognition (2016)

  22. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 624–632 (2017)

  23. Ying, T., Jian, Y., Liu, X.: Image super-resolution via deep recursive residual network. In: IEEE Conference on Computer Vision & Pattern Recognition (2017)

  24. Mao, X.-J., Shen, C., Yang, Y.-B.: Image Denoising Using Very Deep Fully Convolutional Encoder-decoder Networks with Symmetric Skip Connections, vol. 2 (2016). arXiv:1603.09056

  25. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

  26. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 1–23 (2018)

  27. Baker, S., Kanade, T.: Hallucinating faces. In: IEEE International Conference on Automatic Face & Gesture Recognition (2000)

  28. Liu, C., Shum, H.Y., Zhang, C.S.: Two-step approach to hallucinating faces: global parametric model and local nonparametric model. In: IEEE Computer Society Conference on Computer Vision & Pattern Recognition (2001)

  29. Wang, X., Tang, X.: Hallucinating face by eigentransformation. IEEE Trans. Syst. Man Cybern. C 35(3), 425–434 (2005)

    Article  Google Scholar 

  30. Zhu, S., Liu, S., Loy, C.C., Tang, X.: Deep cascaded bi-network for face hallucination. In: European Conference on Computer Vision, pp. 614–630. Springer (2016)

  31. Yu, X., Porikli, F.: Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3760–3768 (2017)

  32. Yu, X., Fernando, B., Ghanem, B., Porikli, F., Hartley, R.: Face super-resolution guided by facial component heatmaps. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 217–233 (2018)

  33. Bulat, A., Yang, J., Tzimiropoulos, G.: To learn image super-resolution, use a gan to learn how to do image degradation first. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 185–200 (2018)

  34. Chen, Y., Tai, Y., Liu, X., Shen, C., Yang, J.: Fsrnet: end-to-end learning face super-resolution with facial priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2492–2501 (2018)

  35. Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis (2018). arXiv:1809.11096

  36. Salimans, T., Kingma, D.P.: Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems, pp. 901–909 (2016)

  37. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  38. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Computer Vision & Pattern Recognition (2016)

  39. Qi, G.-J.: Loss-sensitive generative adversarial networks on lipschitz densities (2017). arXiv:1701.06264

  40. Johnson, J., Alahi, A., Li, F.F.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (2016)

  41. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv:1409.1556

  42. Learned-Miller, E., Huang, G.B., Roychowdhury, A., Li, H., Gang, H.: Labeled Faces in the Wild: A Survey. In: Advances in face detection and facial image analysis, pp. 189–248. Springer (2016)

  43. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation (2017). arXiv:1710.10196

  44. Zhang, K., Zhang, Z., Li, Z., Yu, Q.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  45. Goodfellow, I.J.: On distinguishability criteria for estimating generative models. Statistics (2015). arXiv:1412.6515

  46. Golub, G.H., Vorst, H.A.V.D.: Eigenvalue computation in the 20th century. J. Comput. Appl. Math. 123(1), 35–65 (2000)

    Article  MathSciNet  Google Scholar 

  47. Yoshida, Y., Miyato, T.: Spectral norm regularization for improving the generalizability of deep learning (2017). arXiv:1705.10941

  48. Xiang, M., Zhang, J., Qi, C.: Hallucinating face by position-patch. Pattern Recogn. 43(6), 2224–2236 (2010)

    Article  Google Scholar 

  49. Chen, Z., Tong, Y.: Face super-resolution through wasserstein gans (2017). arXiv:1705.02438

Download references

Acknowledgements

The authors would like to thank Karras et al. for sharing the CelebAHQ dataset and Han, Hu et al. for sharing LFW dataset. This work was supported in part by the National Natural Science Foundation of China (61876099), in part by the National Key R&D Program of China (2019YFB1311001), in part by the Scientific and Technological Development Project of Shandong Province (2019GSF111002), in part by the Shenzhen Science and Technology Research and Development Funds (JCYJ20180305164401921), in part by the Foundation of Ministry of Education Key Laboratory of System Control and Information Processing (Scip201801), in part by the Foundation of Key Laboratory of Intelligent Computing & Information Processing of Ministry of Education (2018ICIP03), and in part by the Foundation of State Key Laboratory of Integrated Services Networks (ISN20-06).

Author information

Authors and Affiliations

  1. School of Control Science and Engineering, Shandong University, Jinan, 250061, China

    Mengxue Wang & Zhenxue Chen

  2. Shenzhen Research Institute of Shandong University, Shandong University, Shenzhen, 518057, China

    Zhenxue Chen

  3. Department of Electrical and Computer Engineering, University of Windsor, Windsor, N9B 3P4, Canada

    Q. M. Jonathan Wu

  4. School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, 250014, China

    Muwei Jian

Authors
  1. Mengxue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenxue Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Q. M. Jonathan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muwei Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenxue Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, M., Chen, Z., Wu, Q.M.J. et al. Improved face super-resolution generative adversarial networks. Machine Vision and Applications 31, 22 (2020). https://doi.org/10.1007/s00138-020-01073-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-020-01073-6

Keywords

Search

Navigation