Skip to main content
Springer Nature Link
Log in

The face image super-resolution algorithm based on combined representation learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Face super-resolution reconstruction is the process of predicting high-resolution face images from one or more observed low-resolution face images, which is a typical pathological problem. As a domain-specific super-resolution task, we can use facial priori knowledge to improve the effect of super-resolution. We propose a method of face image super-resolution reconstruction based on combined representation learning method, using deep residual networks and deep neural networks as generators and discriminators, respectively. First, the model uses residual learning and symmetrical cross-layer connection to extract multilevel features. Local residual mapping improves the expressive capability of the network to enhance performance, solves gradient dissipation in network training, and reduces the number of convolution cores in the model through feature reuse. The feature expression of the face image at the high-dimensional visual level is obtained. The visual feature is sent to the decoder through the cross-layer connection structure. The deconvolution layer is used to restore the spatial dimension gradually and repair the details and texture features of the face. Finally, combine the attention block and the residual block reconstruction in the deep residual network to super-resolution face images that are highly similar to high-resolution images and difficult to be discriminated by the discriminator. On this basis, combined representation learning is conducted to obtain numerous realistic results of visual perception. The experimental results on the face datasets can show that the Peak Signal-to-Noise Ratio of the proposed method is improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

Not applicable.

References

  1. Caruana R (1994) Learning many related tasks at the same time with backpropagation. In: Proceedings of international conference on neural information processing systems, Denver, Colorado. MIT Press, USA, pp 657–664

    Google Scholar 

  2. Chang H, Yeung DY, Xiong YM (2004) Super-resolution through neighbor embedding. In: proceedings of IEEE conference on computer vision and pattern recognition, Washington, DC, USA, 27 June-2 July 2004, pp 275–282

  3. Chen YT, Xiong J, Xu WH, Zuo JW (2019) A novel online incremental and decremental learning algorithm based on variable support vector machine. Clust Comput 22:7435–7445

    Article  Google Scholar 

  4. Chen YT, Wang J, Xia RL, Zhang Q, Cao ZH, Yang K (2019) The visual object tracking algorithm research based on adaptive combination kernel. J Ambient Intell Humaniz Comput 10(12):4855–4867

    Article  Google Scholar 

  5. Chen YT, Wang J, Chen X, Zhu MW, Yang K, Wang Z, Xia RL (2019) Single-image super-resolution algorithm based on structural self-similarity and deformation block features. IEEE Access 7:58791–58801

    Article  Google Scholar 

  6. Chen YT, Xu WH, Zuo JW, Yang K (2019) The fire recognition algorithm using dynamic feature fusion and IV-SVM classifier. Clust Comput 22:7665–7675

    Article  Google Scholar 

  7. Dong C, Loy CC, He KM, Tang XO (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  8. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: Proceedings of Annual Conference on Neural Information Processing Systems, Montreal, Quebec, Canada, 8–13 December 2014, pp 5672–2680

  9. He KM, Zhang XY, Ren SQ, Sun J (2016) Deep residual learning for image recognition. In: proceedings of IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27-30 June 2016, pp 770–778

  10. Huang KB, Hu RM, Jiang JJ, Han Z, Wang F (2017) HRM graph constrained dictionary learning for face image super-resolution. Multimed Tools Appl 76:3139–3162

    Article  Google Scholar 

  11. Jiang J, Yu Y, Tang S, Ma J, Aizawa A, Aizawa K (2020) Context-patch based face hallucination via thresholding locality-constrained representation and reproducing learning. IEEE Transactions on Cybernetics 50(1):324–337

    Article  Google Scholar 

  12. Johnson J, Alahi A, Li FF (2016) Perceptual losses for real-time style transfer and super-resolution. In: proceedings of European conference on computer vision, Amsterdam, Netherlands, 11-14 October 2016, pp 694–711

  13. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: proceedings of IEEE conference on computer vision and pattern recognition, Columbus, OH, USA, 23-28 June 2014, pp 1867–1874

  14. Kim J, Kwon Lee J, Mu Lee K (2016) Deeply-recursive convolutional network for image super-resolution. In: proceedings of IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27-30 June 2016, pp 1637–1645

  15. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. ArXiv preprint, arXiv 1412:6980

    Google Scholar 

  16. Le V, Brandt J, Lin Z, Bourdev LD, Huang TS (2012) Interactive facial feature localization. In: proceedings of European conference on computer vision, Florence, Italy, 7-13 October 2012, pp 679–692

  17. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken AP, Tejani A, Totz J, Wang ZH, Shi WZ, et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: proceedings of the IEEE conference on computer vision and pattern recognition, Piscataway, NJ, USA, 21-26 July 2017, pp 4681–4690

  18. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of IEEE conference on computer vision and pattern recognition workshops, Honolulu, HI, USA, 21-26 July 2017, pp 136-–144

  19. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: proceedings of the international conference on computer vision, Santiago, Chile, 7-13 December 2015, pp 3730-3738

  20. Liu L, Chen CLP, Li S, Tang YY, Chen L (2018) Robust face hallucination via locality-constrained bi-layer representation. IEEE Transactions on Cybernetics 48(4):1189–1201

    Article  Google Scholar 

  21. Metropolis N, Ulam S (1949) The Monte Carlo method. J Am Stat Assoc 44:335–341

    Article  Google Scholar 

  22. Rajput SS, Arya KV (2020) A robust face super-resolution algorithm and its application in low-resolution face recognition system. Multimed Tools Appl 79:23909–23934. https://doi.org/10.1007/s11042-020-09072-5

    Article  Google Scholar 

  23. Tai Y, Yang J, Liu XM (2017) Image super-resolution via deep recursive residual network. In: proceedings of IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21-26 July 2017, pp 3147–3155

  24. Thomaz CE, Giraldi GA (2010) A new ranking method for principal components analysis and its application to face image analysis. Image Vis Comput 28(6):902–913

    Article  Google Scholar 

  25. Timofte R, De Smet V, Van Gool L (2013) Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of IEEE conference on computer vision, Sydney, Australia, 1-8 December 2013, pp 1920-1927

  26. Wang F, Jiang MQ, Qian C, Yang S, Li C, Zhang HG, Wang XG, Tang XO (2017) Residual attention network for image classification. In: proceedings of IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21-26 July 2017, pp 6450–6458

  27. Xiang LY, Guo GQ, Yu JM, Sheng VS, Yang P (2020) A convolutional neural network-based linguistic steganalysis for synonym substitution steganography. Math Biosci Eng 17(2):1041–1058

    Article  MathSciNet  Google Scholar 

  28. Yang J, Wright J, Huang TS, Yu L (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  Google Scholar 

  29. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

  30. Zhang KB, Gao XB, Tao DC, Li XL (2012) Single image super-resolution with non-local means and steering kernel regression. IEEE Trans Image Process 21(11):4544–4556

    Article  MathSciNet  Google Scholar 

  31. Zhang H, Goodfellow I, Metaxas D, Odena A (2018) Self-attention generative adversarial networks. ArXiv preprint, arXiv 1705:02438

    Google Scholar 

  32. Zhang H, Goodfellow I, Metaxas D, Odena A (2018) Self-attention generative adversarial networks. ArXiv preprint, arXiv 1805:08318

    Google Scholar 

Download references

Funding

This study was funded by the National Natural Science Foundation of China (Grant number 61972056, 61772454, 61402053, 61981340416), the Hunan Provincial Natural Science Foundation of China (Grant number 2020JJ4623), the Scientific Research Fund of Hunan Provincial Education Department (Grant number 17A007, 19C0028, 19B005), the Changsha Science and Technology Planning (Grant number KQ1703018, KQ1804023, KQ1902007), the Junior Faculty Development Program Project of Changsha University of Science and Technology (Grant number 2019QJCZ011), the “Double First-class” International Cooperation and Development Scientific Research Project of Changsha University of Science and Technology (Grant number 2019IC34), the Practical Innovation and Entrepreneurship Ability Improvement Plan for Professional Degree Postgraduate of Changsha University of Science and Technology (Grant number SJCX202072), the Postgraduate Training Innovation Base Construction Project of Hunan Province (Grant number 2019-248-51, 2020-172-48), the Beidou Micro Project of Hunan Provincial Education Department (Grant number XJT[2020] No.149).

Author information

Authors and Affiliations

  1. School of Computer and Communication Engineering, Changsha University of Science and Technology, Changsha, 410114, Hunan, China

    Yuantao Chen, Volachith Phonevilay, Jiajun Tao & Xi Chen

  2. Hunan Institute of Scientific and Technical Information, Changsha, 411105, Hunan, China

    Runlong Xia & Jingbo Xie

  3. Department of Electronic Products, Hunan ZOOMLION Intelligent Technology Corporation Limited, Changsha, 410005, Hunan, China

    Qian Zhang & Kai Yang

  4. Electronics & Information School, Yangtze University, Jingzhou, 434023, China

    Jie Xiong

Authors
  1. Yuantao Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Volachith Phonevilay
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jiajun Tao
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xi Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Runlong Xia
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Qian Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Kai Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Jie Xiong
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Jingbo Xie
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yuantao Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Code availability

Not applicable.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Phonevilay, V., Tao, J. et al. The face image super-resolution algorithm based on combined representation learning. Multimed Tools Appl 80, 30839–30861 (2021). https://doi.org/10.1007/s11042-020-09969-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09969-1

Keywords