Skip to main content
SpringerLink
Log in

Facial image super-resolution using progressive network interleaved correlation filter

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The very low resolution (VLR) issue arises in many face application systems because of the rising demand for surveillance camera-based applications. On the VLR face image, the face recognition algorithms in use are unable to work satisfactorily. Although face super-resolution (SR) techniques can be used to improve the resolution of the images, these VLR face images do not respond well to the existing learning-based face SR techniques. This work suggests a novel method for learning the link between the high-resolution image space and the VLR image space for face SR to solve this issue. The paper aims to address the problem of face image super-resolution and solutions for resolving the issues related to facial images. The major issues related to face image super-resolution during reconstruction end is a fusion of more than one kind of feature which may cause noise in the image, blind spot generation, low perceptual quality, and checkboard issues. The existing models are designed to improvise the perceptual quality of face super-resolution but still fail to generate better perceptual quality of image due to loss during the reconstruction stage. Therefore, this paper proposes a novel progressive face hallucination super-resolution (FHSR) model with a loss-aware upscaling network layer. The upscaling layer is integrated with correlation filters and used a combined loss function. The model is designed as cascading of upscaling modules that are progressive with dense skip connection layers. Loss functions from each upscaling module are evaluated and cascaded together and fed forwards to the next layer to minimize the reconstruction losses. The architecture upscaled the images with 3 cascading modules which magnify 2x,4x, and 8x. The paper also presented the ablation study that assured that the designed FHSR model is better as compared to baseline models. This architecture overcomes the issue of extraction of more features with minimum losses and constructs the high magnification at the reconstruction end. The comparative study shows enhanced performance over state-of-art models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ahn N et al. (2018) “Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network”, in ECCV

  2. Anari V et al (2019) “A Sparse Analysis-Based Single Image Super-Resolution”, in Computers.

  3. Anari V et al. (2019) “A Sparse Analysis-Based Single Image Super-Resolution”, in Computers. 8, 41. https://doi.org/10.3390/computers8020041

  4. Anwar S, Khan S, Barnes N (2019) “A deep journey into super-resolution: a survey”, in ArXiv abs/1904.07523

  5. Aodha OM et al. (2012) “Patch Based Synthesis for Single Depth Image Super-Resolution”, in Springer-Verlag Berlin Heidelberg, pp. 71–84

  6. Baker S, Kanade T (2000) ‘Hallucinating faces. Proc. Fourth IEEE Int. Conf. on Automatic Face and Gesture Recognition, Grenoble, France, pp. 83–88

  7. Choi J-H et al. (2019) “Deep learning-based image super-resolution considering quantitative and perceptual quality”, in neuro computing Elsevier

  8. Dong C, Loy CC, He K, Tang X (2014) “Learning a deep convolutional network for image super-resolution”, Proc. Eur. Conf. Comput. Vis. (ECCV). pp. 184–199

  9. Gao H, Xiao J, Yin Y, Liu T, Shi J (2022) "A Mutually Supervised Graph Attention Network for Few-Shot Segmentation: The Perspective of Fully Utilizing Limited Samples," in IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2022.3155486

  10. Gao H, Qiu B, Duran Barroso RJ, Hussain W, Xu Y, Wang X (2022) "TSMAE: A Novel Anomaly Detection Approach for Internet of Things Time Series Data Using Memory-Augmented Autoencoder," in IEEE Transactions on Network Science and Engineering. https://doi.org/10.1109/TNSE.2022.3163144

  11. Gao H, Huang J, Tao Y, Hussain W, Huang Y (2022) "The joint method of triple attention and novel loss function for entity relation extraction in small data-driven computational social systems," In: IEEE transactions on computational social systems. https://doi.org/10.1109/TCSS.2022.3178416

  12. Gao H, Xu K, Cao M, Xiao J, Xu Q, Yin Y (Feb. 2022) The deep features and attention mechanism-based method to dish healthcare under social IoT systems: an empirical study with a hand-deep local–global net. EEE Transac Comput Soc Syst 9(1):336–347. https://doi.org/10.1109/TCSS.2021.3102591

    Article  Google Scholar 

  13. Gao H, et al. (2022) "A Novel GAPG Approach to Automatic Property Generation for Formal Verification: The GAN Perspective." ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)

  14. Grm K (2020) “Face Hallucination Using Cascaded Super-Resolution and Identity Priors”, IEEE transactions on image processing, vol. 29

  15. Grm K, Scheirer WJ (2020) “Face Hallucination Using Cascaded Super-Resolution and Identity Priors”, IEEE Transactions on image processing, vol. 29

  16. Haris M, Shakhnarovich G, Ukita N (2018) "Task-driven super-resolution: Object detection in low-resolution images", Arxiv:1803.11316

  17. He K et al. (2015) “Deep Residual Learning for Image Recognition”, in arXiv:1512.03385v1 Cscv

  18. Huang G et al. (2018) “Densely Connected Convolutional Networks”, in arxiv

  19. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. Proc Eur Conf Comput Vis (ECCV):694–711

  20. Kim J, Lee JK, Lee KM (2016) “Accurate image super-resolution using very deep convolution networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 1646–1654

  21. Kim J. et al. (2016) “Accurate Image Super-Resolution Using Very Deep Convolutional Networks”, in arXiv:1511.04587v2 [cs.CV]

  22. Kim JS, Ko K, Kim C-S (2021) Gluing reference patches together for face super-resolution. IEEE Access 9:169321–169334. https://doi.org/10.1109/ACCESS.2021.3138442

    Article  Google Scholar 

  23. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Fast and accurate image super-resolution with deep Laplacian pyramid networks. IEEE Trans Pattern Anal Mach Intell 41:2599–2613

    Article  Google Scholar 

  24. Le, V., Brandt, J., Lin, Z., et al. (2012) 'Interactive facial feature localization, in European Conf. on Computer Vision, Florence, Italy. pp. 679–692

  25. Ledig C et al. (2017) “Photo-realistic single image super-resolution using a generative adversarial network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 4681–4690

  26. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 136–144

  27. Liu C, Shum HY, Freeman WT (2007) Face hallucination: theory and practice. Int J Comput Vis 75(1):115–134

    Article  Google Scholar 

  28. Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) Sphere face: deep hypersphere embedding for face recognition

  29. Luo S, Lu J (2022) GFNet: a gradient information compensation-based face super-resolution network. IEEE Access 10:8073–8080. https://doi.org/10.1109/ACCESS.2022.3143499

    Article  Google Scholar 

  30. Seif G et al. (2018) “Edge-based loss function for single image super-resolution”, in ©2018 IEEE

  31. Thasarathan KNH (2019) “Edge-Informed Single Image Super-Resolution”, in ICCV

  32. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  33. Wang Y, Xie Z, Xu K, Dou Y, Lei Y (2016) An efficient and effective convolutional auto-encoder extreme learning machine network for 3d feature learning. Neurocomputing 174:988–998

    Article  Google Scholar 

  34. Wang X et al. (2019) “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks”, in springer nature Switzerland ag

  35. Wang Z et al. (2020) “Deep Learning for Image Super-resolution A Survey”, in arXiv:1902.06068v2 [cs.CV]

  36. Xin Y, Fernando B, Ghanem B, Porikli F, Hartley R (2018) Face super-resolution guided by facial component heatmaps. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 217–233

  37. Xue S et al. (2019) “Faster image super-resolution by improved frequency-domain neural networks”, In: Signal, Image and Video Processing, Springer

  38. Yu X, Porikli F (2016) “Ultra-resolving face images by discriminative generative networks”, In: European Conf. on Computer Vision, Amsterdam, Netherlands. pp. 318–333

  39. Yu X., et al. (2017) “Imagining the Unimaginable Faces by Deconvolutional Networks”, in Journal of Latex Class Files

  40. Zhang K et al. (2018) “Super-identity convolutional neural network for face hallucination,” in Proc. Eur. Conf. Comput. Vis. (ECCV). pp. 183–198

  41. Zhang Q, Yang G, Zhang G (2022) "Collaborative Network for Super-Resolution and Semantic Segmentation of Remote Sensing Images," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-12, Art no 4404512. https://doi.org/10.1109/TGRS.2021.3099300

  42. Zhu X (2017) “Image super-resolution based on sparse representation via direction and edge dictionaries”, In: Hindawi mathematical problems in engineering, volume

Download references

Author information

Authors and Affiliations

  1. Maulana Azad National Institute of Technology, Bhopal, India

    Ajay Sharma

  2. Electronics Department, MANIT, Bhopal, India

    Bhavana P Shrivastava

Authors
  1. Ajay Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bhavana P Shrivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ajay Sharma.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, A., Shrivastava, B.P. Facial image super-resolution using progressive network interleaved correlation filter. Multimed Tools Appl 82, 29587–29606 (2023). https://doi.org/10.1007/s11042-023-14765-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14765-8

Keywords

Search

Navigation