Skip to main content
SpringerLink
Log in

Face super-resolution via nonlinear adaptive representation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Face super-resolution is an example of super-resolution technique, where it takes one or multiple observed low-resolution images and then converts them to high-resolution image. Learning-based face super-resolution depends on prior information from training database. Most patch-based face super-resolution methods assume the homoscedasticity of the reconstruction error in an objective function and solve it with regularized least squares. In fact, the heteroscedasticity generally exists both in the observed data and in the reconstruction error. To access accurate prior information, we propose a nonlinear adaptive representation (NAR) scheme for hallucinating the individuality of facial images. First, we apply a weighted regularization process to both the reconstruction error and representation coefficients terms to eliminate the heteroscedasticity of the input data. Then, the contextual patches and residual high-frequency components are explored to enrich the prior information. Moreover, a nonlinear extension of the adaptive representation fully utilizes accurate prior information to achieve better reconstruction performance. Experiments on the CAS-PEAL-R1, Webface and LDHF databases show that NAR outperforms some state-of-the-art face super-resolution methods including some deep learning-based approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Baker S, Kanade T (2000) Hallucinating faces. In: IEEE international conference on automatic face and gesture recognition, p 83

  2. Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004, pp I–I

  3. Chen Y, Li F, Fan J (2015) Mining association rules in big data with NGNP. Cluster Comput 18(2):577–585

    Article  Google Scholar 

  4. Dong C, Chen CL, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  5. Gao G, Jing XY, Huang P, Zhou Q, Wu S, Yue D (2016) Locality-constrained double low-rank representation for effective face hallucination. IEEE Access 4(99):8775–8786

    Article  Google Scholar 

  6. Gao W, Cao B, Shan S, Chen X, Zhou D, Zhang X, Zhao D (2008) The cas-peal large-scale chinese face database and baseline evaluations. IEEE Trans Syst Man Cybern A Syst Hum 38(1):149–161

    Article  Google Scholar 

  7. Huang Z, Li Q, Zhang T, Sang N, Hong H (2017) Iterative weighted sparse representation for X-ray cardiovascular angiogram image denoising over learned dictionary. IET Image Proc 12(2):254–261

    Article  Google Scholar 

  8. Huang Z, Zhang Y, Li Q, Zhang T, Sang N (2018) Spatially adaptive denoising for X-ray cardiovascular angiogram images. Biomed Signal Process Control 40:131–139

    Article  Google Scholar 

  9. Jiang J, Chen C, Huang K, Cai Z, Hu R (2016) Noise robust position-patch based face super-resolution via Tikhonov regularized neighbor representation. Inf Sci 367(C):354–372

    Article  Google Scholar 

  10. Jiang J, Hu R, Han Z, Lu T, Huang K (2012) Position-patch based face hallucination via locality-constrained representation. In: IEEE international conference on multimedia and expo, pp 212–217

  11. Jiang J, Hu R, Han Z, Wang Z, Lu T, Chen J (2013) Locality-constraint iterative neighbor embedding for face hallucination. In: IEEE international conference on multimedia and expo, pp 1–6

  12. Jung C, Jiao L, Liu B, Gong M (2011) Position-patch based face hallucination using convex optimization. IEEE Signal Process Lett 18(6):367–370

    Article  Google Scholar 

  13. Kang D, Han H, Jain AK, Lee SW (2014) Nighttime face recognition at large standoff: cross-distance and cross-spectral matching. Pattern Recogn 47(12):3750–3766

    Article  Google Scholar 

  14. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Computer vision and pattern recognition, pp 1646–1654

  15. Lan C, Hu R, Han Z et al (2010) A face super-resolution approach using shape semantic mode regularization. In: 2010 IEEE international conference on image processing. IEEE, pp 2021–2024

  16. Lanckriet GRG, Cristianini N, Bartlett P, Ghaoui LE, Jordan MI (2002) Learning the kernel matrix with semidefinite programming. J Mach Learn Res 5(1):27–72

    MathSciNet  MATH  Google Scholar 

  17. Lipsitz SR, Laird NM, Harrington DP (1994) Weighted least squares analysis of repeated categorical measurements with outcomes subject to nonresponse. Biometrics 50(1):11–24

    Article  MATH  Google Scholar 

  18. Liu C, Shum HY, Zhang CS (2001) A two-step approach to hallucinating faces: global parametric model and local nonparametric model. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, 2001. CVPR 2001, vol 1, pp I–192–I–198

  19. Liu J, Wang X, Chen M, Liu S, Shao Z, Zhou X, Liu P (2014) Illumination and contrast balancing for remote sensing images. Remote Sens 6(2):1102–1123

    Article  Google Scholar 

  20. Liu X, Xu Q, Ma J, Jin H, Zhang Y (2014) MsLRR: a unified multiscale low-rank representation for image segmentation. IEEE Trans Image Process 23(5):2159–2167

    Article  MathSciNet  MATH  Google Scholar 

  21. Lu T, Hu R, Han Z, Jiang J, Zhang Y (2013) From local representation to global face hallucination: a novel super-resolution method by nonnegative feature transformation. In: Visual communications and image processing, pp 1–6

  22. Lu T, Pan L, Wang J, Zhang Y, Wang Z, Xiong Z (2017) AWCR: adaptive and weighted collaborative representations for face super-resolution with context residual-learning. In: Pacific rim conference on multimedia, pp 107–116

  23. Ma J, Zhao J, Jiang J, Zhou H, Guo X (2019) Locality preserving matching. Int J Comput Vis 127(5):512–531

    Article  MathSciNet  Google Scholar 

  24. Ma X, Zhang J, Qi C (2010) Hallucinating face by position-patch. Pattern Recogn 43(6):2224–2236

    Article  Google Scholar 

  25. Peng L, Zhang Y, Zhou H, Lu T (2018) A robust method for estimating image geometry with local structure constraint. IEEE Access 6:20734–20747

    Article  Google Scholar 

  26. Romano Y, Elad M (2016) Con-patch: when a patch meets its context. IEEE Trans Image Process 25(9):3967–3978

    Article  MathSciNet  MATH  Google Scholar 

  27. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–6

    Article  Google Scholar 

  28. Saitoh S (2003) Theory of reproducing kernels. Trans Am Math Soc 68(3):337–404

    MathSciNet  MATH  Google Scholar 

  29. Shao Z, Chen M, Liu C (2015) Feature matching for illumination variation images. J Electron Imaging 24(3):033011

    Article  Google Scholar 

  30. Shi J, Liu X, Zong Y, Qi C, Zhao G (2018) Hallucinating face image by regularization models in high-resolution feature space. IEEE Trans Image Process PP(99):1–1

    MathSciNet  MATH  Google Scholar 

  31. Shi W, Caballero J, Huszár F et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  32. Tikhonov AN, Arsenin VY (1977) Solution of ill-posed problems. Math Comput 32(144):491–491

    Google Scholar 

  33. Timofte R, De V, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: IEEE international conference on computer vision, pp 1920–1927

  34. Timofte R, Smet VD, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. Springer, Berlin

    Google Scholar 

  35. Timofte R, Van Gool L (2012) Weighted collaborative representation and classification of images. In: International conference on pattern recognition, pp 1606–1610

  36. Wang N, Tao D, Gao X, Li X, Li J (2014) A comprehensive survey to face hallucination. Int J Comput Vis 106(1):9–30

    Article  Google Scholar 

  37. Wang X, Tang X (2005) Hallucinating face by eigentransformation. IEEE Press, Piscataway

    Book  Google Scholar 

  38. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  39. Wang Z, Hu R, Jiang J, Han Z, Shao Z (2016) Heteroskedasticity tuned mixed-norm sparse regularization for face hallucination. Multimed Tools Appl 75(24):1–29

    Article  Google Scholar 

  40. Wang Z, Hu R, Wang S, Jiang J (2014) Face hallucination via weighted adaptive sparse regularization. IEEE Trans Circuits Syst Video Technol 24(5):802–813

    Article  Google Scholar 

  41. Wang Z, Liu D, Yang J, Han W, Huang T (2016) Deep networks for image super-resolution with sparse prior. In: IEEE international conference on computer vision, pp 370–378

  42. Waqas J, Yi Z, Zhang L (2013) Collaborative neighbor representation based classification using l 2-minimization approach. Pattern Recogn Lett 34(2):201–208

    Article  Google Scholar 

  43. Yang J, Wright J, Huang T et al (2008) Image super-resolution as sparse representation of raw image patches. In: 2008 IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8

  44. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  45. Yi D, Lei Z, Liao S et al (2014) Learning face representation from scratch. arXiv preprint. arXiv:1411.7923

  46. Yu Y, Tang S, Aizawa K, Aizawa A (2018) Category-based deep CCA for fine-grained venue discovery from multimodal data. IEEE Trans Neural Netw Learn Syst 30(4):1250–1258

    Article  MathSciNet  Google Scholar 

  47. Zeng K, Lu T, Liang X et al (2019) Face super-resolution via bilayer contextual representation. Signal Process Image Commun 75:147–157

    Article  Google Scholar 

  48. Zhang L, Zhou WD, Chang PC, Liu J, Yan Z, Wang T, Li FZ (2015) Kernel sparse representation-based classifier. Multimed Tools Appl 74(1):123–137

    Article  MATH  Google Scholar 

  49. Zhang Y, Zhang Z, Hu G, Hancock ER (2017) Face image super-resolution via weighted patches regression. In: International conference on pattern recognition, pp 3892–3897

  50. Zhao T, Wang Y, Wang H, Sheng G, Song H (2009) Face recognition method by using large and representative datasets. In: Control and decision conference, 2009. CCDC’09. Chinese

  51. Zhou H, Hu J, Lam KM (2015) Global face reconstruction for face hallucination using orthogonal canonical correlation analysis. In: Asia-Pacific signal and information processing association summit and conference, pp 537–542

  52. Zhou H, Ma J, Yang C, Sun S, Liu R, Zhao J (2016) Nonrigid feature matching for remote sensing images via probabilistic inference with global and local regularizations. IEEE Geosci Remote Sens Lett 13(3):374–378

    Google Scholar 

  53. Zhou H, Ma J, Zhang Y, Yu Z, Ren S, Chen D (2017) Feature guided non-rigid image/surface deformation via moving least squares with manifold regularization. In: 2017 IEEE international conference on multimedia and expo (ICME). IEEE, pp 1063–1068

  54. Zhou L, Wang Z, Luo Y et al (2019) Separability and compactness network for image recognition and superresolution. IEEE Trans Neural Netw Learn Syst 30(11):3275–3286

    Article  Google Scholar 

  55. Zhu S, Liu S, Loy CC et al (2016) Deep cascaded bi-network for face hallucination. In: European conference on computer vision. Springer, Cham, pp 614–630

    Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (61502354, 61501413, 61671332, 41501505), the Natural Science Foundation of Hubei Province of China (2018ZYYD059, 2015CFB451, 2014CFA130, 2012FFA099, 2012FFA134, 2013CF125), Hubei Technology Innovation Project (2019AAA045), Scientific Research Foundation of Wuhan Institute of Technology (K201713). The authors declared that they have no conflicts of interest to this work.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, 430205, China

    Tao Lu, Kangli Zeng & Yanduo Zhang

  2. School of Software, Henan University, Kaifeng, 475004, China

    Shenming Qu

  3. School of Economics and Management, Minjiang University, Fuzhou, 350108, China

    Wei He

Authors
  1. Tao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kangli Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shenming Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanduo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei He.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, T., Zeng, K., Qu, S. et al. Face super-resolution via nonlinear adaptive representation. Neural Comput & Applic 32, 11637–11649 (2020). https://doi.org/10.1007/s00521-019-04652-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04652-5

Keywords

Search

Navigation