Skip to main content
SpringerLink
Log in

A deep attention-based ensemble network for real-time face hallucination

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Face hallucination (FH) aims to reconstruct high-resolution faces from low-resolution face inputs, making it significant to other face-related tasks. Different from general super resolution issue, it often requires facial priors other than general extracted features thus leading to fusion of more than one kind of feature. The existing CNN-based FH methods often fuse different features indiscriminately which may introduce noises. Also the latent relations among different features which may be useful are taken into less consideration. To address the above issues, we propose an end-to-end deep ensemble network which aggregates three extraction sub-nets in attention-based manner. In our ensemble strategy, both relations among different features and inter-dependencies among different channels are dug out through the exploitation of spatial attention and channel attention. And for the diversity of extracted features, we aggregate three different sub-nets, which are the basic sub-net for basic features, the auto-encoder sub-net for facial shape priors and the dense residual attention sub-net for fine-grained texture features. Conducted ablation studies and experimental results show that our method achieves effectiveness not only in PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index) metrics but more importantly in clearer details within both key facial areas and whole range. Also results show that our method achieves real-time hallucinating faces by generating one image in 0.0237s.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. BAKER, S.: Hallucinating faces. In: IEEE international conference on automatic face and gesture recognition, pp 83–88 (2000)

  2. Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)

    Article  MathSciNet  Google Scholar 

  3. Cao, Q., Lin, L., Shi, Y., Liang, X., Li, G.: Attention-aware face hallucination via deep reinforcement learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 690–698 (2017)

  4. Chang, H., Yeung, DY., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), IEEE, vol 1, pp I–I (2004)

  5. Chen, Y., Tai, Y., Liu, X., Shen, C., Yang, J.: Fsrnet: end-to-end learning face super-resolution with facial priors. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2492–2501 (2018)

  6. Chen, Z., Wang, R., Zhang, Z., Wang, H., Xu, L.: Background-foreground interaction for moving object detection in dynamic scenes. Inf. Sci. 483(5), 65–81 (2019)

    Article  Google Scholar 

  7. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  8. Gao, L., Li, X., Song, J., Shen, H.T.: Hierarchical lstms with adaptive attention for visual captioning. IEEE Trans. Pattern Anal. Mach. Intell. 42, 1112–1131 (2019)

    Google Scholar 

  9. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27, 2672–2680 (2014)

    Google Scholar 

  10. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 7132–7141 (2018)

  11. Jaderberg, M., Simonyan, K., Zisserman, A., kavukcuoglu, k: Spatial transformer networks. Adv. Neural Inf. Process. Syst. 28, 2017–2025 (2015)

    Google Scholar 

  12. Jiang, H., Deng, W., Shen, Z.: Surveillance video processing using compressive sensing. Inverse Probl. Imaging 6(2), 201–214 (2012)

    Article  MathSciNet  Google Scholar 

  13. Jourabloo, A., Ye, M., Liu, X., Ren, L.: Pose-invariant face alignment with a single cnn. In: Proceedings of international conference on computer vision(ICCV), pp 3200–3209 (2017)

  14. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 4681–4690 (2017)

  15. Li, M., Sun, Y., Zhang, Z., Yu, J.: A coarse-to-fine face hallucination method by exploiting facial prior knowledge. In: 2018 25th IEEE international conference on image processing (ICIP), IEEE, pp 61–65 (2018a)

  16. Li, X., Liu, M., Ye, Y., Zuo, W., Lin, L., Yang, R.: Learning warped guidance for blind face restoration. In: Proceedings of the European conference on computer vision (ECCV), pp 272–289 (2018b)

  17. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops, pp 136–144 (2017)

  18. Liu, C., Shum, HY., Zhang, CS.: A two-step approach to hallucinating faces: global parametric model and local nonparametric model. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Citeseer, pp 192–198 (2001)

  19. Liu, C., Shum, H.Y., Freeman, W.T.: Face hallucination: theory and practice. Int. J. Comput. Vis. 75(1), 115–134 (2007)

    Article  Google Scholar 

  20. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of international conference on computer vision(ICCV), pp 3730–3738 (2015)

  21. Ma, X., Zhang, J., Qi, C.: Hallucinating face by position-patch. Pattern Recognit. 43(6), 2224–2236 (2010)

    Article  Google Scholar 

  22. Shamsolmoali, P., Zareapoor, M., Wang, R., Jain, D.K., Yang, J.: G-GANISR: gradual generative adversarial network for image super resolution. Neurocomputing 366, 140–153 (2019a)

    Article  Google Scholar 

  23. Shamsolmoali, P., Zareapoor, M., Wang, R., Zhou, H., Yang, J.: A novel deep structure u-net for sea-land segmentation in remote sensing images. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sensing 12(9), 3219–3232 (2019b)

    Article  Google Scholar 

  24. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, AP., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1874–1883 (2016)

  25. Shi, Y., Guanbin, L., Cao, Q., Wang, K., Lin, L.: Face hallucination by attentive sequence optimization with reinforcement learning. IEEE transactions on pattern analysis and machine intelligence (2019)

  26. Song, Y., Zhang, J., He, S., Bao, L., Yang, Q.: Learning to hallucinate face images via component generation and enhancement. In: 26th international joint conference on artificial intelligence (IJCAI 2017), International joint conferences on artificial intelligence, pp 4537–4543 (2017)

  27. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1701–1708 (2014)

  28. Tappen, MF., Liu, C.: A bayesian approach to alignment-based image hallucination. In: Proceedings of the European conference on computer vision (ECCV), Springer, pp 236–249 (2012)

  29. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: Proceedings of international conference on computer vision(ICCV), pp 4799–4807 (2017)

  30. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Adv. Neural Inf. Process. Syst. pp 5998–6008 (2017)

  31. Xin, J., Wang, N., Gao, X., Li, J.: Residual attribute attention network for face image super-resolution. Proceedings of the AAAI conference on artificial intelligence 33, 9054–9061 (2019)

  32. Yang, CY., Liu, S., Yang, MH.: Structured face hallucination. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1099–1106 (2013)

  33. Yu, X., Porikli, F.: Ultra-resolving face images by discriminative generative networks. In: Proceedings of the European conference on computer vision (ECCV), Springer, pp 318–333 (2016)

  34. Yu, X., Porikli, F.: Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3760–3768 (2017)

  35. Yu, X., Fernando, B., Ghanem, B., Porikli, F., Hartley, R.: Face super-resolution guided by facial component heatmaps. In: Proceedings of the European conference on computer vision (ECCV), pp 217–233 (2018)

  36. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp 286–301 (2018)

  37. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: International conference on machine learning, pp 7354–7363 (2019a)

  38. Zhang, Y., Li, K., Li, K., Zhong, B., Fu, Y.: Residual non-local attention networks for image restoration. arXiv preprint arXiv:190310082 (2019b)

  39. Zhu, S., Liu, S., Loy, CC., Tang, X.: Deep cascaded bi-network for face hallucination. In: Proceedings of the European conference on computer vision (ECCV), Springer, pp 614–630 (2016)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under the Grant No.61672246, No.61272068, and the Fundamental Research Funds for the Central Universities, HUST:2016YXMS018.

Author information

Authors and Affiliations

  1. Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China

    Dongdong Liu, Jincai Chen & Zhenxing Huang

  2. School of Compter Science and Technology, Huazhong University of Science and Technology, Wuhan, China

    Jincai Chen & Ping Lu

  3. Beijing Key Laboratory of Mobile Computing and Pervasive Devices, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Ni Zeng

  4. Beijing Institute of Computer Technology and Applications, Beijing, China

    Lin Yang, Haofeng Wang, Jinqiao Kou & Min Wu

Authors
  1. Dongdong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jincai Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenxing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ni Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ping Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Haofeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jinqiao Kou
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Min Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jincai Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, D., Chen, J., Huang, Z. et al. A deep attention-based ensemble network for real-time face hallucination. J Real-Time Image Proc 17, 1927–1937 (2020). https://doi.org/10.1007/s11554-020-01009-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-01009-3

Keywords

Search

Navigation