Skip to main content
SpringerLink
Log in

Feature-level interpolation-based GAN for image super-resolution

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Image super-resolution is widely applied in face recognition, video perception, medical imaging, and many other fields. Although significant progress has been made, existing methods remain limited in reconstructing fine-grained texture details, making the pixels of the resulting images coarse. To address this problem, we propose a novel interpolation-based generative adversarial network (GAN) for high-resolution image reconstruction. First, an interpolation algorithm is introduced into the generator to carry out self-interpolation and channel interpolation using advanced features extracted from the low-resolution images. Second, the idea of residuals is introduced into both the generator and discriminator to expand the receptive field of the model and fully exploit the global features of the image jointly improving the visual perception of the resulting super-resolution image. Extensive experiments are conducted to evaluate the performances of the proposed models from two aspects: convergence speed and the resolution improvement effect. The experimental results demonstrate that the proposed model reaches a faster convergence speed with and comparable resolution improvement effect with respect to other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

  2. Li S, He F, Du B, Zhang L, Xu Y, Tao D (2019) Fast spatio-temporal residual network for video super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 10522–10531

  3. Caballero J, Ledig C, Aitken A, Acosta A, Totz J, Wang Z, Shi W (2017) Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4778–4787

  4. Ma C, Jiang Z, Rao Y, Lu J, Zhou J (2020) Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5569–5578

  5. Meishvili G, Jenni S, Favaro P (2020) Learning to have an ear for face super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1364–1374

  6. Nasrollahi K, Moeslund TB (2014) Super-resolution: a comprehensive survey. Mach Vis Appl 25(6):1423–1468

    Article  Google Scholar 

  7. Xiang X, Tian Y, Zhang Y, Fu Y, Allebach JP, Xu C (2020) Zooming slow-mo: fast and accurate one-stage space-time video super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3370–3379

  8. Haris M, Shakhnarovich G, Ukita Ni (2019) Recurrent back-projection network for video super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3897–3906

  9. Zheng X, Cai Z (2017) Real-time big data delivery in wireless networks: a case study on video delivery. IEEE Trans Industr Inform 13(4):2048–2057

    Article  Google Scholar 

  10. Hui B, Liu Y, Qiu J, Cao L, Ji L, He Z (2020) Study of texture segmentation and classification for grading small hepatocellular carcinoma based on ct images. Tsinghua Sci Technol, 25(6)

  11. Hui B, Qiu J-J, Liu J-H, Ke N-W (2020) Identification of pancreaticoduodenectomy resection for pancreatic head adenocarcinoma: a preliminary study of radiomics. Comput Math Methods Med, 2020

  12. Liu J, Pan Y, Li M, Chen Z, Tang L, Lu C, Wang J (2018) Applications of deep learning to mri images: a survey. Big Data Mining Analytics 1(1):1

    Article  Google Scholar 

  13. Yu Y, Li M, Liu L, Li Y, Wang J (2019) Clinical big data and deep learning: applications, challenges, and future outlooks. Big Data Mining Analytics 2(4):288

    Article  Google Scholar 

  14. Haut JM, Fernandez-Beltran R, Paoletti ME, Plaza J, Plaza A, Pla F (2018) A new deep generative network for unsupervised remote sensing single-image super-resolution. IEEE Trans Geosci Remote Sens 56(11):6792–6810

    Article  Google Scholar 

  15. Jiang K, Wang Z, Yi P, Wang G, Tao L, Jiang J (2019) Edge-enhanced gan for remote sensing image superresolution. IEEE Trans Geosci Remote Sens 57(8):5799–5812

    Article  Google Scholar 

  16. Honma M, Akiyama K, Uemura M, Ikeda S (2014) Super-resolution imaging with radio interferometry using sparse modeling. Publ Astronom Soc Japan 66(5):95

    Google Scholar 

  17. Harmeling S, Sra S, Hirsch M, Schölkopf B (2010) Multiframe blind deconvolution, super-resolution, and saturation correction via incremental em. In: 2010 IEEE International conference on image processing. IEEE, pp 3313–3316

  18. Yuan X, Guo S, Li C, Lu B, Lou S (2019) Near infrared star centroid detection by area analysis of multi-scale super pixel saliency fusion map. Tsinghua Sci Technol 24(03):291

    Article  Google Scholar 

  19. Zheng X, Cai Z (2020) Privacy-preserved data sharing towards multiple parties in industrial iots. IEEE J Selected Areas Commun 38(5):968–979

    Article  Google Scholar 

  20. Cai Z, Zheng X (2018) A private and efficient mechanism for data uploading in smart cyber-physical systems. IEEE Transactions on Network Science and Engineering

  21. Liang Y, Cai Z, Yu J, Han Q, Li Y (2018) Deep learning based inference of private information using embedded sensors in smart devices. IEEE Netw 32(4):8–14

    Article  Google Scholar 

  22. Li K, Lu G, Luo G, Cai Z (2017) Seed-free graph de-anonymiztiation with adversarial learning. In: The 29th ACM International conference on information and knowledge management

  23. Cai Z, He Z (2019) Trading private range counting over big iot data. In: 2019 IEEE 39th international conference on distributed computing systems (ICDCS). IEEE, pp 144–153

  24. Cai Z, Zheng X, Jiguo Y (2019) A differential-private framework for urban traffic flows estimation via taxi companies. IEEE Trans Industr Inform 15(12):6492–6499

    Article  Google Scholar 

  25. Hui B, Liu L, Chen J, Zhou X, Nian Y (2020) Few-shot relation classification by context attention-based prototypical networks with bert. EURASIP J Wirel Commun Netw 2020(1):1–17

    Article  Google Scholar 

  26. Yang J, Wright J, Huang TS, Yi M (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  Google Scholar 

  27. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  28. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  29. Yi L, Cai Z, Jiguo Y, Han Q, Li Y (2018) Deep learning based inference of private information using embedded sensors in smart devices. IEEE Netw 32(4):8–14

    Article  Google Scholar 

  30. Guo Y, Ma H (2019) Image blind deblurring using an adaptive patch prior. Tsinghua Sci Technol 24(2):238

    Article  Google Scholar 

  31. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  32. Wang X, Ke Y, Shixiang W, Jinjin G, Liu Y, Dong C, Qiao Y, Loy CC (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV), pp 0–0

  33. Xiong Z, Li W, Han Q, Cai Z (2019) Privacy-preserving auto-driving: a gan-based approach to protect vehicular camera data. In: 2019 IEEE International conference on data mining (ICDM). IEEE, pp 668–677

  34. Kim SY, Jihyong O, Kim M (2020) Jsi-gan: Gan-based joint super-resolution and inverse tone-mapping with pixel-wise task-specific filters for uhd hdr video. In: AAAI, pp 11287–11295

  35. Chang H, Yeung D-Y, Xiong Y (2004) Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004, vol 1. IEEE, pp I–I

  36. Timofte R, De Smet V, Van Gool L (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: Asian conference on computer vision. Springer, pp 111–126

  37. Salvador J, Perez-Pellitero E (2015) Naive Bayes super-resolution forest. In: Proceedings of the IEEE international conference on computer vision, pp 325–333

  38. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European conference on computer vision. Springer, pp 391–407

  39. Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  40. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  41. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645

  42. Tai Y, Yang Ji, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3147–3155

  43. Goodfellow I, Pouget-Abadie J, Mirza M, Bing X, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  44. Bruna J, Sprechmann P, LeCun Y (2015) Super-resolution with deep convolutional sufficient statistics. arXiv:1511.05666

  45. Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Advances in neural information processing systems, pp 262–270

  46. Chu M, Xie Y, Leal-Taixé L, Thuerey N (2018) Temporally coherent gans for video super-resolution (tecogan). arXiv:1811.09393 1(2):3

    Google Scholar 

  47. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein gan. arXiv:1701.07875

  48. Zheng H, Ji M, Wang H, Liu Y, Lu F (2018) Crossnet: an end-to-end reference-based super resolution network using cross-scale warping. In: Proceedings of the European conference on computer vision (ECCV), pp 88–104

  49. Zhang Z, Wang Z, Lin Z, Qi H (2019) Image super-resolution by neural texture transfer. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7982–7991

  50. Yi S, Wang X, Tang X (2015) Deeply learned face representations are sparse, selective, and robust. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pps 2892–2900

  51. Timofte R, De Smet V, Van Gool Luc (2013) Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE international conference on computer vision, pp 1920–1927

  52. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167

  53. Burger HC, Schuler CJ, Harmeling S (2012) Image denoising: can plain neural networks compete with bm3d? In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2392–2399

  54. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: International conference on curves and surfaces. Springer, pp 711–730

  55. Dang-Nguyen D-T, Pasquini C, Conotter V, Boato G (2015) Raise: a raw images dataset for digital image forensics. In: Proceedings of the 6th ACM multimedia systems conference, pp 219–224

Download references

Funding

This work was supported by the National Key R&D Program of China (No. 2018YFC0807500), by National Natural Science Foundation of China (No. U19A2059), and by Ministry of Science and Technology of Sichuan Province Program (No. 20ZDYF0343, 2018GZDZX0048).

Author information

Authors and Affiliations

  1. University of Electronic Science and Technology of China, Chengdu, China

    Lizong Zhang, Wei Zhang, Guoming Lu & Pengcheng Yang

  2. Trusted Cloud Computing and Big Data Key Laboratory of Sichuan Province, Chengdu, China

    Lizong Zhang

  3. China Electronic Technology Cyber Security Co., Ltd., Chengdu, 610000, China

    Zhihong Rao

Authors
  1. Lizong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guoming Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pengcheng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhihong Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Lizong Zhang or Guoming Lu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Zhang, W., Lu, G. et al. Feature-level interpolation-based GAN for image super-resolution. Pers Ubiquit Comput 26, 995–1010 (2022). https://doi.org/10.1007/s00779-020-01488-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-020-01488-y

Keywords

Search

Navigation