Abstract
Image super-resolution is widely applied in face recognition, video perception, medical imaging, and many other fields. Although significant progress has been made, existing methods remain limited in reconstructing fine-grained texture details, making the pixels of the resulting images coarse. To address this problem, we propose a novel interpolation-based generative adversarial network (GAN) for high-resolution image reconstruction. First, an interpolation algorithm is introduced into the generator to carry out self-interpolation and channel interpolation using advanced features extracted from the low-resolution images. Second, the idea of residuals is introduced into both the generator and discriminator to expand the receptive field of the model and fully exploit the global features of the image jointly improving the visual perception of the resulting super-resolution image. Extensive experiments are conducted to evaluate the performances of the proposed models from two aspects: convergence speed and the resolution improvement effect. The experimental results demonstrate that the proposed model reaches a faster convergence speed with and comparable resolution improvement effect with respect to other state-of-the-art methods.
Similar content being viewed by others
References
Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823
Li S, He F, Du B, Zhang L, Xu Y, Tao D (2019) Fast spatio-temporal residual network for video super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 10522–10531
Caballero J, Ledig C, Aitken A, Acosta A, Totz J, Wang Z, Shi W (2017) Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4778–4787
Ma C, Jiang Z, Rao Y, Lu J, Zhou J (2020) Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5569–5578
Meishvili G, Jenni S, Favaro P (2020) Learning to have an ear for face super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1364–1374
Nasrollahi K, Moeslund TB (2014) Super-resolution: a comprehensive survey. Mach Vis Appl 25(6):1423–1468
Xiang X, Tian Y, Zhang Y, Fu Y, Allebach JP, Xu C (2020) Zooming slow-mo: fast and accurate one-stage space-time video super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3370–3379
Haris M, Shakhnarovich G, Ukita Ni (2019) Recurrent back-projection network for video super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3897–3906
Zheng X, Cai Z (2017) Real-time big data delivery in wireless networks: a case study on video delivery. IEEE Trans Industr Inform 13(4):2048–2057
Hui B, Liu Y, Qiu J, Cao L, Ji L, He Z (2020) Study of texture segmentation and classification for grading small hepatocellular carcinoma based on ct images. Tsinghua Sci Technol, 25(6)
Hui B, Qiu J-J, Liu J-H, Ke N-W (2020) Identification of pancreaticoduodenectomy resection for pancreatic head adenocarcinoma: a preliminary study of radiomics. Comput Math Methods Med, 2020
Liu J, Pan Y, Li M, Chen Z, Tang L, Lu C, Wang J (2018) Applications of deep learning to mri images: a survey. Big Data Mining Analytics 1(1):1
Yu Y, Li M, Liu L, Li Y, Wang J (2019) Clinical big data and deep learning: applications, challenges, and future outlooks. Big Data Mining Analytics 2(4):288
Haut JM, Fernandez-Beltran R, Paoletti ME, Plaza J, Plaza A, Pla F (2018) A new deep generative network for unsupervised remote sensing single-image super-resolution. IEEE Trans Geosci Remote Sens 56(11):6792–6810
Jiang K, Wang Z, Yi P, Wang G, Tao L, Jiang J (2019) Edge-enhanced gan for remote sensing image superresolution. IEEE Trans Geosci Remote Sens 57(8):5799–5812
Honma M, Akiyama K, Uemura M, Ikeda S (2014) Super-resolution imaging with radio interferometry using sparse modeling. Publ Astronom Soc Japan 66(5):95
Harmeling S, Sra S, Hirsch M, Schölkopf B (2010) Multiframe blind deconvolution, super-resolution, and saturation correction via incremental em. In: 2010 IEEE International conference on image processing. IEEE, pp 3313–3316
Yuan X, Guo S, Li C, Lu B, Lou S (2019) Near infrared star centroid detection by area analysis of multi-scale super pixel saliency fusion map. Tsinghua Sci Technol 24(03):291
Zheng X, Cai Z (2020) Privacy-preserved data sharing towards multiple parties in industrial iots. IEEE J Selected Areas Commun 38(5):968–979
Cai Z, Zheng X (2018) A private and efficient mechanism for data uploading in smart cyber-physical systems. IEEE Transactions on Network Science and Engineering
Liang Y, Cai Z, Yu J, Han Q, Li Y (2018) Deep learning based inference of private information using embedded sensors in smart devices. IEEE Netw 32(4):8–14
Li K, Lu G, Luo G, Cai Z (2017) Seed-free graph de-anonymiztiation with adversarial learning. In: The 29th ACM International conference on information and knowledge management
Cai Z, He Z (2019) Trading private range counting over big iot data. In: 2019 IEEE 39th international conference on distributed computing systems (ICDCS). IEEE, pp 144–153
Cai Z, Zheng X, Jiguo Y (2019) A differential-private framework for urban traffic flows estimation via taxi companies. IEEE Trans Industr Inform 15(12):6492–6499
Hui B, Liu L, Chen J, Zhou X, Nian Y (2020) Few-shot relation classification by context attention-based prototypical networks with bert. EURASIP J Wirel Commun Netw 2020(1):1–17
Yang J, Wright J, Huang TS, Yi M (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873
Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307
Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654
Yi L, Cai Z, Jiguo Y, Han Q, Li Y (2018) Deep learning based inference of private information using embedded sensors in smart devices. IEEE Netw 32(4):8–14
Guo Y, Ma H (2019) Image blind deblurring using an adaptive patch prior. Tsinghua Sci Technol 24(2):238
Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690
Wang X, Ke Y, Shixiang W, Jinjin G, Liu Y, Dong C, Qiao Y, Loy CC (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV), pp 0–0
Xiong Z, Li W, Han Q, Cai Z (2019) Privacy-preserving auto-driving: a gan-based approach to protect vehicular camera data. In: 2019 IEEE International conference on data mining (ICDM). IEEE, pp 668–677
Kim SY, Jihyong O, Kim M (2020) Jsi-gan: Gan-based joint super-resolution and inverse tone-mapping with pixel-wise task-specific filters for uhd hdr video. In: AAAI, pp 11287–11295
Chang H, Yeung D-Y, Xiong Y (2004) Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004, vol 1. IEEE, pp I–I
Timofte R, De Smet V, Van Gool L (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: Asian conference on computer vision. Springer, pp 111–126
Salvador J, Perez-Pellitero E (2015) Naive Bayes super-resolution forest. In: Proceedings of the IEEE international conference on computer vision, pp 325–333
Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European conference on computer vision. Springer, pp 391–407
Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645
Tai Y, Yang Ji, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3147–3155
Goodfellow I, Pouget-Abadie J, Mirza M, Bing X, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680
Bruna J, Sprechmann P, LeCun Y (2015) Super-resolution with deep convolutional sufficient statistics. arXiv:1511.05666
Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Advances in neural information processing systems, pp 262–270
Chu M, Xie Y, Leal-Taixé L, Thuerey N (2018) Temporally coherent gans for video super-resolution (tecogan). arXiv:1811.09393 1(2):3
Arjovsky M, Chintala S, Bottou L (2017) Wasserstein gan. arXiv:1701.07875
Zheng H, Ji M, Wang H, Liu Y, Lu F (2018) Crossnet: an end-to-end reference-based super resolution network using cross-scale warping. In: Proceedings of the European conference on computer vision (ECCV), pp 88–104
Zhang Z, Wang Z, Lin Z, Qi H (2019) Image super-resolution by neural texture transfer. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7982–7991
Yi S, Wang X, Tang X (2015) Deeply learned face representations are sparse, selective, and robust. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pps 2892–2900
Timofte R, De Smet V, Van Gool Luc (2013) Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE international conference on computer vision, pp 1920–1927
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167
Burger HC, Schuler CJ, Harmeling S (2012) Image denoising: can plain neural networks compete with bm3d? In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2392–2399
Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: International conference on curves and surfaces. Springer, pp 711–730
Dang-Nguyen D-T, Pasquini C, Conotter V, Boato G (2015) Raise: a raw images dataset for digital image forensics. In: Proceedings of the 6th ACM multimedia systems conference, pp 219–224
Funding
This work was supported by the National Key R&D Program of China (No. 2018YFC0807500), by National Natural Science Foundation of China (No. U19A2059), and by Ministry of Science and Technology of Sichuan Province Program (No. 20ZDYF0343, 2018GZDZX0048).
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, L., Zhang, W., Lu, G. et al. Feature-level interpolation-based GAN for image super-resolution. Pers Ubiquit Comput 26, 995–1010 (2022). https://doi.org/10.1007/s00779-020-01488-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00779-020-01488-y