Skip to main content
SpringerLink
Log in

Deep learning methods in real-time image super-resolution: a survey

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Super-resolution is generally defined as a process to obtain high-resolution images form inputs of low-resolution observations, which has attracted quantity of attention from researchers of image-processing community. In this paper, we aim to analyze, compare, and contrast technical problems, methods, and the performance of super-resolution research, especially real-time super-resolution methods based on deep learning structures. Specifically, we first summarize fundamental problems, perform algorithm categorization, and analyze possible application scenarios that should be considered. Since increasing attention has been drawn in utilizing convolutional neural networks (CNN) or generative adversarial networks (GAN) to predict high-frequency details lost in low- resolution images, we provide a general overview on background technologies and pay special attention to super-resolution methods built on deep learning architectures for real-time super-resolution, which not only produce desirable reconstruction results, but also enlarge possible application scenarios of super resolution to systems like cell phones, drones, and embedding systems. Afterwards, benchmark datasets with descriptions are enumerated, and performance of most representative super-resolution approaches is provided to offer a fair and comparative view on performance of current approaches. Finally, we conclude the paper and suggest ways to improve usage of deep learning methods on real-time image super-resolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Agustsson, E., Timofte, R., Van Gool, L.: Anchored regression networks applied to age estimation and super resolution. In: Proceedings of International Conference on Computer Vision, pp. 1652–1661 (2017)

  2. Ahn, N., Kang, B., Sohn, K.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of European Conference on Computer Vision, pp. 256–272 (2018)

  3. Akae, N., Makihara, Y., Yagi, Y.: Gait recognition using periodic temporal super resolution for low frame-rate videos. In: Proceedings of International Joint Conference on Biometrics, pp. 1–7 (2011)

  4. Alonso-Fernandez, F., Farrugia, R.A., Bigun, J., Fierrez, J., Gonzalez-Sosa, E.: A survey of super-resolution in iris biometrics with evaluation of dictionary-learning. IEEE Access (2018)

  5. Aly, H.A., Dubois, E.: Image up-sampling using total-variation regularization with a new observation model. IEEE Trans. Image Process. 14(10), 1647–1659 (2005)

    Google Scholar 

  6. Belekos, S.P., Galatsanos, N.P., Katsaggelos, A.K.: Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans. Image Process. 19(6), 1451–1464 (2010)

    MathSciNet  MATH  Google Scholar 

  7. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of British Machine Vision Conference (2012)

  8. Bose, N.K., Ahuja, N.A.: Superresolution and noise filtering using moving least squares. IEEE Trans. Image Process. 15(8), 2239–2248 (2006)

    Google Scholar 

  9. Bulat, A., Tzimiropoulos, G.: Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans. In: Proceedings of Computer Vision and Pattern Recognition, pp. 109–117 (2018)

  10. Bulat, A., Yang, J., Tzimiropoulos, G.: To learn image super-resolution, use a gan to learn how to do image degradation first. In: Proceedings of the European Conference on Computer Vision, pp. 185–200 (2018)

  11. Caballero, J., Ledig, C., Aitken, A., Acosta, A., Totz, J., Wang, Z., Shi, W.: Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings of Computer Vision and Pattern Recognition, pp. 2848–2857 (2017)

  12. Cao, Q., Lin, L., Shi, Y., Liang, X., Li, G.: Attention-aware face hallucination via deep reinforcement learning. CoRR abs/1708.03132 (2017)

  13. Capel, D., Zisserman, A.: Automated mosaicing with super-resolution zoom. In: Proceedings of Computer Vision and Pattern Recognition, pp. 885–891 (1998)

  14. Chai, Y., Ren, J., Zhao, H., Li, Y., Ren, J., Murray, P.: Hierarchical and multi-featured fusion for effective gait recognition under variable scenarios. Pattern Anal. Appl. 19(4), 905–917 (2016)

    MathSciNet  Google Scholar 

  15. Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. Proc. Comput. Vis. Pattern Recognit. 1, 275–282 (2004)

    Google Scholar 

  16. Che, T., Li, Y., Jacob, A.P., Bengio, Y., Li, W.: Mode regularized generative adversarial networks. CoRR arxiv:abs/1612.02136 (2016)

  17. Chen, R., Qu, Y., Li, C., Zeng, K., Xie, Y., Li, C.: Single-image super-resolution via joint statistical models-guided deep auto-encoder network. Neural Comput. Appl. pp. 1–11 (2019)

  18. Chen, Y., Tai, Y., Liu, X., Shen, C., Yang, J.: Fsrnet: End-to-end learning face super-resolution with facial priors. In: Proceedings of Computer Vision and Pattern Recognition, pp. 2492–2501 (2018)

  19. Cristóbal, G., Gil, E., Šroubek, F., Flusser, J., Miravet, C., Rodríguez, F.d.B.: Superresolution imaging: a survey of current techniques. In: Advanced signal processing algorithms, architectures, and implementations XVIII, vol. 7074, p. 70740C (2008)

  20. Dai, D., Timofte, R., Van Gool, L.: Jointly optimized regressors for image super-resolution. Comput. Graph. Forum 34(2), 95–104 (2015)

    Google Scholar 

  21. Dai, S., Han, M., Xu, W., Wu, Y., Gong, Y.: Soft edge smoothness prior for alpha channel super resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

  22. Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)

  23. Demirel, H., Anbarjafari, G.: Image resolution enhancement by using discrete and stationary wavelet decomposition. IEEE Trans. Image Process. 20(5), 1458–1460 (2011)

    MathSciNet  MATH  Google Scholar 

  24. Denton, E.L., Chintala, S., Fergus, R., et al.: Deep generative image models using a laplacian pyramid of adversarial networks. In: Proceedings of Advances in neural information processing systems, pp. 1486–1494 (2015)

  25. Dey, N., Li, S., Bermond, K., Heintzmann, R., Curcio, C.A., Ach, T., Gerig, G.: Multi-modal image fusion for multispectral super-resolution in microscopy. In: Proceedings of Medical Imaging 2019: Image Processing, p. 109490D (2019)

  26. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Proceedings of European Conference on Computer Vision, pp. 184–199 (2014)

  27. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Proceedings of European Conference on Computer Vision, pp. 391–407 (2016)

  28. Dong, W., Zhang, L., Shi, G., Wu, X.: Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Trans. Image Process. 20(7), 1838–1857 (2011)

    MathSciNet  MATH  Google Scholar 

  29. Duchon, C.E.: Lanczos filtering in one and two dimensions. J. Appl. Meteorol. 18(8), 1016–1022 (1979)

    Google Scholar 

  30. Fahmy, G.: Super-resolution construction of iris images from a visual low resolution face video. In: National Radio Science Conference, pp. 1–6 (2007)

  31. Fujimoto, A., Ogawa, T., Yamamoto, K., Matsui, Y., Yamasaki, T., Aizawa, K.: Manga109 dataset and creation of metadata. In: Proceedings of International Workshop on Comics Analysis, Processing and Understanding, pp. 2–3 (2016)

  32. Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. CoRR arxiv:abs/1508.06576 (2015)

  33. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)

  34. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: Proceedings of IEEE International Conference on Computer Vision, pp. 349–356 (2009)

  35. Gong, W., Qi, L., Xu, Y.: Privacy-aware multidimensional mobile service quality prediction and recommendation in distributed fog environment. Wireless Communications and Mobile Computing 2018, (2018)

  36. Goodfellow, I.J., Abadie, J.P., Mirza, M., Xu, B., Farley, D.W., Ozair, S., Courville, A.C., Bengio, Y.: Generative adversarial nets. In: Proceedings of Neural Information Processing Systems, pp. 2672–2680 (2014)

  37. Greenspan, H.: Super-resolution in medical imaging. Comput. J. 52(1), 43–63 (2009)

    Google Scholar 

  38. Gu, S., Sang, N., Ma, F.: Fast image super resolution via local regression. In: Proceedings of International Conference on Pattern Recognition, pp. 3128–3131 (2012)

  39. Ha, V.K., Ren, J., Xu, X., Zhao, S., Xie, G., Vargas, V.M.: Deep learning based single image super-resolution: A survey. In: International Conference on Brain Inspired Cognitive Systems, pp. 106–119 (2018)

  40. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018)

  41. Hayat, K.: Multimedia super-resolution via deep learning: a survey. Digital Signal Processing (2018)

  42. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  43. Hong, C., Yu, J., Tao, D., Wang, M.: Image-based three-dimensional human pose recovery by multiview locality-sensitive sparse retrieval. IEEE Trans. Ind. Electr. 62(6), 3742–3751 (2015)

    Google Scholar 

  44. Hong, C., Yu, J., Wan, J., Tao, D., Wang, M.: Multimodal deep autoencoder for human pose recovery. IEEE Trans. Image Process. 24(12), 5659–5670 (2015)

    MathSciNet  MATH  Google Scholar 

  45. Hu, Y., Li, J., Huang, Y., Gao, X.: Channel-wise and spatial feature modulation network for single image super-resolution. CoRR arXiv:abs/1809.11130 (2018)

  46. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

  47. Huang, Y., Shao, L., Frangi, A.F.: Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weakly-supervised joint convolutional sparse coding. arXiv preprint arXiv:1705.02596 (2017)

  48. Hung, K.W., Siu, W.C.: New motion compensation model via frequency classification for fast video super-resolution. In: Proceedings of IEEE International Conference on Image Processing, pp. 1193–1196 (2009)

  49. Ji, H., Fermüller, C.: Robust wavelet-based super-resolution reconstruction: theory and algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 31(4), 649–660 (2009)

    Google Scholar 

  50. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of European Conference on Computer Vision, pp. 694–711 (2016)

  51. Joshi, M.V., Chaudhuri, S., Panuganti, R.: Super-resolution imaging: use of zoom as a cue. Image Vis. Comput. 22(14), 1185–1196 (2004)

    Google Scholar 

  52. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  53. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

  54. Krämer, P., Benois-Pineau, J., Domenger, J.P.: Local object-based super-resolution mosaicing from low-resolution video. Signal Process. 91(8), 1771–1780 (2011)

    MATH  Google Scholar 

  55. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of Advances in neural information processing systems, pp. 1097–1105 (2012)

  56. Lai, W., Huang, J., Ahuja, N., Yang, M.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5835–5843 (2017)

  57. Lai, W., Huang, J., Ahuja, N., Yang, M.: Fast and accurate image super-resolution with deep laplacian pyramid networks. CoRR arXiv:abs/1710.01992 (2017)

  58. Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of Computer Vision and Pattern Recognition (2017)

  59. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Google Scholar 

  60. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 105–114 (2017)

  61. Li, J., Feng, J., Kuo, C.C.J.: Deep convolutional neural network for latent fingerprint enhancement. Signal Process. Image Commun. 60, 52–63 (2018)

    Google Scholar 

  62. Li, K., Zhu, Y., Yang, J., Jiang, J.: Video super-resolution using an adaptive superpixel-guided auto-regressive model. Pattern Recognit. 51, 59–71 (2016)

    Google Scholar 

  63. Li, X., Orchard, M.T.: New edge-directed interpolation. IEEE Trans. Image Process. 10(10), 1521–1527 (2001)

    Google Scholar 

  64. Li, Y., Agustsson, E., Gu, S., Timofte, R., Van Gool, L.: Carn: convolutional anchored regression network for fast and accurate single image super-resolution. In: Proceedings of European Conference on Computer Vision, pp. 166–181 (2018)

  65. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1132–1140 (2017)

  66. Lin, M., Chen, Q., Yan, S.: Network in network. CoRR arXiv:abs/1312.4400 (2013)

  67. Mao, X., Shen, C., Yang, Y.: Image restoration using convolutional auto-encoders with symmetric skip connections. CoRR arXiv:abs/1606.08921 (2016)

  68. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proc. Int. Conf. Comput. Vis. 2, 416–423 (2001)

    Google Scholar 

  69. Metz, L., Poole, B., Pfau, D., Sohl-Dickstein, J.: Unrolled generative adversarial networks. CoRR arXiv:abs/1611.02163 (2016)

  70. Milanfar, P.: Super-resolution imaging. CRC Press, Boca Raton (2010)

    Google Scholar 

  71. Minaee, S., Abdolrashidi, A.: Iris-gan: Learning to generate realistic iris images using convolutional gan. CoRR arXiv:abs/1812.04822 (2018)

  72. Minetto, R., Thome, N., Cord, M., Leite, N.J., Stolfi, J.: Snoopertext: a text detection system for automatic indexing of urban scenes. Comput. Vis. Image Underst. 122, 92–104 (2014)

    Google Scholar 

  73. Nasrollahi, K., Escalera, S., Rasti, P., Anbarjafari, G., Baro, X., Escalante, H.J., Moeslund, T.B.: Deep learning based super-resolution for improved action recognition. In: Proceedings of Image Processing Theory, Tools and Applications, pp. 67–72 (2015)

  74. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)

    Google Scholar 

  75. Nguyen, K., Fookes, C., Sridharan, S., Tistarelli, M., Nixon, M.: Super-resolution for biometrics: a comprehensive survey. Pattern Recognit. 78, 23–42 (2018)

    Google Scholar 

  76. Nguyen, T., Le, T., Vu, H., Phung, D.Q.: Dual discriminator generative adversarial nets. In: Proceedings of Advances in Neural Information Processing Systems, pp. 2670–2680 (2017)

  77. Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20(3), 21–36 (2003)

    Google Scholar 

  78. Patti, A.J., Sezan, M.I., Tekalp, A.M.: Super-resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Trans. Image Process. 6(8), 1064–1076 (1997)

    Google Scholar 

  79. Pérez-Pellitero, E., Salvador, J., Ruiz-Hidalgo, J., Rosenhahn, B.: Psyco: manifold span reduction for super resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1837–1845 (2016)

  80. Petrou, M., Jaward, M.H., Chen, S., Briers, M.: Super-resolution in practice: the complete pipeline from image capture to super-resolved subimage creation using a novel frame selection method. Mach. Vis. Appl. 23(3), 441–459 (2012)

    Google Scholar 

  81. Qi, L., Wang, R., Hu, C., Li, S., He, Q., Xu, X.: Time-aware distributed service recommendation with privacy-preservation. Inf. Sci. 480, 354–364 (2019)

    Google Scholar 

  82. Qi, L., Zhang, X., Dou, W., Hu, C., Yang, C., Chen, J.: A two-stage locality-sensitive hashing based approach for privacy-preserving mobile service recommendation in cross-platform edge environment. Future Gener. Comput. Syst. 88, 636–643 (2018)

    Google Scholar 

  83. Qi, L., Zhang, X., Dou, W., Ni, Q.: A distributed locality-sensitive hashing-based approach for cloud service recommendation from multi-source data. IEEE J. Sel. Areas Commun. 35(11), 2616–2624 (2017)

    Google Scholar 

  84. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR arXiv:abs/1511.06434 (2015)

  85. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: Proceedings of 4th International Conference on Learning Representations (2016)

  86. Robinson, M.D., Chiu, S.J., Toth, C.A., Izatt, J.A., Lo, J.Y., Farsiu, S.: New applications of super-resolution in medical imaging. In: Super-Resolution Imaging, pp. 401–430. CRC Press (2017)

  87. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)

    Google Scholar 

  88. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    MathSciNet  Google Scholar 

  89. Ryoo, M.S., Rothrock, B., Fleming, C., Yang, H.J.: Privacy-preserving human activity recognition from extreme low resolution. In: Proceedings of AAAI Conference on Artificial Intelligence, pp. 4255–4262 (2017)

  90. Sajjadi, M.S., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: Proceedings of IEEE International Conference on Computer Vision, pp. 4501–4510. IEEE (2017)

  91. Sajjadi, M.S.M., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: Proceedings of IEEE International Conference on Computer Vision, pp. 4501–4510 (2017)

  92. Salimans, T., Goodfellow, I.J., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. In: Proceedings of Neural Information Processing Systems, pp. 2226–2234 (2016)

  93. Salvador, J., Perez-Pellitero, E.: Naive bayes super-resolution forest. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 325–333 (2015)

  94. Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)

  95. Schultz, R.R., Stevenson, R.L.: A bayesian approach to image expansion for improved definition. IEEE Trans. Image Process. 3(3), 233–242 (1994)

    Google Scholar 

  96. Shamsolmoali, P., Zhang, J., Yang, J., et al.: Image super resolution by dilated dense progressive network. Image Vis. Comput. 88, 9–18 (2019)

    Google Scholar 

  97. Shan, Q., Li, Z., Jia, J., Tang, C.K.: Fast image/video upsampling. ACM Trans. Graph. 27(5), 153 (2008)

    Google Scholar 

  98. Sheikh, H.R., Wang, Z., Cormack, L., Bovik, A.C.: Live image quality assessment database release 2, 2005 (2005)

  99. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  100. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  101. Singh, K., Gupta, A., Kapoor, R.: Fingerprint image super-resolution via ridge orientation-based clustered coupled sparse dictionaries. J. Electr. Imaging 24(4), 043015 (2015)

    Google Scholar 

  102. Song, H., Zhang, L., Wang, P., Zhang, K., Li, X.: An adaptive l 1–l 2 hybrid error model to super-resolution. In: Proceedings of IEEE International Conference on Image Processing, pp. 2821–2824 (2010)

  103. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  104. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2798 (2017)

  105. Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18(9), 1958–1975 (2009)

    MathSciNet  MATH  Google Scholar 

  106. Tian, J., Ma, K.K.: Stochastic super-resolution image reconstruction. J. Vis. Commun. Image Represent. 21(3), 232–244 (2010)

    Google Scholar 

  107. Tian, J., Ma, K.K.: A survey on super-resolution imaging. Signal Image Video Process. 5(3), 329–342 (2011)

    Google Scholar 

  108. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L.: Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)

  109. Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1920–1927 (2013)

  110. Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: Proceedings of Asian Conference on Computer Vision, pp. 111–126 (2014)

  111. Timofte, R., Rothe, R., Van Gool, L.: Seven ways to improve example-based single image super resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1865–1873 (2016)

  112. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: Proceedings of IEEE International Conference on Computer Vision, pp. 4809–4817 (2017)

  113. Trinh, D.H., Luong, M., Dibos, F., Rocchisani, J.M., Pham, C.D., Nguyen, T.Q.: Novel example-based method for super-resolution and denoising of medical images. IEEE Trans. Image Process. 23(4), 1882–1895 (2014)

    MathSciNet  MATH  Google Scholar 

  114. Wallach, D., Lamare, F., Kontaxakis, G., Visvikis, D.: Super-resolution in respiratory synchronized positron emission tomography. IEEE Trans. Med. Imaging 31(2), 438–448 (2012)

    Google Scholar 

  115. Wang, X., Tang, X.: Hallucinating face by eigentransformation. IEEE Trans. Syst. Man Cybern. Part C 35(3), 425–434 (2005)

    Google Scholar 

  116. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Loy, C.C.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of European Conference on Computer Vision Workshops, pp. 63–79 (2018)

  117. Wang, Y., Perazzi, F., McWilliams, B., Sorkine-Hornung, A., Sorkine-Hornung, O., Schroers, C.: A fully progressive approach to single-image super-resolution. In: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 864–873 (2018)

  118. Wang, Y.H., Qiao, J., Li, J.B., Fu, P., Chu, S.C., Roddick, J.F.: Sparse representation-based mri super-resolution reconstruction. Measurement 47, 946–953 (2014)

    Google Scholar 

  119. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 370–378 (2015)

  120. Wu, H., Zheng, S., Zhang, J., Huang, K.: GP-GAN: towards realistic high-resolution image blending. CoRR arXiv:abs/1703.07195 (2017)

  121. Wu, Y., Shivakumara, P., Lu, T., Tan, C.L., Blumenstein, M., Kumar, G.H.: Contour restoration of text components for recognition in video/scene images. IEEE Trans. Image Process. 25(12), 5622–5634 (2016)

    MathSciNet  MATH  Google Scholar 

  122. Wu, Y., Shivakumara, P., Wei, W., Lu, T., Pal, U.: A new ring radius transform-based thinning method for multi-oriented video characters. Int. J. Doc. Anal. Recognit. 18(2), 137–151 (2015)

    Google Scholar 

  123. Xu, X., Fu, S., Qi, L., Zhang, X., Liu, Q., He, Q., Li, S.: An iot-oriented data placement method with privacy preservation in cloud environment. J. Netw. Comput. Appl. 124, 148–157 (2018)

    Google Scholar 

  124. Xu, X., Ma, Y., Sun, W.: Towards real scene super-resolution with raw images. CoRR arXiv:abs/1905.12156 (2019)

  125. Xu, X., Zhang, X., Khan, M., Dou, W., Xue, S., Yu, S.: A balanced virtual machine scheduling method for energy-performance trade-offs in cyber-physical cloud systems. Future Generation Computer Systems (2017)

  126. Xu, X., Zhao, X., Ruan, F., Zhang, J., Tian, W., Dou, W., Liu, A.X.: Data placement for privacy-aware applications over big data in hybrid clouds. Secur. Commun. Netw. 2017, 2376484:1–2376484:15 (2017)

    Google Scholar 

  127. Xu, Y., Qi, L., Dou, W., Yu, J.: Privacy-preserving and scalable service recommendation based on simhash in a distributed cloud environment. Complexity 2017, 3437854:1–3437854:9 (2017)

    Google Scholar 

  128. Yamanaka, J., Kuwashima, S., Kurita, T.: Fast and accurate image super resolution by deep cnn with skip connection and network in network. In: Proceedings of Neural Information Processing, pp. 217–225 (2017)

  129. Yang, C.Y., Yang, M.H.: Fast direct super-resolution by simple functions. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 561–568 (2013)

  130. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    MathSciNet  MATH  Google Scholar 

  131. Ye, Q., Doermann, D.S.: Text detection and recognition in imagery: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 37(7), 1480–1500 (2015)

    Google Scholar 

  132. Yu, X., Fernando, B., Ghanem, B., Porikli, F., Hartley, R.: Face super-resolution guided by facial component heatmaps. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 217–233 (2018)

  133. Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., Lin, L.: Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 701–710 (2018)

  134. Yuan, Z., Wu, J., Kamata, S.i., Ahrary, A., Yan, P.: Fingerprint image enhancement by super resolution with early stopping. In: Proceedings of IEEE International Conference on Intelligent Computing and Intelligent Systems, vol. 4, pp. 527–531 (2009)

  135. Yue, L., Shen, H., Jie, L., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Signal Process. 128, 389–408 (2016)

    Google Scholar 

  136. Zareapoor, M., Zhou, H., Yang, J.: Perceptual image quality using dual generative adversarial network. Neural Computing and Applications pp. 1–11 (2019)

  137. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Proceedings of International Conference on Curves and Surfaces, pp. 711–730 (2010)

  138. Zhang, H., Yang, Z., Zhang, L., Shen, H.: Super-resolution reconstruction for multi-angle remote sensing images considering resolution differences. Rem. Sens. 6(1), 637–657 (2014)

    Google Scholar 

  139. Zhang, H., Zhang, L., Shen, H.: A super-resolution reconstruction algorithm for hyperspectral images. Signal Process. 92(9), 2082–2096 (2012)

    Google Scholar 

  140. Zhang, J., Pu, J., Chen, C., Fleischer, R.: Low-resolution gait recognition. IEEE Trans. Syst. Man Cybern. Part B 40(4), 986–996 (2010)

    Google Scholar 

  141. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262–3271 (2018)

  142. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision, pp. 286–301 (2018)

  143. Zhang, Y., Liu, S., Dong, C., Zhang, X., Yuan, Y.: Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution. IEEE transactions on Image Processing: (2019)

  144. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)

  145. Zhao, S., Han, H., Peng, S.: Wavelet-domain hmt-based image super-resolution. In: Proceedings os International Conference on Image Processing, vol. 2, p. 953 (2003)

  146. Zhao, T., Ren, W., Zhang, C., Ren, D., Hu, Q.: Unsupervised degradation learning for single image super-resolution. CoRR arXiv:abs/1812.04240 (2018)

  147. Zheng, H., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 723–731 (2018)

  148. Zheng, Z., Zheng, L., Yang, Y.: Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3754–3762 (2017)

  149. Zhu, J., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of IEEE International Conference on Computer Vision, pp. 2242–2251 (2017)

  150. Zhu, J.Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: Proceedings of European Conference on Computer Vision, pp. 597–613. Springer (2016)

  151. Zhuang, Y., Zhang, J., Wu, F.: Hallucinating faces: Lph super-resolution and neighbor reconstruction for residue compensation. Pattern Recognit. 40(11), 3178–3194 (2007)

    MATH  Google Scholar 

Download references

Acknowledgments

This work was supported by National Key R&D Program of China under Grant 2018YFC0407901, the Natural Science Foundation of China under Grant 61702160, the Natural Science Foundation of Jiangsu Province under Grant BK20170892, and the open Project of the National Key Lab for Novel Software Technology in NJU under Grant K-FKT2017B05.

Author information

Authors and Affiliations

  1. College of Computer and Information, Hohai University, Nanjing, China

    Yirui Wu & Wen Zhang

  2. School of Computer Science and Information Engineering, Changzhou Institute of Technology, Changzhou, China

    Xiaofang Li

  3. School of Computer Science, University College Dublin, Dublin, Ireland

    Ruichao Wang

  4. School of Natural and Computational Science, Massey University, Palmerston North, New Zealand

    Feng Hou

Authors
  1. Xiaofang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yirui Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruichao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Feng Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yirui Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Wu, Y., Zhang, W. et al. Deep learning methods in real-time image super-resolution: a survey. J Real-Time Image Proc 17, 1885–1909 (2020). https://doi.org/10.1007/s11554-019-00925-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-019-00925-3

Keywords

Search

Navigation