Skip to main content
SpringerLink
Log in

Joint restoration convolutional neural network for low-quality image super resolution

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, a joint restoration convolutional neural network (JRCNN) is proposed to produce a visually pleasing super resolution (SR) image from a single low-quality (LQ) image. The LQ image is a low resolution (LR) image with ringing, blocking and blurring artifacts arising due to compression. JRCNN consists of three deep dense residual blocks (DRB). Each DRB comprises of parallel convolutional layers with cross residual connections. The representational power of JRCNN is improved by depth-wise concatenation of feature representations from each of the DRBs. Moreover, these connections mitigate the problem of vanishing of gradients. Different from the previous networks, JRCNN exploits the contextual information directly in the LR image space without using any interpolation. This strategy improves the training efficiency of the network. The exhaustive experimentation on different datasets show that the proposed JRCNN produces state-of-the-art performance. Furthermore, ablation experiments are performed to assess the effectiveness of JRCNN. In addition, individual experiments are conducted for SR and compression artifact removal on benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision—ECCV 2018, pp. 256–272. Springer International Publishing, Cham (2018)

    Chapter  Google Scholar 

  2. Amaranageswarao, G., Deivalakshmi, S., Ko, S.-B.: Residual learning based densely connected deep dilated network for joint deblocking and super resolution. Appl. Intell. (2020). https://doi.org/10.1007/s10489-020-01670-y

    Article  Google Scholar 

  3. Bevilacqua, M., Roumy, A., Guillemot, C.: line Alberi Morel M (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British Machine Vision Conference, BMVA Press, pp 135.1–135.10, https://doi.org/10.5244/C.26.135

  4. Cavigelli, L., Hager, P., Benini, L.: Cas-cnn: A deep convolutional neural network for image compression artifact suppression. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp 752–759 (2017) https://doi.org/10.1109/IJCNN.2017.7965927

  5. Chen, H., He, X., Ren, C., Qing, L., Teng, Q.: Cisrdcnn: super-resolution of compressed images using deep convolutional neural networks. Neurocomputing 285, 204–219 (2018). https://doi.org/10.1016/j.neucom.2018.01.043

    Article  Google Scholar 

  6. Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2017). https://doi.org/10.1109/TPAMI.2016.2596743

    Article  Google Scholar 

  7. Dong, C., Deng, Y., Loy, C.C., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 576–584 (2015) https://doi.org/10.1109/ICCV.2015.73

  8. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016). https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  9. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision—ECCV 2016, pp. 391–407. Springer International Publishing, Cham (2016)

    Chapter  Google Scholar 

  10. Fan, Y., Shi, H., Yu, J., Liu, D., Han, W., Yu, H., Wang, Z., Wang, X., Huang, T.S.: Balanced two-stage residual networks for image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1157–1164 (2017) https://doi.org/10.1109/CVPRW.2017.154

  11. Foi, A., Katkovnik, V., Egiazarian, K.: Pointwise shape-adaptive dct for high-quality denoising and deblocking of grayscale and color images. IEEE Trans. Image Process. 16(5), 1395–1411 (2007). https://doi.org/10.1109/TIP.2007.891788

    Article  MathSciNet  Google Scholar 

  12. Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206 (2015) https://doi.org/10.1109/CVPR.2015.7299156

  13. Kang, L., Hsu, C., Zhuang, B., Lin, C., Yeh, C.: Learning-based joint super-resolution and deblocking for a highly compressed image. IEEE Trans. Multimed. 17(7), 921–934 (2015)

    Article  Google Scholar 

  14. Kayhan, S.K.: Efficient robust filtering technique for blocking artifacts reduction. Vis. Comput. (2015). https://doi.org/10.1007/s00371-015-1068-0

    Article  Google Scholar 

  15. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process. 29(6), 1153–1160 (1981)

    Article  MathSciNet  Google Scholar 

  16. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2016) https://doi.org/10.1109/CVPR.2016.182

  17. Kim, Y., Soh, J.W., Park, J., Ahn, B., Lee, H., Moon, Y., Cho, N.I.: A pseudo-blind convolutional neural network for the reduction of compression artifacts. IEEE Trans. Circuits Syst. Video Technol. (2019). https://doi.org/10.1109/TCSVT.2019.2901919

    Article  Google Scholar 

  18. Lai, W., Huang, J., Ahuja, N., Yang, M.: Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2599–2613 (2019). https://doi.org/10.1109/TPAMI.2018.2865304

    Article  Google Scholar 

  19. Lan, R., Sun, L., Liu, Z., Lu, H., Su, Z., Pang, C., Luo, X.: Cascading and enhanced residual networks for accurate single-image super-resolution. IEEE Trans. Cybern. (2020)

  20. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017) https://doi.org/10.1109/CVPRW.2017.151

  21. Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-cnn for image restoration. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 886–88609 (2018) https://doi.org/10.1109/CVPRW.2018.00121

  22. Ma, T., Tian, W.: Back-projection-based progressive growing generative adversarial network for single image super-resolution. In: Vis Comput (2020)

  23. Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, Curran Associates, Inc., pp. 2802–2810 (2016)

  24. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol 2, vol. 2, pp. 416–423 (2001) https://doi.org/10.1109/ICCV.2001.937655

  25. Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimed. Tools Appl. 76(20), 21811–21838 (2017). https://doi.org/10.1007/s11042-016-4020-z

    Article  Google Scholar 

  26. Qiu, Y., Wang, R., Tao, D., Cheng, J.: Embedded block residual network: A recursive restoration model for single-image super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4179–4188 (2019)

  27. Song, Q., Xiong, R., Fan, X., Liu, X., Huang, T., Gao, W.: Compressed image restoration via external-image assisted band adaptive pca model learning. In: 2018 Data Compression Conference, pp. 97–106 (2018) https://doi.org/10.1109/DCC.2018.00018

  28. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2798 (2017) https://doi.org/10.1109/CVPR.2017.298

  29. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4549–4557 (2017b) https://doi.org/10.1109/ICCV.2017.486

  30. Timofte, R., DéSmet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. ACCV 9006, 111–126 (2015). https://doi.org/10.1007/978-3-319-16817-3_8

    Article  Google Scholar 

  31. Timofte, R., Agustsson, E., Gool, L.V., Yang, M., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M., Wang, X., Tian, Y., Yu, K., Zhang, Y., Wu, S., Dong, C., Lin, L,, Qiao, Y., Loy, C.C., Bae, W., Yoo, J., Han, Y,, Ye, J.C., Choi, J., Kim, M., Fan, Y,, Yu, J., Han, W., Liu, D., Yu, H., Wang, Z., Shi, H., Wang, X,, Huang, T.S., Chen, Y., Zhang, K., Zuo, W., Tang, Z., Luo, L., Li. S., Fu, M., Cao, L., Heng, W., Bui, G., Le, T., Duan, Y., Tao, D., Wang, R., Lin, X., Pang, J., Xu, J., Zhao, Y., Xu, X., Pan, J., Sun, D., Zhang, Y., Song, X., Dai, Y., Qin, X., Huynh, X., Guo, T., Mousavi, H.S., Vu, T.H., Monga, V., Cruz, C., Egiazarian, K., Katkovnik, V., Mehta, R., Jain, A.K., Agarwalla, A., Praveen, C.V.S., Zhou, R., Wen, H., Zhu, C., Xia, Z., Wang, Z., Guo, Q.: Ntire 2017 challenge on single image super-resolution: Methods and results. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1110–1121 (2017) https://doi.org/10.1109/CVPRW.2017.149

  32. Wallace, G.K.: The jpeg still picture compression standard. IEEE Trans. Consum. Electron. 38(1), xviii–xxxiv (1992)

  33. Xiao, J., Wang, C., Hu, X.: Single image super-resolution in compressed domain based on field of expert prior. In: 2012 5th International Congress on Image and Signal Processing, pp. 607–611 (2012)

  34. Zeyde, R., Elad, M., Protter M.: On single image scale-up using sparse-representations. 6920, 711–730 (2010). https://doi.org/10.1007/978-3-642-27413-8_47

  35. Zhang, J., Xiong, R., Zhao, C., Zhang, Y., Ma, S., Gao, W.: Concolor: Constrained non-convex low-rank model for image deblocking. IEEE Trans. Image Process. 25(3), 1246–1259 (2016). https://doi.org/10.1109/TIP.2016.2515985

    Article  MathSciNet  MATH  Google Scholar 

  36. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017). https://doi.org/10.1109/TIP.2017.2662206

    Article  MathSciNet  MATH  Google Scholar 

  37. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3262–3271 (2018) https://doi.org/10.1109/CVPR.2018.00344

  38. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision—ECCV 2018, pp. 294–310. Springer International Publishing, Cham (2018)

    Chapter  Google Scholar 

  39. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018) https://doi.org/10.1109/CVPR.2018.00262

  40. Zhao, C., Zhang, J., Ma, S., Fan, X., Zhang, Y., Gao, W.: Reducing image compression artifacts by structural sparse representation and quantization constraint prior. IEEE Trans. Circuits Syst. Video Technol. 27(10), 2057–2071 (2017). https://doi.org/10.1109/TCSVT.2016.2580399

    Article  Google Scholar 

  41. Wang, Zhou, Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, National Institute of Technology, Thiruchirappalli, India

    Gadipudi Amaranageswarao & S. Deivalakshmi

  2. Department of Electrical and Computer Engineering, University of Saskatchewan, Saskatoon, Canada

    Seok-Bum Ko

Authors
  1. Gadipudi Amaranageswarao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Deivalakshmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seok-Bum Ko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gadipudi Amaranageswarao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amaranageswarao, G., Deivalakshmi, S. & Ko, SB. Joint restoration convolutional neural network for low-quality image super resolution. Vis Comput 38, 31–50 (2022). https://doi.org/10.1007/s00371-020-01998-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01998-z

Keywords

Search

Navigation