Skip to main content
Log in

Closed-loop Feedback Network with Cross Back-Projection for Lightweight Image Super-Resolution

  • Published:
Journal of Signal Processing Systems Aims and scope Submit manuscript

Abstract

Since the development of deep learning, image super-resolution (SR) has made great progress, and become the focus of academic research. Because high-level features are more informative for the reconstruction, most SR networks have a large number of layers and parameters, which restrict their application in resource-constrained devices. Recently, lightweight networks got a lot attention for their broad application prospect. To improve the performance of lightweight networks by informative high-level features, we introduce feedback mechanism into our method, which can feed back high-level features to refine low-level ones. In this paper, we propose a closed-loop feedback network with cross back-projection for lightweight image super-resolution (CCFN), which uses feedback mechanism in three manners. First, based on error feedback, we propose a cross back-projection feedback block (CFB). CFB uses error feedback to correct the features of multi-scale fusion, which also can be viewed as two cross-learning back-projection units. Second, CFB works in a self-feedback manner, which feeds back the output high-level features to refine the low-level ones of the input. Third, we propose a global feedback, which feeds back the degradation results of SR to LR, to guide the learning of mapping functions from LR to HR. Finally, we use attention-based model as the basic block in CFB, and since our method works in an iterative manner, recursive concatenation is more suitable than multi-reconstruction. The final experimental results show that our CCFN has a competitive performance with few parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12

Similar content being viewed by others

References

  1. Chao Dong, C. C. L., & Tang, X. (2016). Accelerating the super-resolution convolutional neural network. In Computer Vision - ECCV 2016 (pp. 391–407). https://doi.org/10.1007/978-3-319-46475-6_25

  2. Chao Dong, K. H., Chen Change Loy, & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In Computer Vision - ECCV 2014 (pp. 184–199). volume 8692. https://doi.org/10.1007/978-3-319-10593-2_13

  3. Kim, J., Lee, J., & Lee, K. (2016). Accurate image super-resolution using very deep convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1646–1654). https://doi.org/10.1109/CVPR.2016.182

  4. Kim, J., Lee, J. K., & Lee, K. M. (2016). Deeply-recursive convolutional network for image super-resolution. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1637–1645). https://doi.org/10.1109/CVPR.2016.181

  5. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 105–114). https://doi.org/10.1109/CVPR.2017.19

  6. Li, Z., Li, Q., Wu, W., Wu, Z., Lu, L., & Yang, X. (2020). Clustering based multiple branches deep networks for single image super-resolution. Multimedia Tools and Applications, 79. https://doi.org/10.1007/s11042-018-7017-y

  7. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1874–1883). https://doi.org/10.1109/CVPR.2016.207

  8. Yang, X., Wu, W., Lu, L., Yan, B., Zhang, L., & Liu, K. (2020). Multiple regressions based image super-resolution. Multimedia Tools and Applications, 79. https://doi.org/10.1007/s11042-019-7716-z

  9. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR).

  10. Szegedy, C., Wei Liu, Yangqing Jia, Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1–9). https://doi.org/10.1109/CVPR.2015.7298594

  11. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90

  12. Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017). Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 1132–1140). https://doi.org/10.1109/CVPRW.2017.151

  13. Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. (2017). Densely connected convolutional networks. https://doi.org/10.1109/CVPR.2017.243

  14. Tai, Y., Yang, J., & Liu, X. (2017a). Image super-resolution via deep recursive residual network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2790–2798). https://doi.org/10.1109/CVPR.2017.298

  15. Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., & Tan, M. (2020). Closed-loop matters: Dual regression networks for single image super-resolution. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5406–5415). https://doi.org/10.1109/CVPR42600.2020.00545

  16. Gilbert, C. D., & Sigman, M. (2007). Brain states: Top-down influences in sensory processing. In Neuron (p. 677–696). volume sps54. https://doi.org/10.1016/j.neuron.2007.05.019

  17. Stollenga, M., Masci, J., Gomez, F., & Schmidhuber, J. (2014). Deep networks with internal selective attention through feedback connections. volume 4.

  18. Zamir, A. R., Wu, T.-L., Sun, L., Shen, W., Shi, B. E., Malik, J., & Savarese, S. (2016). Feedback networks. In Computer Science - Computer Vision and Pattern Recognition.

  19. Haris, M., Shakhnarovich, G., & Ukita, N. (2018). Deep back-projection networks for super-resolution. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1664–1673). https://doi.org/10.1109/CVPR.2018.00179

  20. Sun, K., Xiao, B., Liu, D., & Wang, J. (2019). Deep High-Resolution Representation Learning for Human Pose Estimation. arXiv e-prints, (p. arXiv:1902.09212).

  21. Zhang, Y., kunpeng Li, Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image super-resolution using very deep residual channel attention networks. In European Conference on Computer Vision (pp. 294–310). volume 11211. https://doi.org/10.1007/978-3-030-01234-2_18

  22. Lai, W.-S., Huang, J.-B., Ahuja, N., & Yang, M.-H. (2017). Deep laplacian pyramid networks for fast and accurate super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2017.618

  23. Hui, Z., Gao, X., Yang, Y., & Wang, X. (2019). Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia (pp. 2024–2032). https://doi.org/10.1145/3343031.3351084

  24. Namhyuk Ahn, B. K., & Sohn, K.-A. (2018). Fast, accurate, and lightweight super-resolution with cascading residual network. In Computer Vision - ECCV 2018 (pp. 256–272). volume 11214. https://doi.org/10.1007/978-3-030-01249-6_16

  25. Hui, Z., Wang, X., & Gao, X. (2018). Fast and accurate single image super-resolution via information distillation network. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 723–731). https://doi.org/10.1109/CVPR.2018.00082

  26. Xie, Y., Zhang, Y., Qu, Y., Li, C., & Fu, Y. (2020). Latticenet: Towards lightweight image super-resolution with lattice block. (pp. 272–289). https://doi.org/10.1007/978-3-030-58542-6_17

  27. Wang, L., Dong, X., Wang, Y., Ying, X., Lin, Z., An, W., & Guo, Y. (2021). Exploring sparsity in image super-resolution for efficient inference. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4915–4924). https://doi.org/10.1109/CVPR46437.2021.00488

  28. Irani, M., & Peleg, S. (2002). Motion analysis for image enhancement: Resolution, occlusion, and transparency. Journal of Visual Communication and Image Representation, 4. https://doi.org/10.1006/jvci.1993.1030

  29. Dai, S., Han, M., Wu, Y., & Gong, Y. (2007). Bilateral back-projection for single image super resolution. (pp. 1039 – 1042). https://doi.org/10.1109/ICME.2007.4284831

  30. Dong, W., Zhang, L., Shi, G., & Wu, X. (2009). Nonlocal back-projection for adaptive image enlargement. In 2009 16th IEEE International Conference on Image Processing (ICIP) (pp. 349–352). https://doi.org/10.1109/ICIP.2009.5414423

  31. Timofte, R., Rothe, R., & Van Gool, L. (2016). Seven ways to improve example-based single image super resolution. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1865–1873). https://doi.org/10.1109/CVPR.2016.206

  32. Fu, S., Li, Z., Liu, K., Din, S., Imran, M., & Yang, X. (2020). Model compression for iot applications in industry 4.0 via multiscale knowledge transfer. IEEE Transactions on Industrial Informatics, 16, 6013–6022. https://doi.org/10.1109/TII.2019.2953106

  33. Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., & Wu, W. (2019). Feedback network for image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  34. Shipeng Fu, H. L. Z. L. W. W. A. P. G. J. X. Y., Lu Lu (2019). A real-time super-resolution method based on convolutional neural networks. In Circuits, Systems, and Signal Processing. https://doi.org/10.1007/s00034-019-01283-y

  35. Zhen Li, W. W. J. Y. Z. L. X. Y., Qilei Li (2020). Deep recursive up-down sampling networks for single image super-resolution. In Neurocomputing (pp. 377–388). volume 398. https://doi.org/10.1016/j.neucom.2019.04.004

  36. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018). Residual dense network for image super-resolution. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2472–2481). https://doi.org/10.1109/CVPR.2018.00262

  37. Carreira, J., Agrawal, P., Fragkiadaki, K., & Malik, J. (2016). Human pose estimation with iterative error feedback. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4733–4742). https://doi.org/10.1109/CVPR.2016.512

  38. Agustsson, E., & Timofte, R. (2017). Ntire 2017 challenge on single image super-resolution: Dataset and study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 1122–1131). https://doi.org/10.1109/CVPRW.2017.150

  39. Cai, J., Zeng, H., Yong, H., Cao, Z., & Zhang, L. (2019). Toward real-world single image super-resolution: A new benchmark and a new model. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 3086–3095). https://doi.org/10.1109/ICCV.2019.00318

  40. Kingma, D., & Ba, J. (2014). Adam: A method for stochastic optimization. International Conference on Learning Representations.

  41. Tai, Y., Yang, J., Liu, X., & Xu, C. (2017b). Memnet: A persistent memory network for image restoration. In 2017 IEEE International Conference on Computer Vision (ICCV) (pp. 4549–4557). https://doi.org/10.1109/ICCV.2017.486

  42. Zhang, K., Zuo, W., & Zhang, L. (2018). Learning a single convolutional super-resolution network for multiple degradations. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3262–3271). https://doi.org/10.1109/CVPR.2018.00344

Download references

Funding

This work was supported by the funding from Science Foundation of Sichuan Science and Technology Department 2021YFH0119 and the funding from Sichuan University under grant 2020SCUNG205.

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Seunggil Jeon or Xiaomin Yang.

Ethics declarations

Conflicts of Interests/Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, B., Liu, C., Jeon, S. et al. Closed-loop Feedback Network with Cross Back-Projection for Lightweight Image Super-Resolution. J Sign Process Syst 95, 305–318 (2023). https://doi.org/10.1007/s11265-022-01764-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-022-01764-5

Keywords

Navigation