Abstract
Since the development of deep learning, image super-resolution (SR) has made great progress, and become the focus of academic research. Because high-level features are more informative for the reconstruction, most SR networks have a large number of layers and parameters, which restrict their application in resource-constrained devices. Recently, lightweight networks got a lot attention for their broad application prospect. To improve the performance of lightweight networks by informative high-level features, we introduce feedback mechanism into our method, which can feed back high-level features to refine low-level ones. In this paper, we propose a closed-loop feedback network with cross back-projection for lightweight image super-resolution (CCFN), which uses feedback mechanism in three manners. First, based on error feedback, we propose a cross back-projection feedback block (CFB). CFB uses error feedback to correct the features of multi-scale fusion, which also can be viewed as two cross-learning back-projection units. Second, CFB works in a self-feedback manner, which feeds back the output high-level features to refine the low-level ones of the input. Third, we propose a global feedback, which feeds back the degradation results of SR to LR, to guide the learning of mapping functions from LR to HR. Finally, we use attention-based model as the basic block in CFB, and since our method works in an iterative manner, recursive concatenation is more suitable than multi-reconstruction. The final experimental results show that our CCFN has a competitive performance with few parameters.
Similar content being viewed by others
References
Chao Dong, C. C. L., & Tang, X. (2016). Accelerating the super-resolution convolutional neural network. In Computer Vision - ECCV 2016 (pp. 391–407). https://doi.org/10.1007/978-3-319-46475-6_25
Chao Dong, K. H., Chen Change Loy, & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In Computer Vision - ECCV 2014 (pp. 184–199). volume 8692. https://doi.org/10.1007/978-3-319-10593-2_13
Kim, J., Lee, J., & Lee, K. (2016). Accurate image super-resolution using very deep convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1646–1654). https://doi.org/10.1109/CVPR.2016.182
Kim, J., Lee, J. K., & Lee, K. M. (2016). Deeply-recursive convolutional network for image super-resolution. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1637–1645). https://doi.org/10.1109/CVPR.2016.181
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 105–114). https://doi.org/10.1109/CVPR.2017.19
Li, Z., Li, Q., Wu, W., Wu, Z., Lu, L., & Yang, X. (2020). Clustering based multiple branches deep networks for single image super-resolution. Multimedia Tools and Applications, 79. https://doi.org/10.1007/s11042-018-7017-y
Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1874–1883). https://doi.org/10.1109/CVPR.2016.207
Yang, X., Wu, W., Lu, L., Yan, B., Zhang, L., & Liu, K. (2020). Multiple regressions based image super-resolution. Multimedia Tools and Applications, 79. https://doi.org/10.1007/s11042-019-7716-z
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR).
Szegedy, C., Wei Liu, Yangqing Jia, Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1–9). https://doi.org/10.1109/CVPR.2015.7298594
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90
Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017). Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 1132–1140). https://doi.org/10.1109/CVPRW.2017.151
Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. (2017). Densely connected convolutional networks. https://doi.org/10.1109/CVPR.2017.243
Tai, Y., Yang, J., & Liu, X. (2017a). Image super-resolution via deep recursive residual network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2790–2798). https://doi.org/10.1109/CVPR.2017.298
Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., & Tan, M. (2020). Closed-loop matters: Dual regression networks for single image super-resolution. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5406–5415). https://doi.org/10.1109/CVPR42600.2020.00545
Gilbert, C. D., & Sigman, M. (2007). Brain states: Top-down influences in sensory processing. In Neuron (p. 677–696). volume sps54. https://doi.org/10.1016/j.neuron.2007.05.019
Stollenga, M., Masci, J., Gomez, F., & Schmidhuber, J. (2014). Deep networks with internal selective attention through feedback connections. volume 4.
Zamir, A. R., Wu, T.-L., Sun, L., Shen, W., Shi, B. E., Malik, J., & Savarese, S. (2016). Feedback networks. In Computer Science - Computer Vision and Pattern Recognition.
Haris, M., Shakhnarovich, G., & Ukita, N. (2018). Deep back-projection networks for super-resolution. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1664–1673). https://doi.org/10.1109/CVPR.2018.00179
Sun, K., Xiao, B., Liu, D., & Wang, J. (2019). Deep High-Resolution Representation Learning for Human Pose Estimation. arXiv e-prints, (p. arXiv:1902.09212).
Zhang, Y., kunpeng Li, Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image super-resolution using very deep residual channel attention networks. In European Conference on Computer Vision (pp. 294–310). volume 11211. https://doi.org/10.1007/978-3-030-01234-2_18
Lai, W.-S., Huang, J.-B., Ahuja, N., & Yang, M.-H. (2017). Deep laplacian pyramid networks for fast and accurate super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2017.618
Hui, Z., Gao, X., Yang, Y., & Wang, X. (2019). Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia (pp. 2024–2032). https://doi.org/10.1145/3343031.3351084
Namhyuk Ahn, B. K., & Sohn, K.-A. (2018). Fast, accurate, and lightweight super-resolution with cascading residual network. In Computer Vision - ECCV 2018 (pp. 256–272). volume 11214. https://doi.org/10.1007/978-3-030-01249-6_16
Hui, Z., Wang, X., & Gao, X. (2018). Fast and accurate single image super-resolution via information distillation network. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 723–731). https://doi.org/10.1109/CVPR.2018.00082
Xie, Y., Zhang, Y., Qu, Y., Li, C., & Fu, Y. (2020). Latticenet: Towards lightweight image super-resolution with lattice block. (pp. 272–289). https://doi.org/10.1007/978-3-030-58542-6_17
Wang, L., Dong, X., Wang, Y., Ying, X., Lin, Z., An, W., & Guo, Y. (2021). Exploring sparsity in image super-resolution for efficient inference. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4915–4924). https://doi.org/10.1109/CVPR46437.2021.00488
Irani, M., & Peleg, S. (2002). Motion analysis for image enhancement: Resolution, occlusion, and transparency. Journal of Visual Communication and Image Representation, 4. https://doi.org/10.1006/jvci.1993.1030
Dai, S., Han, M., Wu, Y., & Gong, Y. (2007). Bilateral back-projection for single image super resolution. (pp. 1039 – 1042). https://doi.org/10.1109/ICME.2007.4284831
Dong, W., Zhang, L., Shi, G., & Wu, X. (2009). Nonlocal back-projection for adaptive image enlargement. In 2009 16th IEEE International Conference on Image Processing (ICIP) (pp. 349–352). https://doi.org/10.1109/ICIP.2009.5414423
Timofte, R., Rothe, R., & Van Gool, L. (2016). Seven ways to improve example-based single image super resolution. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1865–1873). https://doi.org/10.1109/CVPR.2016.206
Fu, S., Li, Z., Liu, K., Din, S., Imran, M., & Yang, X. (2020). Model compression for iot applications in industry 4.0 via multiscale knowledge transfer. IEEE Transactions on Industrial Informatics, 16, 6013–6022. https://doi.org/10.1109/TII.2019.2953106
Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., & Wu, W. (2019). Feedback network for image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Shipeng Fu, H. L. Z. L. W. W. A. P. G. J. X. Y., Lu Lu (2019). A real-time super-resolution method based on convolutional neural networks. In Circuits, Systems, and Signal Processing. https://doi.org/10.1007/s00034-019-01283-y
Zhen Li, W. W. J. Y. Z. L. X. Y., Qilei Li (2020). Deep recursive up-down sampling networks for single image super-resolution. In Neurocomputing (pp. 377–388). volume 398. https://doi.org/10.1016/j.neucom.2019.04.004
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018). Residual dense network for image super-resolution. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2472–2481). https://doi.org/10.1109/CVPR.2018.00262
Carreira, J., Agrawal, P., Fragkiadaki, K., & Malik, J. (2016). Human pose estimation with iterative error feedback. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4733–4742). https://doi.org/10.1109/CVPR.2016.512
Agustsson, E., & Timofte, R. (2017). Ntire 2017 challenge on single image super-resolution: Dataset and study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 1122–1131). https://doi.org/10.1109/CVPRW.2017.150
Cai, J., Zeng, H., Yong, H., Cao, Z., & Zhang, L. (2019). Toward real-world single image super-resolution: A new benchmark and a new model. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 3086–3095). https://doi.org/10.1109/ICCV.2019.00318
Kingma, D., & Ba, J. (2014). Adam: A method for stochastic optimization. International Conference on Learning Representations.
Tai, Y., Yang, J., Liu, X., & Xu, C. (2017b). Memnet: A persistent memory network for image restoration. In 2017 IEEE International Conference on Computer Vision (ICCV) (pp. 4549–4557). https://doi.org/10.1109/ICCV.2017.486
Zhang, K., Zuo, W., & Zhang, L. (2018). Learning a single convolutional super-resolution network for multiple degradations. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3262–3271). https://doi.org/10.1109/CVPR.2018.00344
Funding
This work was supported by the funding from Science Foundation of Sichuan Science and Technology Department 2021YFH0119 and the funding from Sichuan University under grant 2020SCUNG205.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflicts of Interests/Competing Interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wang, B., Liu, C., Jeon, S. et al. Closed-loop Feedback Network with Cross Back-Projection for Lightweight Image Super-Resolution. J Sign Process Syst 95, 305–318 (2023). https://doi.org/10.1007/s11265-022-01764-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11265-022-01764-5