Skip to main content
SpringerLink
Log in

Lightweight feature separation, fusion and optimization networks for accurate image super-resolution

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Recently, single-image super-resolution (SISR) methods based on deep learning have demonstrated great superiority by deepening or widening the network. However, excessive network layers will not only weaken the information flow during training process, but also increase the storage load and computation cost in practical application. To achieve a better trade-off between model efficiency and accuracy, we propose a lightweight feature separation, fusion and optimization network (SFON) for SISR. For the architecture, we design an efficient feature separation, fusion and optimization block (SFOB) to effectively capture the local cross-level features through successive channel splitting and concatenation first, and then refine them with an improved channel attention mechanism. We also adopt a MAE pooling-based feature optimization and fusion block (MAE-FOFB) to enhance the distinction and utilization of global multi-level features extracted from every SFOB. For the loss function, except for L1 loss, the structural similarity (SSIM) loss is additionally introduced to fine-tune the network, which helps to bring a slight improvement in accuracy. Moreover, we develop a variant of SFON (SFON-P) by applying progressive reconstruction strategy to further boost performance. Extensive experiments show that both SFON and SFON-P achieve favorable reconstruction accuracy against other state-of-the-art lightweight models with relatively low model complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1122–1131, https://doi.org/10.1109/CVPRW.2017.150 (2017)

  2. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 252–268 (2018)

  3. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.: Low complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of British Machine Vision Conference (BMVC), p 1–10 (2012)

  4. Chu, X., Zhang, B., Xu, R., Ma, H.: Multi-objective reinforced evolution in mobile neural architecture search. 1901.01074 (2019)

  5. Chu, X., Zhang, B., Ma, H., Xu, R., Li, Q.: Fast, accurate and lightweight super-resolution with neural architecture search. 1901.07261 (2020)

  6. Dong, C., Loy, C. C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision. Springer, New York, pp 184–199 (2014)

  7. Dong, C., Loy, C. C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 391–407 (2016)

  8. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 770–778. https://doi.org/10.1109/CVPR.2016.90 (2016)

  9. He, Z., Cao, Y., Du, L., Xu, B., Zhuang, Y.: Mrfn: multi-receptive-field network for fast and accurate single image super-resolution. IEEE Trans. Multimed. PP(99), 1–1 (2019)

    Google Scholar 

  10. Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2011–2023 (2020)

    Article  Google Scholar 

  11. Hu, Y., Gao, X., Li, J., Huang, Y., Wang, H.: Single image super-resolution via cascaded multi-scale cross network. 1802.08808 (2021)

  12. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K. Q.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2261–2269, (2017) https://doi.org/10.1109/CVPR.2017.243

  13. Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5197–5206, (2015) https://doi.org/10.1109/CVPR.2015.7299156

  14. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 723–731 (2018)

  15. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM International Conference on Multimedia (MM’19), pp 2024–2032 (2019)

  16. Kim, J., Lee, J. K., Lee, K. M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1646–1654 (2016)

  17. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1637–1645 (2016)

  18. Kim, J.H., Choi, J.H., Cheon, M., Lee, J.S.: Mamnet: multi-path adaptive modulation network for image super-resolution. Neurocomputing 402, 38–49 (2020). https://doi.org/10.1016/j.neucom.2020.03.069

    Article  Google Scholar 

  19. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR), p 1–15 (2015)

  20. Lai, W., Huang, J., Ahuja, N., Yang, M.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5835–5843 (2017)

  21. Lan, R., Sun, L., Liu, Z., Lu, H., Pang, C., Luo, X.: Madnet: a fast and lightweight network for single-image super resolution. IEEE Trans. Cybern. 51(3), 1443–1453 (2021). https://doi.org/10.1109/TCYB.2020.2970104

    Article  Google Scholar 

  22. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 105–114 (2017)

  23. Li, B., Wang, B., Liu, J., Qi, Z., Shi, Y.: s-lwsr: Super lightweight super-resolution network. IEEE Trans. Image Process. 29, 8368–8380 (2020). https://doi.org/10.1109/TIP.2020.3014953

    Article  Google Scholar 

  24. Li, J., Fang, F., Mei, K., Zhang, G.: Multi-scale residual network for image super-resolution. In: European Conference on Computer Vision, pp 527–542 (2018)

  25. Li, W., Li, S., Liu, A.: Lightweight image super-resolution reconstruction with hierarchical feature-driven network. In: 2020 IEEE International Conference on Image Processing (ICIP), pp 573–577, (2020) https://doi.org/10.1109/ICIP40778.2020.9191110

  26. Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1132–1140 (2017)

  27. Liu, H., Cao, F., Wen, C., Zhang, Q.: Lightweight multi-scale residual networks with attention for image super-resolution. Knowl.-Based Syst. 203(4), 106103 (2020)

    Article  Google Scholar 

  28. Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. In: European Conference on Computer Vision, Springer, pp 41–55 (2020)

  29. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol 2, pp 416–423 (2001)

  30. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 4510–4520, (2018) https://doi.org/10.1109/CVPR.2018.00474

  31. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1874–1883 (2016)

  32. Sun, L., Liu, Z., Sun, X., Liu, L., Luo, X.: Lightweight image super-resolution via weighted multi-scale residual network. IEEE/CAA J. Autom. Sin. PP(99), 1–10 (2021)

    Google Scholar 

  33. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2790–2798 (2017)

  34. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 4549–4557, (2017) https://doi.org/10.1109/ICCV.2017.486

  35. Tian, C., Zhuge, R., Wu, Z., Xu, Y., Zuo, W., Chen, C., Lin, C.W.: Lightweight image super-resolution with enhanced cnn. Knowl.-Based Syst. (2020)

  36. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 4809–4817 (2017)

  37. Wenming, Y., Wei, W., Xuechen, Z., Shuifa, S., Qingmin, L.: Lightweight feature fusion network for single image super-resolution. IEEE Signal Process. Lett. (2019)

  38. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Computer Vision—ECCV 2018, pp. 3–19. Springer International Publishing, Cham (2018)

    Chapter  Google Scholar 

  39. Xu, W., Song, H., Zhang, K., Liu, Q., Liu, J.: Learning lightweight multi-scale feedback residual network for single image super-resolution. Comput. Vis. Image Underst. 197–198, 103005 (2020)

    Article  Google Scholar 

  40. Yan, C., Hao, Y., Li, L., Yin, J., Liu, A., Mao, Z., Chen, Z., Gao, X.: Task-adaptive attention for image captioning. IEEE Trans. Circ. Syst. Video Technol. (2021)

  41. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Proceedings of the International Conference on Curves and Surfaces, Springer, pp 711–730 (2010)

  42. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision, pp 286–301 (2018)

  43. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2472–2481 (2018)

  44. Zoph, B., Vasudevan, V., Shlens, J., QV, L.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 8697–8710 (2018)

Download references

Acknowledgements

This work was supported in part by the Fundamental Research Funds for the Central Universities (no. 292021000242), in part by National Key R&D Program of China (2017YFB0403604), in part by the National Natural Science Foundation of China (Grant nos. 61571416, 61072045, 61032006).

Author information

Authors and Affiliations

  1. University of Chinese Academy of Sciences, Beijing, China

    Lin Tian, Shaoshuai Gao & Guofang Tu

Authors
  1. Lin Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaoshuai Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guofang Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shaoshuai Gao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by Y. Zhang.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tian, L., Gao, S. & Tu, G. Lightweight feature separation, fusion and optimization networks for accurate image super-resolution. Multimedia Systems 28, 611–622 (2022). https://doi.org/10.1007/s00530-021-00862-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-021-00862-x

Keywords

Search

Navigation