Skip to main content
SpringerLink
Log in

FSFN: feature separation and fusion network for single image super-resolution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, image super-resolution (SR) based on deep learning technology has made significant progress. However, most methods are difficult to apply in real life because of their large parameters and heavy computation. Recently, residual learning has been widely applied to the problem of super-resolution. It can make the shallow features extracted from the input image act on each middle layer through long and short connection. Therefore, residual learning can be focused on processing high-frequency feature information, which significantly improves the SR performance of the network. However, with the improvement of network depth, the features that can be effectively utilized are still the shallow ones extracted from the input image. In this paper, we propose the feature separation and fusion network(FSFN). We further enrich the high-frequency feature information by separating and fusing the extracted and unextracted features in the internal shallow layer of each feature separation and fusion module. As the depth of the network increases, the shallow features extracted from the input image can be updated in a direction closer to those extracted from the real high-resolution image. A large number of experimental results show that this method has a strong performance compared with the existing SR algorithm with similar parameters and computation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Agustsson E, Timofte R (2017) NTIRE 2017 challenge on single image super-resolution: Dataset and study. In: CVPR, pp 1122–1131

  2. Ahn N, Kang B, Sohn K (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: ECCV, pp 256–272

  3. Allebach JP, Wong PW (1996) Edge-directed interpolation. In: Proceedings 1996 International conference on image processing, Lausanne, Switzerland, September 16-19, 1996, pp 707–710

  4. Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel M (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: British machine vision conference, BMVC 2012, Surrey, UK, September 3-7, 2012, pp 1–10

  5. Chang H, Yeung D, Xiong Y (2004) Super-resolution through neighbor embedding. In: CVPR, pp 275–282

  6. Chu X, Zhang B, Ma H, Xu R, Li J, Li Q (2019) Fast, accurate and lightweight super-resolution with neural architecture search. arXiv:1901.07261

  7. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38 (2):295–307

    Article  Google Scholar 

  8. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: ECCV, pp 391–407

  9. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778

  10. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: CVPR, pp 2261–2269

  11. Huang J, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: CVPR, pp 5197–5206

  12. Hui Z, Gao X, Yang Y, Wang X (2019) Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM International conference on multimedia, MM 2019, Nice, France, October 21-25, 2019, pp 2024–2032

  13. Irani M, Peleg S (1991) Improving resolution by image registration. CVGIP Graph Model. Image Process. 53(3):231–239

    Article  Google Scholar 

  14. Jin X, Xiong Q, Xiong C, Li Z, Gao Z (2019) Single image super-resolution with multi-level feature fusion recursive network. Neurocomputing 370:166–173

    Article  Google Scholar 

  15. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp 1646–1654

  16. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: CVPR, pp 1637–1645

  17. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: 3rd International conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  18. Lai W, Huang J, Ahuja N, Yang M (2019) Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(11):2599–2613

    Article  Google Scholar 

  19. Li X, Orchard MT (2000) New edge directed interpolation. In: Proceedings of the 2000 international conference on image processing, ICIP 2000, Vancouver, BC, Canada, September 10-13, 2000, pp 311–314

  20. Lim B, Son S, Kim H, Nah S, M Lee K (2017) Enhanced deep residual networks for single image super-resolution. In: CVPR, pp 1132–1140

  21. Liu B, Ait-Boudaoud D (2020) Effective image super resolution via hierarchical convolutional neural network. Neurocomputing 374:109–116

    Article  Google Scholar 

  22. Liu J, Zhang W, Tang Y, Tang J, Wu G (2020) Residual feature aggregation network for image super-resolution. In: CVPR, pp 2356–2365

  23. Martin DR, Fowlkes CC, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, pp 416–425

  24. Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, Aizawa K (2017) Sketch-based manga retrieval using manga109 dataset. Multim. Tools Appl. 76(20):21811–21838

    Article  Google Scholar 

  25. Qiu Y, Wang R, Tao D, Cheng J (2019) Embedded block residual network: A recursive restoration model for single-image super-resolution. In: ICCV, pp 4179–4188

  26. Ren C, He X, Pu Y, Nguyen TQ (2019) Enhanced non-local total variation model and multi-directional feature prediction prior for single image super resolution. IEEE Trans. Image Process. 28(8):3778–3793

    Article  MathSciNet  Google Scholar 

  27. Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp 1874–1883

  28. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: CVPR, pp 2790–2798

  29. Tai Y, Yang J, Liu X, Xu C (2017) Memnet: A persistent memory network for image restoration. In: ICCV, pp 4549–4557

  30. Tai Y, Yang J, Liu X, Xu C (2017) Memnet: A persistent memory network for image restoration. In: ICCV, pp 4549–4557

  31. Tong T, Li G, Liu X, Gao Q (2017) Image super-resolution using dense skip connections. In: ICCV, pp 4809–4817

  32. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4):600–612

    Article  Google Scholar 

  33. Wang X, Gu Y, Gao X, Hui Z (2019) Dual residual attention module network for single image super resolution. Neurocomputing 364:269–279

    Article  Google Scholar 

  34. Yang X, Mei H, Zhang J, Xu K, Yin B, Zhang Q, Wei X (2019) DRFN: deep recurrent fusion network for single-image super-resolution with large factors. IEEE Trans. Multimed 21(2):328–337

    Article  Google Scholar 

  35. Yang W, Wang W, Zhang X, Sun S, Liao Q (2019) Lightweight feature fusion network for single image super-resolution. IEEE Signal Process. Lett. 26 (4):538–542

    Article  Google Scholar 

  36. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: Curves and surfaces - 7th international conference, Avignon, France, June 24-30, 2010, Revised Selected Papers, pp 711–730

  37. Zhang K, Gool LV, Timofte R (2020) Deep unfolding network for image super-resolution. In: CVPR, pp 3214–3223

  38. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: ECCV, pp 294–310

  39. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: ECCV, pp 294–310

  40. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: CVPR, pp 2472–2481

  41. Zhang Z, Wang X, Jung C (2019) DCSR: dilated convolutions for single image super-resolution. IEEE Trans. Image Process. 28(4):1625–1635

    Article  MathSciNet  Google Scholar 

  42. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15 (8):2226–2238

    Article  Google Scholar 

Download references

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China (61876099), in part by the National Key R&D Program of China (2019YFB1311001), in part by the National Natural Science Foundation of China (U1806202), in part by the National Natural Science Foundation of China (61533011), in part by the Scientific and Technological Development Project of Shandong Province (2019GSF111002), in part by the Shenzhen Science and Technology Research and Development Funds (JCYJ20180305164401921), in part by the Foundation of Ministry of Education Key Laboratory of System Control and Information Processing (Scip201801), and in part by the Foundation of State Key Laboratory of Integrated Services Networks (ISN20-06). Kai Zhu and Zhenxue Chen contributed equally to this work and should be considered as the co-first authors.

Author information

Author notes

    Authors and Affiliations

    1. School of Control Science and Engineering, Shandong University, Jinan, 250061, China

      Kai Zhu, Zhenxue Chen, Jie Zhao & Gan Zhang

    2. State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, 710071, China

      Zhenxue Chen & Nannan Wang

    3. Department of Electrical and Computer Engineering, University of Windsor, Windsor, N9B 3P4, Canada

      Q. M. Jonathan Wu

    4. Shenzhen Research Institute of Shandong University, Shandong University, Shenzhen, 518057, China

      Jie Zhao

    Authors
    1. Kai Zhu
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Zhenxue Chen
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Q. M. Jonathan Wu
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Nannan Wang
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Jie Zhao
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Gan Zhang
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Zhenxue Chen.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Kai Zhu and Zhenxue Chen have contributed equally.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Zhu, K., Chen, Z., Wu, Q.M.J. et al. FSFN: feature separation and fusion network for single image super-resolution. Multimed Tools Appl 80, 31599–31618 (2021). https://doi.org/10.1007/s11042-021-11121-6

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-021-11121-6

    Keywords

    Search

    Navigation