Skip to main content
SpringerLink
Log in

Multipath feedforward network for single image super-resolution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Single image super-resolution (SR) models which based on convolutional neural network mostly use chained stacking to build the network. It ignores the role of hierarchical features and relationship between layers, resulting in the loss of high-frequency components. To address these drawbacks, we introduce a novel multipath feedforward network (MFNet) based on staged feature fusion unit (SFF). By changing the connection between networks, MFNet strengthens the inter-layer relationship and improves the information flow in the network, thereby extracting more abundant high-frequency components. Firstly, SFF extracts and integrates hierarchical features by dense connection, which expands the information flow of the network. Afterwards, we use adaptive method to learn effective features in hierarchical features. Then, in order to strengthen relationship between layers and fully use the hierarchical features, we use multi-feedforward structure to connect each SFF, which enables multipath feature re-usage and explores more abundant high-frequency components on this basis. Finally, the image reconstruction is realized by combining the shallow features and the global residual. Extensive benchmark evaluation shows that the performance of MFNet has a significant improvement over the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Bevilacqua M, Roumy A, Guillemot C, Morel M-A (2012) Low-complexity single image super-resolution based on nonnegative neighbor embedding. In: British Machine Vision Conference (BMVC). 1–10

  2. Chen Y, Li J, Xiao H, Jin X, Yan S, Feng J (2017) Dual path networks. In: Neural Information Processing Systems (NIPS). 4467–4475

  3. Cheong JY, Park IK (2017) Deep CNN-based super-resolution using external and internal examples. IEEE Sig Proc Lett 24(8):1252–1256

    Article  Google Scholar 

  4. Cho D, Tai YW, KweonI (2016) Natural image matting using deep convolutional neural networks. In: European Conference on Computer Vision (ECCV). 626–643

  5. Dai S, Han M, Xu W, Wu Y, Gong Y, Katsaggelos AK (2009) Softcuts: a soft edge smoothness prior for color image super-resolution. IEEE Trans Image Process 18(5):969–981

    Article  MathSciNet  MATH  Google Scholar 

  6. Dong C, Chen CL, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision (ECCV). 184–199

  7. Dong C, Chen CL, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  8. Dong C, Chen CL,Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision (ECCV). 391–407

  9. Gregor K, Yann L (2010) Learning fast approximations of sparse coding. In: International Conference on Machine Learning (ICML). 399–406

  10. Gu S, Zuo W, Xie Q, Meng D, Feng X,Zhang L (2015) Convolutional sparse coding for image super resolution. In: IEEE Int Conf Comput Vis (ICCV). 1823–1831

  11. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: IEEE Int Conf Comput Vis (ICCV). 1026–1034

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE Conf Comput Vis Patt Recog (CVPR). 770–778

  13. Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. IEEE Conf Comput Vis Patt Recog (CVPR):5197–5206

  14. Huang G, Liu Z, Weinberger KQ, Maaten L-vd (2017) Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2261–2269

  15. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. IEEE Conf Comput Vis Patt Recog (CVPR):1637–1645

  16. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: IEEE Conf Comput Vis Patt Recog (CVPR). 1646–1654

  17. Lai W, Huang J, Ahuja N, Yang M (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: IEEE Conf Comput Vis Patt Recog (CVPR). 5835–5843

  18. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conf Comput Vis Patt Recog (CVPR). 105–114

  19. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527

    Article  Google Scholar 

  20. Li S, Fan R, Lei G, Hou G (2018) A two-channel convolutional neural network for image super-resolution. Neurocomputing 275:267–277

    Article  Google Scholar 

  21. Mao X, Shen C, Yang Y (2016) Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Neural Information Processing Systems (NIPS). 2802–2810

  22. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. 416–423

  23. Qian N (1999) On the momentum term in gradient descent learning algorithms. Neural Netw 12(1):145–151

    Article  MathSciNet  Google Scholar 

  24. Schulter S, Leistner C, Bischof H (2015) Fast and accurate image upscaling with super-resolution forests. In:IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3791–3799

  25. Shi W, Caballero J, Huszar F, Totz J, Aitken A-P, Bishop R, Rueckert D, Wang Z (2016) Real-Time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1874–1883

  26. Shi W, Jiang F, Zhao D (2017) Single image super-resolution with dilated convolution based multi-scale information learning inception module. In: IEEE International Conference on Image Processing (ICIP). 977–981

  27. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR)

  28. Sun J, Sun J, Xu Z, Shum H (2008) Image super-resolution using gradient profile prior. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1–8

  29. SzegedyC LW, Jia Y, Sermanet P, Reed S (2015) Going deeper with convolutions. IEEE Conf Comput Vis Patt Recog (CVPR):1–9

  30. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: IEEE Conf Comput Vis Patt Recog (CVPR). 2790–2798

  31. Tai Y, Yang Y, Liu X, Xu C (2017) MemNet: A persistent memory network for image restoration. In: IEEE Int Conf Comput Vis (ICCV). 4549–4557

  32. Tang Y, Chen H, Liu Z, Song B, Wang Q (2016) Example-based super-resolution via social images. Neurocomputing 172:38–47

    Article  Google Scholar 

  33. Timofte R, De V, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: Proc IEEE International Conference on Computer Vision (ICCV). 1920–1927

  34. Timofte R, De V, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: Proc. Asian Conference on Computer Vision (ACCV), pp 111–126

  35. Wang Q, Yuan Y (2014) High quality image resizing [J]. Neurocomputing 131:348–356

    Article  Google Scholar 

  36. Wang Z, Yang Y, Wang Z, Chang S, Han W, Yang J, Huang T (2015) Self-tuned deep super resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1–8

  37. Wang Z, Liu D, Wen B, Yang J, Han W, Huang T-S (2015) Deep networks for image super-resolution with sparse prior. In: IEEE Int Conf Comput Vis (ICCV). 370–378

  38. Wang Y, Wang L, Wang H, Li P (2016) End-to-End image super resolution via deep and shallow convolutional networksarXiv preprint arXiv:1607.07680

  39. Xu J, Zhao Y, Dong Y, Bai H (2017) Fast and accurate image super-resolution using a combined loss. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1093–1099

  40. Xu J, Chae Y, Stenger B (2017) BYNET-SR: Image super resolution with a bypass connection network. In: IEEE International Conference on Image Processing (ICIP). 4053–4057

  41. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  42. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: Proceedings of the 7th International Conference on Curves and Surfaces. 711–730

  43. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

  44. Zhang K, Zuo W, Gu S, Zhang L (2017) Learning deep CNN denoiser prior for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2808–2817

  45. Zhang K, Zuo W, Chen Y, Meng D, Zhang L (2017) Beyond a Gaussian Denoiser: residual learning of deep CNN for image Denoising. IEEE Trans Image Process 26(7):3142–3155

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We express our sincere thanks to the anonymous reviewers for their helpful comments and suggestions to raise the standard of our paper.

Funding

This study was funded by the National Natural Science Foundation of China under (grant number 61672202).

Author information

Authors and Affiliations

  1. College of Computer and Information, Hefei University of Technology, Hefei, China

    Mingyu Shen, Pengfei Yu, Ronggui Wang, Juan Yang, Lixia Xue & Min Hu

Authors
  1. Mingyu Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengfei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ronggui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lixia Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Min Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Yang.

Ethics declarations

Conflict of interest

The whole authors are fulltime teachers of Hefei University of Technology besides the second author Pengfei Yu, and he is the fulltime student of Hefei University of Technology. The whole authors declare that we have no conflicts of interest to this work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shen, M., Yu, P., Wang, R. et al. Multipath feedforward network for single image super-resolution. Multimed Tools Appl 78, 19621–19640 (2019). https://doi.org/10.1007/s11042-019-7334-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7334-9

Keywords

Search

Navigation