Skip to main content
SpringerLink
Log in

MBMR-Net: multi-branches multi-resolution cross-projection network for single image super-resolution

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Deep convolutional neural networks (CNNs) have achieved significant developments in the field of single image super resolution (SISR) due to their nonlinear expression ability. However, existing architectures either rely on the representations learned from a single scale or extract deep features by cascading multiple resolutions, which unuse or underutilize the interdependence between low-resolution (LR) images and high-resolution (HR) images. In view of this trait, we propose a deep network called the multi-branches multi-resolution cross-projection network (MBMR-Net), which has multiple parallel branches, and cross-projection is performed between multiple branches to exchange information. Then, we introduce a novel attention unit that integrates second-order channel attention with spatial attention to better fuse information from multiple resolutions. Moreover, in terms of the characteristics of the model, we devise a loss function for enhancing the restoration of high-frequency details while ensuring the content information. Extensive quantitative and qualitative evaluations on benchmark datasets illustrate the effectiveness of our method and its competitive performance over state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Bulat A, Tzimiropoulos G (2018) Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 109–117

    Google Scholar 

  2. Chen Y, Tai Y, Liu X, Shen C, Yang J (2018) FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 2492–2501

    Google Scholar 

  3. Pham CH, Ducournau A, Fablet R, Rousseau F (2017) Brain MRI Super-Resolution using Deep 3D Convolutional Networks. In: IEEE International Symposium on Biomedical Imaging, pp 197–200

    Google Scholar 

  4. Lyu Q, Shan H, Steber C, Helis C, Wang G (2020) Multi-Contrast Super-Resolution MRI Through a Progressive Network. IEEE Transactions on Medical Imaging 39(9):2738–2749

    Article  Google Scholar 

  5. Rasti P, Uiboupin T, Escalera S, Anbarjafari G (2016) Convolutional Neural Network Super Resolution for Face Recognition in Surveillance Monitoring. In: International Conference on Articulated Motion and Deformable Objects, pp 175–184

    Chapter  Google Scholar 

  6. Xun Y, Zhou P, Wang M (2018) Person Reidentification via Structural Deep Metric Learning. IEEE Transactions on Neural Networks and Learning System 30(10):2987–2998

    Google Scholar 

  7. Chao D, Chen CL, He K, Tang X (2014) Learning a Deep Convolutional Network for Image Super-Resolution. In: European Conference on Computer Vision (ECCV), pp 184–199

    Google Scholar 

  8. Chao D, Chen CL, Tang X (2016) Accelerating the Super-Resolution Convolutional Neural Network. In: European Conference on Computer Vision (ECCV), pp 391–407

    Google Scholar 

  9. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: IEEE conference on computer vision and pattern recognition, pp 1646–1654

    Google Scholar 

  10. Mao XJ, Shen C, Yang YB (2016) Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in neural information processing systems (NIPS)

    Google Scholar 

  11. Ying T, Jian Y, Liu X (2017) Image Super-Resolution via Deep Recursive Residual Network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2790–2798

    Google Scholar 

  12. Tai Y, Yang J, Liu X, Xu C (2017) MemNet: A Persistent Memory Network for Image Restoration. In: IEEE International Conference on Computer Vision (ICCV), pp 4549–4557

    Chapter  Google Scholar 

  13. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced Deep Residual Networks for Single Image Super-Resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1132–1140

    Google Scholar 

  14. Lai WS, Huang JB, Ahuja N, Yang MH (2017) Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5835–5843

    Google Scholar 

  15. Haris M, Shakhnarovich G, Ukita NJA (2018) Deep Back-Projection Networks For Super-Resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1664–1673

    Google Scholar 

  16. Zhu F, Zhao Q (2019) Efficient Single Image Super-Resolution via Hybrid Residual Feature Learning with Compact Back-Projection Network. In: IEEE International Conference on Computer Vision Workshop (ICCVW), pp 2453–2460

    Google Scholar 

  17. Jie H, Li S, Gang S, Albanie S Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(8, 2017):2011–2023

  18. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In: European Conference on Computer Vision, pp 294–310

    Google Scholar 

  19. Hu Y, Li J, Huang Y, Gao X (2019) Channel-Wise and Spatial Feature Modulation Network for Single Image Super-Resolution. In: IEEE Transactions on Circuits and Systems for Video Technology, pp 3911–3927

    Google Scholar 

  20. Dai T, Cai J, Zhang Y, Xia ST, Zhang L (2019) Second-order Attention Network for Single Image Super-Resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 11057–11066

    Google Scholar 

  21. Gao Z, Xie J, Wang Q, Li P (2020) Global Second-Order Pooling Convolutional Networks. In: IEEEConference on Computer Vision and Pattern Recognition (CVPR), pp 3019–3028

    Google Scholar 

  22. Fe W, Jiang IM, Chen Q, Yang S, Tang X (2017) Residual Attention Network for Image Classification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 6450–6458

    Google Scholar 

  23. Li K, Wu Z, Peng KC, Ernst J, Fu Y (2018) Tell Me Where to Look: Guided Attention Inference Network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 9215–9223

    Google Scholar 

  24. Zhou W, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  25. Kim J, Lee JK, Lee KM (2016) Deeply-Recursive Convolutional Network for Image Super-Resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5016–5023

    Google Scholar 

  26. He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

    Google Scholar 

  27. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z (2016) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 105–114

    Google Scholar 

  28. Huang G, Liu Z, Pleiss G, Van DML, Weinberger K (2019) Convolutional networks with dense connectivity. In: IEEE transactions on pattern analysis machine intelligence

    Google Scholar 

  29. Tong T, Li G, Liu X, Gao Q (2017) Image Super-Resolution Using Dense Skip Connections. In: IEEE International Conference on Computer Vision (ICCV), pp 4809–4817

    Chapter  Google Scholar 

  30. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative Adversarial Networks. In: Advances in Neural Information Processing Systems (NIPS), pp 2672–2680

    Google Scholar 

  31. Haris M, Shakhnarovich G, Ukita N (2021) Deep Back-ProjectiNetworks for Single Image Super-Resolution. IEEE Transactions on Pattern Analysis Machine Intelligence 43(12):4323–4337

    Article  Google Scholar 

  32. Wang Z, Chen J, Hoi S (2020) Deep Learning for Image Super-resolution: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(10):3365–3387

    Article  Google Scholar 

  33. Qin D, Gu X (2019) Deep Residual-Dense Attention Network for Image Super-Resolution. In: International Conference on Neural Information Processing, pp 3–10

    Chapter  Google Scholar 

  34. Y. Zhang, K. Li, K. Li, B. Zhong, Y. Fu, Residual non-local attention networks for image restoration, 2019. https://arxiv.org/pdf/1903.10082v1.pdf

    Google Scholar 

  35. Wang J, Sun K, Cheng T, Jiang B, Xiao B (2019) Deep High-Resolution Representation Learning for Visual Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(10):3349–3364

    Article  Google Scholar 

  36. Wang Q, Li P, Zuo W, Lei Z (2016) RAID-G: Robust Estimation of Approximate Infinite Dimensional Gaussian with Application to Material Recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 4433–4441

    Google Scholar 

  37. Koniusz P, Fei Y, Gosselin PH, Mikolajczyk K (2017) Higher-order Occurrence Pooling for Bags-of-Words: Visual Concept Detection. IEEE Transactions on Pattern Analysis Machine Intelligence 39(2):313–326

    Article  Google Scholar 

  38. Li P, Xie J, Wang Q, Zuo W (2017) Is second-order information helpful for large-scale visual recognition? In: IEEE international conference on computer vision (ICCV), pp 2089–2097

    Chapter  Google Scholar 

  39. Zhao H, Gallo O, Frosio L, Kautz J (2017) Loss Functions for Image Restoration With Neural Networks. IEEE Transactions on Computational Imaging 3(1):47–57

    Article  Google Scholar 

  40. Yuan Y, Liu S, Zhang J, Zhang Y, Dong C, Lin L (2018) Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networks. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 814–823

    Google Scholar 

  41. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In: IEEE International Conference on Computer Vision (ICCV), pp 2242–2251

    Chapter  Google Scholar 

  42. Agustsson E, Timofte R, NTIRE (2017) Challenge on single image super-resolution: dataset and study. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), vol 2017, pp 1122–1131

  43. Bevilacqua M, Roumy A, Guillemot C, Morel A (2012) Low-Complexity Single Image Super-Resolution Based on Nonnegative Neighbor Embedding. In: British Machine Vision Conference, pp 1–10

    Google Scholar 

  44. Zey De R, Elad M, Protter M (2010) On Single Image Scale-Up Using Sparse-Representations. Springer, pp 711–730

    MATH  Google Scholar 

  45. Martin D, Fowlkes C, Tal D, Malik J (2002) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE International Conference on Computer Vision (ICCV), pp 416–423

    Google Scholar 

  46. Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 5197–5206

    Google Scholar 

  47. Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, Aizawa K (2017) Sketch-based Manga Retrieval using Manga109 Dataset. Multimedia Tools and Applications 76(20):21811–21838

    Article  Google Scholar 

  48. Kingma D, Ba JJCS (2014) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR)

    Google Scholar 

  49. Shi W, Caballero J, Huszár F, Totz J, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1874–1883

    Google Scholar 

  50. Fang F, Li J, Zeng T (2020) Soft-edge Assisted Network for Single Image Super-Resolution. IEEE Transactions on Image Processing:4656–4668

  51. Li J, Fang F, Mei K, Zhang G (2018) Multi-scale Residual Network for Image Super-Resolution. In: European Conference on Computer Vision (ECCV), pp 527–542

    Google Scholar 

  52. Li F, Bai H, Zhao Y (2020) FilterNet: Adaptive Information Filtering Network for Accurate and Fast Image Super-Resolution. IEEE Transactions on Circuits Systems for Video Technology 30(6):1511–1523

    Article  Google Scholar 

  53. He Z, Cao Y, Du L, Xu B, Zhuang Y (2019) MRFN: Multi-Receptive-Field Network for Fast and Accurate Single Image Super-Resolution. IEEE Transactions on Multimedia 22(4):1042–1054

    Article  Google Scholar 

  54. R. Timofte, E. Agustsson, L.V. Gool, e. al., NTIRE 2017 Challenge on single image super-resolution: methods and results, in: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017, pp. 1110–1121

Download references

Author information

Authors and Affiliations

  1. School of Microelectronics and Communication Engineering, Chongqing University, Chongqing, China

    Dan Zhang, Binglian Zhu & Yuanhong Zhong

Authors
  1. Dan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Binglian Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuanhong Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuanhong Zhong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, D., Zhu, B. & Zhong, Y. MBMR-Net: multi-branches multi-resolution cross-projection network for single image super-resolution. Appl Intell 52, 12975–12989 (2022). https://doi.org/10.1007/s10489-022-03322-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03322-9

Keywords

Search

Navigation