Skip to main content
SpringerLink
Log in

Accurate stereo image super-resolution using spatial-attention-enhance residual network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Stereo images can improve the performance of super-resolution (SR) by providing additional information from another viewpoint. However, the existing CNN-based stereo SR methods guide the reconstruction of high-frequency features in an indirect way, which hinders the network representation. In order to solve the issue, we firstly introduce spatial attention mechanism into stereo SR and propose the corresponding spatial-attention-enhance module (SAEM). The SAEM can capture spatial-wise feature correlations and directly guides the high-frequency feature reconstruction in the spatial dimension. This paper presents a novel spatial-attention-enhance super-resolution network (SAESRnet) for stereo images. The network representation is enhanced by SAEM, as extensive experiments show that our SAESRnet can achieve better accuracy and visual improvements against other existing stereo SR methods. Our method can outperform PASSRnet by 0.30 dB, 0.26 dB, and 0.26 dB respectively in the term of PSNR on Middlebury, KITTI2012, and KITTI2015 test datasets. In addition, the results of experiments also prove that our SAEM can also be possible to have a positive effect on improving the performance of single image super-resolution (SISR).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Ahn N, Kang B, Sohn K (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European conference on computer vision (ECCV), pp 252–268

  2. Barzegar S, Sharifi A, Manthouri M (2020) Super-resolution using lightweight detailnet network. Multimed Tools Appl 79:1119–1136

    Article  Google Scholar 

  3. Bhavsar AV, Rajagopalan AN (2010) Resolution enhancement in multi-image stereo. IEEE Trans Pattern AnalMach Intell 32(9):1721–1728

    Article  Google Scholar 

  4. Chang K, Ding PLK, Li B (2018) Single image super resolution using joint regularization. IEEE Signal Proc Lett 25(4):596–600

    Article  Google Scholar 

  5. Chen C, Qing C, Xu X, Dickinson P (2021) Cross parallax attention network for stereo image super-resolution. IEEE Trans Multimed 24:202–216

  6. Chu J, Zhang J, Lu W, Huang X (2018) A novel multiconnected convolutional network for super-resolution. IEEE Signal Proc Lett 25(7):946–950

    Article  Google Scholar 

  7. Dai T, Cai J, Zhang Y, Xia S.-T, Zhang L (2019) Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 11065–11074

  8. Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision. Springer, pp 184–199

  9. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 3354–3361

  10. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1664–1673

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  12. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  13. Huang J (2020) Image super-resolution reconstruction based on generative adversarial network model with double discriminators. Multimed Tools Appl 79:29639–29662

    Article  Google Scholar 

  14. Jeon DS, Baek S-H, Choi I, Kim MH (2018) Enhancing the spatial resolution of stereo images using a parallax prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1721–1730

  15. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  16. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645

  17. Kingma D, Ba J (2015) Adam: a method for stochastic optimization. In: International Conference on Learning Representations (Poster)

  18. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5835–5843

  19. Li K, Wu Z, Peng K-C, Ernst J, Fu Y (2018) Tell me where to look: Guided attention inference network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 9215–9223

  20. Li F, Cong R, Bai H, He Y (2020) Deep interleaved network for single image super-resolution with asymmetric co-attention. In: IJCAI, pp 537–543

  21. Lim B, Son S, Kim H, Nah Sm, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1132–1140

  22. Menze M, Geiger A (2015) Object scene flow for autonomous vehicles. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3061–3070

  23. Scharstein D (2014) H. Hirschm¨uller, Y. Kitajima, G. Krathwohl, N. Nesic, X. Wang and P. Westling, “high-resolution stereo datasets with subpixel-accurate ground truth,” in German Conf. For. Pattern Recogn 8753:31–42

  24. Shen P, Zhang L, Wang M, Yin G (2021) Deeper super-resolution generative adversarial network with gradient penalty for sonar image enhancement. Multimed Tools Appl 80:28087–28107

    Article  Google Scholar 

  25. Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645

  26. Song Z, Zhao X, Jiang H (2021) Gradual deep residual network for super-resolution. Multimed Tools Appl 80:9765–9778

    Article  Google Scholar 

  27. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2790–2798

  28. Wang F, Jiang M, Qian C, Yang S, C. Li, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6450–6458

  29. Wang L, Wang Y, Liang Z, Lin Z, Yang J, An W, Guo Y (2019) Learning parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12242–12251

  30. Wang Y, Wang L, Yang J, An W, Guo Y (2019) Flickr1024: A large-scale dataset for stereo image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops

  31. Woo S, Park J, Lee J-Y, Kweon IS (2018) CBAM: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  32. Yang H, Tong J, Dou Q, Xiao L, Jeon G, Yang X (2021) Wide receptive field networks for single image super-resolution. Multimed Tools Appl 81:4859–4876

    Article  Google Scholar 

  33. Ying X, Wang Y, Wang L, Sheng W, An W, Guo Y (2020) A stereo attention module for stereo image super-resolution. IEEE Signal Proc Lett 27:496–500

    Article  Google Scholar 

  34. Zhang Y, Li K, Li K, Wang L, Zhong B and Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computervision (ECCV), pp 286–301

  35. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y(2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481

  36. Zhu X, Guo K, Fang H, Chen L, Ren S Hu B (n.d.) “Cross View Capture for Stereo Image Super-Resolution,” in IEEE Transactions on Multimedia, https://doi.org/10.1109/TMM.2021.3092571

Download references

Acknowledgments

This research was supported by the National Natural Science Foundation of China under Grant No. 62072405 and Zhejiang Provincial Natural Science Foundation of China under Grant No. LGF20F020017.

Author information

Authors and Affiliations

  1. Zhejiang University of Technology, Hangzhou, China

    Wenyuan Ying, Tianyang Dong & Chen Shentu

Authors
  1. Wenyuan Ying
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tianyang Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chen Shentu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tianyang Dong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ying, W., Dong, T. & Shentu, C. Accurate stereo image super-resolution using spatial-attention-enhance residual network. Multimed Tools Appl 82, 12117–12133 (2023). https://doi.org/10.1007/s11042-022-13815-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13815-x

Keywords

Search

Navigation