Skip to main content
SpringerLink
Log in

Multi-orientation depthwise extraction for stereo image super-resolution

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The increasing trend of binocular imaging in recent years has sparked a surge of interest in stereo image super-resolution. While considerable progress has been made in improving model performance, the potential of single-view and cross-view features remains largely unexplored. To address this, we present a novel network that incorporates both intra-view and inter-view feature extraction to enhance stereo image super-resolution. Specially, we design a multi-orientation depthwise extraction module to sufficiently extract various orientation features within a single view. Additionally, a cross focus module is proposed to capture more reliable hierarchical cross-view features. These modules can be integrated together to exploit trustworthier complementary information for HR image reconstruction. Our experimental results showcase the excellent performance of our method, surpassing all previous state-of-the-art methods for stereo image super-resolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The data generated during the current study are available from the corresponding author on reasonable request.

References

  1. Jeon, D.S., Baek, S., Choi, I., Kim, M.H.: Enhancing the spatial resolution of stereo images using a parallax prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1721–1730. (2018)

  2. Wang, L., Wang, Y., Liang, Z., Lin, Z., Yang, J., An, W., Guo, Y.: Learning parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12250–12259. (2019)

  3. Chen, C., Qing, C., Xu, X., Dickinson, P.: Cross parallax attention network for stereo image super-resolution. IEEE Trans. Multimed. 24, 202–216 (2021)

    Article  Google Scholar 

  4. He, Z., Chen, D., Cao, Y., Yang, J., Cao, Y., Li, X., Tang, S., Zhuang, Y., Lu, Z.: Single image super-resolution based on progressive fusion of orientation-aware features. Pattern Recognit. 133, 109038 (2023)

    Article  Google Scholar 

  5. Li, L., Xu, M., Liu, H., Li, Y., Wang, X., Jiang, L., Wang, Z., Fan, X., Wang, N.: A large-scale database and a cnn model for attention-based glaucoma detection. IEEE Trans. Med. Imaging 39(2), 413–424 (2019)

    Article  Google Scholar 

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  7. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  8. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144. (2017)

  9. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301. (2018)

  10. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1664–1673. (2018)

  11. Han, W., Chang, S., Liu, D., Yu, M., Witbrock, M., Huang, T.S.: Image super-resolution via dual-state recurrent networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1654–1663. (2018)

  12. Shocher, A., Cohen, N., Irani, M.: “zero-shot” super-resolution using deep internal learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3118–3126. (2018)

  13. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262–3271. (2018)

  14. Ying, X., Wang, Y., Wang, L., Sheng, W., An, W., Guo, Y.: A stereo attention module for stereo image super-resolution. IEEE Signal Process Lett. 27, 496–500 (2020)

    Article  Google Scholar 

  15. Wang, Y., Ying, X., Wang, L., Yang, J., An, W., Guo, Y.: Symmetric parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 766–775. (2021)

  16. Dai, Q., Li, J., Yi, Q., Fang, F., Zhang, G.: Feedback network for mutually boosted stereo image super-resolution and disparity estimation. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 1985–1993 (2021)

  17. Chu, X., Chen, L., Yu, W.: Nafssr: stereo image super-resolution using nafnet. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1239–1248 (2022)

  18. Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pp. 17–33. Springer, (2022)

  19. Wang, L., Guo, Y., Wang, Y., Li, J., Gu, S., Timofte, R.: Ntire 2022 challenge on stereo image super-resolution: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 906–919. (2022)

  20. Lin, J., Yin, L., Wang, Y.: Steformer: Efficient stereo image super-resolution with transformer. IEEE Trans. Multimedia (2023)

  21. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883. (2016)

  22. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE, (2012)

  23. Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3061–3070. (2015)

  24. Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., Westling, P.: High-resolution stereo datasets with subpixel-accurate ground truth. In: German Conference on Pattern Recognition, pp. 31–42. Springer, (2014)

  25. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

  26. Wang, Y., Wang, L., Yang, J., An, W., Guo, Y.: Flickr1024: A large-scale dataset for stereo image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, (2019)

  27. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481. (2018)

  28. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: Image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833–1844. (2021)

  29. Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)

Download references

Funding

This work was supported by the National Natural Science Foundation of China (61303093, 61402278) and the Shanghai Natural Science Foundation (19ZR1419100).

Author information

Authors and Affiliations

  1. College of Shanghai Film, Shanghai University, 788 Guangzhong Road, Shanghai, 200072, China

    Xiangyang Fan, Feifan Cai, Jinhua Liu, Yuhang Li, Liting Huang & Youdong Ding

  2. School of Life Sciences, Shanghai University, 99 Shangda Road, Shanghai, 200444, China

    Renjia Ye

  3. Shanghai Engineering Research Center of Motion Picture Special Effects, 788 Guangzhong Road, Shanghai, 200072, China

    Youdong Ding

Authors
  1. Xiangyang Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Renjia Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feifan Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinhua Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuhang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Liting Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Youdong Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The first draft of the manuscript was written by Xiangyang Fan and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Youdong Ding.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, X., Ye, R., Cai, F. et al. Multi-orientation depthwise extraction for stereo image super-resolution. SIViP 17, 4087–4095 (2023). https://doi.org/10.1007/s11760-023-02640-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02640-w

Keywords

Search

Navigation