Skip to main content
SpringerLink
Log in

SGNet: Semantics Guided Deep Stereo Matching

  • Conference paper
  • First Online:
Computer Vision – ACCV 2020 (ACCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12622))

Included in the following conference series:

Abstract

Stereovision has been an intensive research area of computer vision. Based on deep learning, stereo matching networks are becoming popular in recent years. Despite of great progress, it’s still challenging to achieve high accurate disparity map due to low texture and illumination changes in the scene. High-level semantic information can be helpful to handle these problems. In this paper a deep semantics guided stereo matching network (SGNet) is proposed. Apart from necessary semantic branch, three semantic guided modules are proposed to embed semantic constraints on matching. The joint confidence module produces confidence of cost volume based on the consistency of disparity and semantic features between left and right images. The residual module is responsible for optimizing the initial disparity results according to its semantic categories. Finally, in the loss module, the smooth of disparity is well supervised based on semantic boundary and region. The proposed network has been evaluated on various public datasets like KITTI 2015, KITTI 2012 and Virtual KITTI, and achieves the state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Schmid, K., Tomic, T., Ruess, F., Hirschmüller, H., Suppa, M.: Stereo vision based indoor/outdoor navigation for flying robots. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3955–3962. IEEE (2013)

    Google Scholar 

  2. Chen, C., Seff, A., Kornhauser, A., Xiao, J.: DeepDriving: learning affordance for direct perception in autonomous driving. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2722–2730. IEEE (2015)

    Google Scholar 

  3. Žbontar, J., LeCun, Y.: Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 17, 2287–2318 (2016)

    MATH  Google Scholar 

  4. Shaked, A., Wolf, L.: Improved stereo matching with constant highway networks and reflective confidence learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4641–4650. IEEE (2017)

    Google Scholar 

  5. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440. IEEE (2015)

    Google Scholar 

  6. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890. IEEE (2017)

    Google Scholar 

  7. Cheng, J., Tsai, Y.H., Wang, S., Yang, M.H.: SegFlow: joint learning for video object segmentation and optical flow. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 686–695. IEEE (2017)

    Google Scholar 

  8. Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., Yang, J.: Joint task-recursive learning for semantic segmentation and depth estimation. In: Proceedings of the European Conference on Computer Vision, pp. 235–251 (2018)

    Google Scholar 

  9. Jiao, J., Cao, Y., Song, Y., Lau, R.: Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss. In: Proceedings of the European Conference on Computer Vision, pp. 53–69 (2018)

    Google Scholar 

  10. Chen, P.Y., Liu, A.H., Liu, Y.C., Wang, Y.C.F.: Towards scene understanding: unsupervised monocular depth estimation with semantic-aware representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2624–2632. IEEE (2019)

    Google Scholar 

  11. Zou, N., Xiang, Z., Chen, Y., Chen, S., Qiao, C.: Simultaneous semantic segmentation and depth completion with constraint of boundary. Sensors 20, 635 (2020)

    Article  Google Scholar 

  12. Yang, G., Zhao, H., Shi, J., Deng, Z., Jia, J.: SegStereo: exploiting semantic information for disparity estimation. In: Proceedings of the European Conference on Computer Vision, pp. 636–651 (2018)

    Google Scholar 

  13. Zhang, J., Skinner, K.A., Vasudevan, R., Johnson-Roberson, M.: DispSegNet: leveraging semantics for end-to-end learning of disparity estimation from stereo imagery. IEEE Robot. Autom. Lett. 4, 1162–1169 (2019)

    Article  Google Scholar 

  14. Wu, Z., Wu, X., Zhang, X., Wang, S., Ju, L.: Semantic stereo matching with pyramid cost volumes. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7484–7493. IEEE (2019)

    Google Scholar 

  15. Chang, J.R., Chen, Y.S.: Pyramid stereo matching network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5410–5418. IEEE (2018)

    Google Scholar 

  16. Scharstein, D., Szeliski, R., Zabih, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2001). https://doi.org/10.1023/A:1014573219977

    Article  MATH  Google Scholar 

  17. Kanade, T., Kano, H., Kimura, S., Yoshida, A., Oda, K.: Development of a video-rate stereo machine. In: Proceedings of the 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots, vol. 3, pp. 95–100. IEEE (1995)

    Google Scholar 

  18. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 1222–1239 (2001)

    Article  Google Scholar 

  19. Klaus, A., Sormann, M., Karner, K.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: 18th International Conference on Pattern Recognition (ICPR 2006), vol. 3, pp. 15–18. IEEE (2006)

    Google Scholar 

  20. Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766. IEEE (2015)

    Google Scholar 

  21. Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4040–4048. IEEE (2016)

    Google Scholar 

  22. Pang, J., Sun, W., Ren, J.S., Yang, C., Yan, Q.: Cascade residual learning: a two-stage convolutional neural network for stereo matching. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 887–895. IEEE (2017)

    Google Scholar 

  23. Xu, H., Zhang, J.: AANet: adaptive aggregation network for efficient stereo matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1959–1968. IEEE (2020)

    Google Scholar 

  24. Kendall, A., et al.: End-to-end learning of geometry and context for deep stereo regression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 66–75. IEEE (2017)

    Google Scholar 

  25. Yu, L., Wang, Y., Wu, Y., Jia, Y.: Deep stereo matching with explicit cost aggregation sub-architecture. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  26. Song, X., Zhao, X., Hu, H., Fang, L.: EdgeStereo: a context integrated residual pyramid network for stereo matching. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11365, pp. 20–35. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20873-8_2

    Chapter  Google Scholar 

  27. Song, X., Zhao, X., Fang, L., Hu, H., Yu, Y.: EdgeStereo: an effective multi-task learning network for stereo matching and edge detection. Int. J. Comput. Vis. 128, 910–930 (2020). https://doi.org/10.1007/s11263-019-01287-w

    Article  Google Scholar 

  28. Zhang, F., Prisacariu, V., Yang, R., Torr, P.H.: GA-Net: guided aggregation net for end-to-end stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 185–194. IEEE (2019)

    Google Scholar 

  29. Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30, 328–341 (2008)

    Article  Google Scholar 

  30. Yang, G., Manela, J., Happold, M., Ramanan, D.: Hierarchical deep stereo matching on high-resolution images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5515–5524. IEEE (2019)

    Google Scholar 

  31. Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: Conference on Computer Vision and Pattern Recognition (2015)

    Google Scholar 

  32. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)

    Google Scholar 

  33. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (2012)

    Google Scholar 

  34. Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE (2016)

    Google Scholar 

  35. Guo, X., Yang, K., Yang, W., Wang, X., Li, H.: Group-wise correlation stereo network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3273–3282. IEEE (2019)

    Google Scholar 

Download references

Funding

The work is supported by NSFC-Zhejiang Joint Fund for the Integration of Industrialization and Informatization under grant No. U1709214.

Author information

Authors and Affiliations

  1. College of Information and Electronic Engineering, Zhejiang University, Hangzhou, China

    Shuya Chen, Chengyu Qiao, Yiman Chen & Tingming Bai

  2. Zhejiang Provincial Key Laboratory of Information Processing, Communication and Networking, Zhejiang University, Hangzhou, China

    Zhiyu Xiang

Authors
  1. Shuya Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhiyu Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengyu Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yiman Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tingming Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiyu Xiang .

Editor information

Editors and Affiliations

  1. Waseda University, Tokyo, Japan

    Hiroshi Ishikawa

  2. Institute of Automation of Chinese Academy of Sciences, Beijing, China

    Cheng-Lin Liu

  3. Czech Technical University in Prague, Prague, Czech Republic

    Tomas Pajdla

  4. University of Pennsylvania, Philadelphia, PA, USA

    Jianbo Shi

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, S., Xiang, Z., Qiao, C., Chen, Y., Bai, T. (2021). SGNet: Semantics Guided Deep Stereo Matching. In: Ishikawa, H., Liu, CL., Pajdla, T., Shi, J. (eds) Computer Vision – ACCV 2020. ACCV 2020. Lecture Notes in Computer Science(), vol 12622. Springer, Cham. https://doi.org/10.1007/978-3-030-69525-5_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-69525-5_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-69524-8

  • Online ISBN: 978-3-030-69525-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation