Skip to main content
SpringerLink
Log in

Self-supervised Indoor 360-Degree Depth Estimation via Structural Regularization

  • Conference paper
  • First Online:
PRICAI 2022: Trends in Artificial Intelligence (PRICAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13631))

Included in the following conference series:

Abstract

Estimating \(360^{\circ }\) depth information has attracted considerable attention due to the fast development of emerging \(360^{\circ }\) cameras. However, most researches only focus on dealing with the distortion of \(360^{\circ }\) images without considering the geometric information of \(360^{\circ }\) images, leading to poor performance. In this paper, we conduct to apply indoor structure regularities for self-supervised \(360^{\circ }\) image depth estimation. Specifically, we carefully design two geometric constraints for efficient model optimization including dominant direction normal constraint and planar consistency depth constraint. The dominant direction normal constraint enables to align the normal of indoor \(360^{\circ }\) images with the direction of vanishing points. The planar consistency depth constraint is utilized to fit the estimated depth of each pixel by its 3D plane. Hence, incorporating these two geometric constraints can further facilitate the generation of accurate depth results for \(360^{\circ }\) images. Extensive experiments illustrate that our designed method improves \(\delta _1\) by an average of 4.82% compared to state-of-the-art methods on Matterport3D and Stanford2D3D datasets within 3D60.

This work was supported in part by the National Natural Science Foundation of China (Grant 61871270, 62171134), in part by the Shenzhen Natural Science Foundation under Grants JCYJ20200109110410133 and 20200812110350001.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Zioulis, N., Karakottas, A., Zarpalas, D., Daras, P.: OmniDepth: dense depth estimation for indoors spherical panoramas. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11210, pp. 453–471. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_28

    Chapter  Google Scholar 

  2. Wang, F.-E., Yeh, Y.-H., Sun, M., Chiu, W.-C., Tsai, Y.-H.: Bifuse: monocular 360 depth estimation via bi-projection fusion. In: Proceedings CVPR, pp. 462–471 (2020)

    Google Scholar 

  3. Wang, F.-E., et al.: Self-supervised learning of depth and camera motion from 360\(^\circ \) videos. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11365, pp. 53–68. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20873-8_4

    Chapter  Google Scholar 

  4. Zioulis, N., Karakottas, A., Zarpalas, D., Alvarez, F., Daras, P.: Spherical view synthesis for self-supervised 360 depth estimation. In: Proceedings 3DV, pp. 690–699. IEEE (2019)

    Google Scholar 

  5. Lai, Z., Chen, D., Su, K.: Olanet: self-supervised \(360^{\circ }\) depth estimation with effective distortion-aware view synthesis and l1 smooth regularization. In: Proceedings ICME, pp. 1–6. IEEE (2021)

    Google Scholar 

  6. Zhang, Y., Song, S., Tan, P., Xiao, J.: PanoContext: a whole-room 3D context model for panoramic scene understanding. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 668–686. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_43

    Chapter  Google Scholar 

  7. Fernandez-Labrador, C., Perez-Yus, A., Lopez-Nicolas, G., Guerrero, J.J.: Layouts from panoramic images with geometry and deep learning. IEEE Robot. Autom. Lett. 3(4), 3153–3160 (2018)

    Article  Google Scholar 

  8. Yang, Z., Wang, P., Wang, Y., Xu, W., Nevatia, R.: Lego: learning edge with geometry all at once by watching videos. In: Proceedings CVPR, pp. 225–234 (2018)

    Google Scholar 

  9. Li, B., Huang, Y., Liu, Z., Zou, D., Yu, W.: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation. In: Proceedings CVPR, pp. 663–673 (2021)

    Google Scholar 

  10. Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004)

    Article  MATH  Google Scholar 

  11. Yu, Z., Jin, L., Gao, S.: P\(^{2}\)Net: patch-match and plane-regularization for unsupervised indoor depth estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12369, pp. 206–222. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58586-0_13

    Chapter  Google Scholar 

  12. Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency, In: Proceedings CVPR, pp. 270–279 (2017)

    Google Scholar 

  13. Paszke, A., et al.: Automatic differentiation in pytorch (2017)

    Google Scholar 

  14. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings AISTATS, pp. 249–256. JMLR (2010)

    Google Scholar 

  15. Zhang, Y., et al.: ActiveStereoNet: end-to-end self-supervised learning for active stereo systems. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 802–819. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_48

    Chapter  Google Scholar 

  16. Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. In: Proceedings CVPR, pp. 1746–1754 (2017)

    Google Scholar 

  17. Chang, A., et al.: Matterport3D: Learning from RGB-D data in indoor environments. In: Proceedings 3DV (2017)

    Google Scholar 

  18. Armeni, I., Sax, S., Zamir, A.R., Savarese, S.: Joint 2D–3D-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105 (2017)

  19. Jiang, H., Sheng, Z., Zhu, S., Dong, Z., Huang, R.: Unifuse: Unidirectional fusion for 360 panorama depth estimation. IEEE Robot. Autom. Lett. 6(2), 1519–1526 (2021)

    Article  Google Scholar 

  20. Cheng, H.-T., Chao, C.-H., Dong, J.-D., Wen, H.-K., Liu, T.-L., Sun, M.: Cube padding for weakly-supervised saliency prediction in 360 videos. In: Proceedings CVPR, pp. 1420–1429 (2018)

    Google Scholar 

  21. Su, Y.-C., Grauman, K.: Learning spherical convolution for fast features from 360 imagery. Adv. NIPS 30, 529–539 (2017)

    Google Scholar 

  22. Tateno, K., Navab, N., Tombari, F.: Distortion-aware convolutional filters for dense prediction in panoramic images. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 732–750. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_43

    Chapter  Google Scholar 

  23. Yang, S., Song, Y., Kaess, M., Scherer, S.: Pop-up slam: Semantic monocular plane slam for low-texture environments. In: Proceedings IROS, pp. 1222–1229. IEEE (2016)

    Google Scholar 

  24. Wang, R., Geraghty, D., Matzen, K., Szeliski, R., Frahm, J.-M.: VPLNet: deep single view normal estimation with vanishing points and lines. In: Proceedings CVPR, pp. 689–698 (2020)

    Google Scholar 

  25. Lu, X., Yaoy, J., Li, H., Liu, Y., Zhang, X.: 2-line exhaustive searching for real-time vanishing point estimation in manhattan world. In: Proceedings WACV, pp. 345–353. IEEE (2017)

    Google Scholar 

  26. Yu, Z., Zheng, J., Lian, D., Zhou, Z., Gao, S.: Single-image piece-wise planar 3D reconstruction via associative embedding. In: Proceedings CVPR, pp. 1029–1037 (2019)

    Google Scholar 

  27. Shah, A., Kadam, E., Shah, H., Shinde, S., Shingade, S.: Deep residual networks with exponential linear unit. In: Proceedings of the Third International Symposium on Computer Vision and the Internet, pp. 59–65 (2016)

    Google Scholar 

  28. Liu, R., et al.: An intriguing failing of convolutional neural networks and the coordconv solution. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  29. Monroy, R., Lutz, S., Chalasani, T., Smolic, A.: Salnet360: saliency maps for omni-directional images with CNN. Sig. Process. Image Commun. 69, 26–34 (2018)

    Article  Google Scholar 

  30. Khasanova, R., Frossard, P.: Graph-based classification of omnidirectional images. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 869–878 (2017)

    Google Scholar 

  31. Sun, C., Sun, M., Chen, H.-T.: HoHoNet: 360 indoor holistic understanding with latent horizontal features. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2573–2582 (2021)

    Google Scholar 

  32. Coughlan, J.M., Yuille, A.L.: Manhattan world: compass direction from a single image by bayesian inference. In: Proceedings of the seventh IEEE international conference on computer vision, vol. 2, pp. 941–947. IEEE (1999)

    Google Scholar 

  33. Zou, D., Wu, Y., Pei, L., Ling, H., Yu, W.: Structvio: visual-inertial odometry with structural regularity of man-made environments. IEEE Trans. Rob. 35(4), 999–1013 (2019)

    Article  Google Scholar 

  34. Tulsiani, S., Tucker, R., Snavely, N.: Layer-structured 3D scene inference via view synthesis. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 311–327. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_19

    Chapter  Google Scholar 

  35. Li, Y., Guo, Y., Yan, Z., Huang, X., Duan, Y., Ren, L.: Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2801–2810 (2022)

    Google Scholar 

  36. Bai, J., Lai, S., Qin, H., Guo, J., Guo, Y.: GlpanoDepth: global-to-local panoramic depth estimation. arXiv preprint arXiv:2202.02796 (2022)

  37. Area, M.R., Yuan, M., Richardt, C.: 360monodepth: high-resolution \(360^{\circ }\) monocular depth estimation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China

    Weifeng Kong, Qiudan Zhang & Xu Wang

  2. College of Electronics and Information Engineering, Shenzhen University, Shenzhen, 518060, China

    Wenhui Wu

  3. School of Electronic Information and Communications, Huangzhong University of Science and Technology, Wuhan, 430074, China

    You Yang

  4. College of Physics and Information Engineering, Fuzhou University, Fuzhou, 350108, China

    Tiesong Zhao

Authors
  1. Weifeng Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiudan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. You Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tiesong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenhui Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xu Wang .

Editor information

Editors and Affiliations

  1. CSIRO Australian e-Health Research Centre, Brisbane, QLD, Australia

    Sankalp Khanna

  2. Shanghai Jiao Tong University, Shanghai, China

    Jian Cao

  3. University of Tasmania, Hobart, TAS, Australia

    Quan Bai

  4. University of Technology Sydney, Sydney, NSW, Australia

    Guandong Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kong, W., Zhang, Q., Yang, Y., Zhao, T., Wu, W., Wang, X. (2022). Self-supervised Indoor 360-Degree Depth Estimation via Structural Regularization. In: Khanna, S., Cao, J., Bai, Q., Xu, G. (eds) PRICAI 2022: Trends in Artificial Intelligence. PRICAI 2022. Lecture Notes in Computer Science, vol 13631. Springer, Cham. https://doi.org/10.1007/978-3-031-20868-3_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20868-3_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20867-6

  • Online ISBN: 978-3-031-20868-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation