Skip to main content

Advertisement

SpringerLink
Log in

Joint bilateral propagation upsampling for unstructured multi-view stereo

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we explore a new way to accelerate and densify unstructured multi-view stereo (MVS). While many unstructured MVS algorithms have been proposed, we discover that the image-guided resizing can easily and significantly benefit their 3D reconstruction results in both efficiency and completeness. Therefore, we build our framework upon a novel selective joint bilateral upsampling and depth propagation strategy. First, we downsample the input unstructured images into lower resolution ones and perform the MVS calculation to efficiently obtain depth and normal maps from these resized pictures. Then, the proposed algorithm upsamples the normal maps with the guidance of input images, and jointly take them into consideration to recover the low-resolution depth maps into high resolution with geometry details simultaneously enriched. Finally by adaptively fusing the reconstructed depth and normal maps, we construct the final dense 3D scene. Quantitative results validate the efficiency and effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R.: Building rome in a day. Commun. ACM 54(10), 105–112 (2011)

    Article  Google Scholar 

  2. Agarwal, S., Snavely, N., Seitz, S.M., Szeliski, R.: Bundle adjustment in the large. In: ECCV, pp. 29–42 (2010)

  3. Barron, J.T., Poole, B.: The fast bilateral solver. In: ECCV, pp. 617–632 (2016)

  4. Chen, J., Adams, A., Wadhwa, N., Hasinoff, S.W.: Bilateral guided upsampling. ACM Trans. Graph. 35(6), 203 (2016)

    Article  Google Scholar 

  5. Fu, Y., Yan, Q., Yang, L., Liao, J., Xiao, C.: Texture mapping for 3d reconstruction with RGB-D sensor. In: CVPR, pp. 4645–4653 (2018)

  6. Fuhrmann, S., Langguth, F., Moehrle, N., Waechter, M., Goesele, M.: Mve: an image-based reconstruction environment. Comput. Graph. 53, 44–53 (2015)

    Article  Google Scholar 

  7. Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Towards internet-scale multi-view stereo. In: CVPR, pp. 1434–1441 (2010)

  8. Furukawa, Y., Hernández, C., et al.: Multi-view stereo: a tutorial. Found. Trends® Comput. Graph. Vis. 9(1–2), 1–148 (2015)

    Google Scholar 

  9. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2010)

    Article  Google Scholar 

  10. Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: ICCV, pp. 873–881 (2015)

  11. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the KITTI vision benchmark suite. In: CVPR, pp. 3354–3361 (2012)

  12. Goesele, M., Snavely, N., Curless, B., Hoppe, H., Seitz, S.M.: Multi-view stereo for community photo collections. In: ICCV, pp. 1–8 (2007)

  13. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  14. Heinly, J., Dunn, E., Frahm, J.M.: Correcting for Duplicate Scene Structure in Sparse 3D Reconstruction. In: ECCV, pp. 780–795 (2014)

  15. Heinly, J., Schonberger, J.L., Dunn, E., Frahm, J.M.: Reconstructing the world* in six days*(as captured by the yahoo 100 million image dataset). In: CVPR, pp. 3287–3295 (2015)

  16. Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. ACM Trans. Graph. 26(3), 96 (2007)

    Article  Google Scholar 

  17. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  18. Lu, X., Chen, W., Schaefer, S.: Robust mesh denoising via vertex pre-filtering and l1-median normal filtering. Comput. Aided Geom. Des. 54, 49–60 (2017)

    Article  MATH  Google Scholar 

  19. Moulon, P., Monasse, P., Perrot, R., Marlet, R.: Openmvg: Open multiple view geometry. In: International Workshop on Reproducible Research in Pattern Recognition, pp. 60–74 (2016)

  20. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR, pp. 4104–4113 (2016)

  21. Schonberger, J.L., Radenovic, F., Chum, O., Frahm, J.M.: From single image query to detailed 3D reconstruction. In: CVPR, pp. 5126–5134 (2015)

  22. Schönberger, J.L., Zheng, E., Frahm, J.M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: ECCV, pp. 501–518 (2016)

  23. Schöps, T., Schönberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: CVPR, vol. 3 (2017)

  24. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. CVPR 1, 519–528 (2006)

    Google Scholar 

  25. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. In: CVPR, pp. 1297–1304 (2011)

  26. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3d. ACM Trans. Graph. 25(3), 835–846 (2006)

    Article  Google Scholar 

  27. Strecha, C., Von Hansen, W., Van Gool, L., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: CVPR, pp. 1–8 (2008)

  28. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV, pp. 839–846 (1998)

  29. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustmenta modern synthesis. In: International Workshop on Vision Algorithms, pp. 298–372 (1999)

  30. Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Multicore bundle adjustment. In: CVPR, pp. 3057–3064 (2011)

  31. Xiao, C., Zheng, W., Miao, Y., Zhao, Y., Peng, Q.: A unified method for appearance and geometry completion of point set surfaces. Vis. Comput. 23(6), 433–443 (2007)

    Article  Google Scholar 

  32. Yagou, H., Ohtake, Y., Belyaev, A.: Mesh smoothing via mean and median filtering applied to face normals. In: GMP, p. 124 (2002)

  33. Yan, Q., Yang, L., Zhang, L., Xiao, C.: Distinguishing the indistinguishable: exploring structural ambiguities via geodesic context. In: CVPR (2017)

  34. Yang, L., Yan, Q., Xiao, C.: Shape-controllable geometry completion for point cloud models. Vis. Comput. 33(3), 385–398 (2016)

    Article  Google Scholar 

  35. Yao, Y., Luo, Z., Li, S., Fang, T., Quan, L.: Mvsnet: Depth inference for unstructured multi-view stereo. In: ECCV (2018)

  36. Yao, Y., Luo, Z., Li, S., Shen, T., Fang, T., Quan, L.: Recurrent mvsnet for high-resolution multi-view stereo depth inference. In: CVPR (2019)

  37. Zheng, E., Dunn, E., Jojic, V., Frahm, J.M.: Patchmatch based joint view selection and depthmap estimation. In: CVPR, pp. 1510–1517 (2014)

  38. Zheng, Y., Fu, H., Au, O.K.C., Tai, C.L.: Bilateral normal filtering for mesh denoising. IEEE Trans. Vis. Comput. Graph. 17(10), 1521–1530 (2011)

    Article  Google Scholar 

Download references

Acknowledgements

This work was partly supported by The National Key Research and Development Program of China (2017YF-B1002600), the NSFC (Nos. 61672390, 41201404), Wuhan Science and Technology Plan Project (No. 2017010201010109) and Key Technological Innovation Projects of Hubei Province (2018AAA062).

Author information

Authors and Affiliations

  1. School of Computer Science, Wuhan University, Wuhan, 430072, China

    Mengqiang Wei, Fei Luo, Chengfang Song & Chunxia Xiao

  2. JD.com American Technologies Corporation, Mountain View, CA, 94043, USA

    Qingan Yan

Authors
  1. Mengqiang Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingan Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fei Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chengfang Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chunxia Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunxia Xiao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 136 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, M., Yan, Q., Luo, F. et al. Joint bilateral propagation upsampling for unstructured multi-view stereo. Vis Comput 35, 797–809 (2019). https://doi.org/10.1007/s00371-019-01688-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01688-5

Keywords

Search

Navigation