Skip to main content
SpringerLink
Log in

Towards Reconstruction of 3D Shapes in a Realistic Environment

  • Conference paper
  • First Online:
Book cover Image Analysis and Processing – ICIAP 2022 (ICIAP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13232))

Included in the following conference series:

Abstract

This paper presents an end-to-end approach for single-view 3D object reconstruction in a realistic environment. Most of the existing reconstruction approaches are trained on synthetic data and they fail when evaluated on real images. On the other hand, some of the methods require pre-processing in order to separate an object from the background. In contrast, the proposed approach learns to compute stable features for an object by reducing the influence of image background. This is achieved by feeding two images simultaneously to the model; synthetic with white background and its realistic variant with a natural background. The encoder extracts the common features from both images and hence separates features of the object from features of the background. The extracted features allow the model to predict an accurate 3D object surface from a real image. The approach is evaluated for both real images of the Pix3D dataset and realistic images rendered from the ShapeNet dataset. The results are compared with state-of-the-art approaches in order to highlight the significance of the proposed approach. Our approach achieves an increase in reconstruction accuracy of approximately 6.1% points in F\(_1\) score with respect to Mesh R-CNN on the Pix3D dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Wang, Z., Isler, V., Lee, D.D.: Surface HOF: surface reconstruction from a single image using higher order function networks. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 2666–2670 (2020)

    Google Scholar 

  2. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3D reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460–4470 (2019)

    Google Scholar 

  3. Deng, B., Genova, K., Yazdani, S., Bouaziz, S., Hinton, G., Tagliasacchi, A.: CVXNET: learnable convex decomposition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 31–44 (2020)

    Google Scholar 

  4. Gkioxari, G., Malik, J., Johnson, J.: Mesh R-CNN. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9785–9795 (2019)

    Google Scholar 

  5. Spezialetti, R., Tan, D.J., Tonioni, A., Tateno, K., Tombari, F.: A divide et Impera approach for 3D shape reconstruction from multiple views. In: 2020 International Conference on 3D Vision (3DV), pp. 160–170 (2020)

    Google Scholar 

  6. Yang, B., Rosa, S., Markham, A., Trigoni, N., Wen, H.: Dense 3D object reconstruction from a single depth view. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 2820–2834 (2018)

    Article  Google Scholar 

  7. Gupta, K., Jabbireddy, S., Shah, K., Shrivastava, A., Zwicker, M.: Improved modeling of 3D shapes with multi-view depth maps. In: International Conference on 3D Vision (3DV), pp. 71–80 (2020)

    Google Scholar 

  8. Gu, J., et al.: Weakly-supervised 3D shape completion in the wild. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12350, pp. 283–299. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58558-7_17

    Chapter  Google Scholar 

  9. Yin, K., Chen, Z., Chaudhuri, S., Fisher, M., Kim, V.G., Zhang, H.: COALESCE: component assembly by learning to synthesize connections. In: 2020 International Conference on 3D Vision (3DV), pp. 61–70 (2020)

    Google Scholar 

  10. Lombardi, S., Oswald, M.R., Pollefeys, M.: Scalable point cloud-based reconstruction with local implicit functions. In: 2020 International Conference on 3D Vision (3DV), pp. 997–1007 (2020)

    Google Scholar 

  11. Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. arXiv:1512.03012 (2015)

  12. Chen, Z., Tagliasacchi, A., Zhang, H.: BSP-NET: generating compact meshes via binary space partitioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 45–54 (2020)

    Google Scholar 

  13. Wu, J., Wang, Y., Xue, T., Sun, X., Freeman, W.T., Tenenbaum, J.B.: MarrNet: 3D shape reconstruction via 2.5 D sketches. In: 31st Conference on Neural Information Processing Systems (2017)

    Google Scholar 

  14. Sun, X., Wu, J., Zhang, X., Zhang, Z., Zhang, C., Xue, T., Tenenbaum, J.B., Freeman, W.T.: Pix3D: dataset and methods for single-image 3D shape modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2974–2983 (2018)

    Google Scholar 

  15. Xie, H., Yao, H., Zhang, S., Zhou, S., Sun, W.: Pix2Vox++: multi-scale context-aware 3D object reconstruction from single and multiple images. Int. J. Comput. Vision 128(12), 2919–2935 (2020)

    Article  Google Scholar 

  16. Girdhar, R., Fouhey, D.F., Rodriguez, M., Gupta, A.: Learning a predictable and generative vector representation for objects. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 484–499. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_29

    Chapter  Google Scholar 

  17. Feng, Q., Luo, Y., Luo, K., Yang, Y.: Look, cast and mold: learning 3D shape manifold from single-view synthetic data. arXiv preprint arXiv:2103.04789 (2021)

  18. Pinheiro, P.O., Rostamzadeh, N., Ahn, S.: Domain-adaptive single-view 3D reconstruction. In: IEEE International Conference on Computer Vision, pp. 7638–7647 (2019)

    Google Scholar 

  19. Yang, S., Xu, M., Xie, H., Perry, S., Xia, J.: Single-view 3D object reconstruction from shape priors in memory. arXiv preprint arXiv:2003.03711, pp. 3152–3161 (2020)

  20. Kaya, B., Timofte, R.: Self-supervised 2D image to 3D shape translation with disentangled representations. In: 2020 International Conference on 3D Vision (3DV), pp. 1039–1048 (2020)

    Google Scholar 

  21. Salvi, A., Gavenski, N., Pooch, E., Tasoniero, F., Barros, R.: Attention-based 3D object reconstruction from a single image. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020)

    Google Scholar 

  22. Oh Song, H., Xiang, Y., Jegelka, S., Savarese, S.: Deep metric learning via lifted structured feature embedding. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4004–4012 (2016)

    Google Scholar 

  23. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Comput. Graph. 21(4), 163–169 (1987)

    Article  Google Scholar 

  24. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  25. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

    Google Scholar 

  26. Choy, C.B., Xu, D., Gwak, J.Y., Chen, K., Savarese, S.: 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 628–644. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_38

    Chapter  Google Scholar 

  27. Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485–3492 (2010)

    Google Scholar 

  28. Bolya, D., Zhou, C., Xiao, F., Lee, Y.J.: YOLACT: real-time instance segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9157–9166 (2019)

    Google Scholar 

  29. Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2015)

    Article  Google Scholar 

  30. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  31. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Pattern Analysis and Computer Vision (PAVIS), Italian Institute of Technology, Genoa, Italy

    Mohammad Zohaib, Matteo Taiana & Alessio Del Bue

  2. Department of Marine, Electrical, Electronic and Telecommunications Engineering, University of Genoa, Genoa, Italy

    Mohammad Zohaib

Authors
  1. Mohammad Zohaib
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matteo Taiana
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alessio Del Bue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Zohaib .

Editor information

Editors and Affiliations

  1. Boston University, Boston, MA, USA

    Stan Sclaroff

  2. National Research Council, Lecce, Italy

    Cosimo Distante

  3. National Research Council, Lecce, Italy

    Marco Leo

  4. University of Catania, Catania, Italy

    Giovanni M. Farinella

  5. Technische Universität München, Garching, Germany

    Federico Tombari

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1127 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zohaib, M., Taiana, M., Bue, A.D. (2022). Towards Reconstruction of 3D Shapes in a Realistic Environment. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13232. Springer, Cham. https://doi.org/10.1007/978-3-031-06430-2_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06430-2_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06429-6

  • Online ISBN: 978-3-031-06430-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation