Skip to main content

Advertisement

Springer Nature Link
Log in

EC-FBNet: embeddable converged front- and back-end network for 3D reconstruction in low-light-level environment

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The implementation of 3D reconstruction for targets in the low-light-level (LLL) environment is an immediate requirement in military, aerospace and other fields related to this environment. However, in such a photon-deficient environment, the amount of available information is extremely limited; thus, leading to the 3D reconstruction task in this environment is challenging. To address this issue, an embeddable converged front- and back-end network (EC-FBNet) is proposed in this paper, it can extract sparse information from the LLL environment by aggregating multi-layer semantic, then according to the similarity of features among object parts, to calculate the global topology structure of the 3D model. For the training approach, the EC-FBNet performs the two-stage integrated training modality. We additionally construct an embedded global inferential attention module (GIAM), to distribute the association weights among the points in the model, and thus reason out the global topology structure of the 3D model. In order to acquire realistic images in the LLL environment, this study leverages the multi-pixel photon counter (MPPC) detector to capture stable photon counting images in this environment, then packages into a dataset for training by the network. In experiment, the proposed approach not only achieves results superior to the state-of-the-art approaches, but also competitive in the quality of the reconstructed model. We believe that this approach can be a useful tool for 3D reconstruction field in the LLL environment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Liu, X., Wang, M., Wang, A., Liu, S., Pi, X.: Light field reconstruction via attention maps of hybrid networks. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02644-6

    Article  Google Scholar 

  2. Chen, L., Wan, L.: Ctunet: automatic pancreas segmentation using a channel-wise transformer and 3d u-net. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02656-2

    Article  Google Scholar 

  3. Fan, M., Liu, L., Deng, N., Xin, B., Wang, Y., He, Y.: Digital 3d system for classifying fabric pilling based on improved active contours and neural network. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02647-3

    Article  Google Scholar 

  4. Tang, K., Chen, Y., Peng, W., Zhang, Y., Fang, M., Wang, Z., Song, P.: Reppvconv: attentively fusing reparameterized voxel features for efficient 3d point cloud perception. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02682-0

  5. Xu, H., Zhou, Z., Wang, Y., Kang, W., Sun, B., Li, H., Qiao, Y.: Digging into uncertainty in self-supervised multi-view stereo. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 6078–6087 (2021)

  6. Pontes, J.K., Kong, C., Sridharan, S., Lucey, S., Eriksson, A., Fookes, C.: Image2mesh: a learning framework for single image 3d reconstruction. In: Asian conference on computer vision, pp. 365–381 (2018). Springer

  7. Fan, H., Yu, X., Yang, Y., Kankanhalli, M.: Deep hierarchical representation of point cloud videos via spatio-temporal decomposition. IEEE Trans. Patt. Anal. Mach. Intell. 44(12), 9918–9930 (2021)

    Article  Google Scholar 

  8. Sun, B., Jiang, P., Kong, D., Shen, T.: Iv-net: single-view 3d volume reconstruction by fusing features of image and recovered volume. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02725-6

    Article  Google Scholar 

  9. Furuya, T., Liu, W., Ohbuchi, R., Kuang, Z.: Hyperplane patch mixing-and-folding decoder and weighted chamfer distance loss for 3d point set reconstruction. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02652-6

    Article  Google Scholar 

  10. Liu, X., Huang, H., Wang, W., Zhou, J.: Multi-view 3d shape style transformation. Vis. Comput. (2022). https://doi.org/10.1007/s00371-020-02042-w

    Article  Google Scholar 

  11. Peng, H., Xian, C., Zhang, Y.: 3d hand mesh reconstruction from a monocular RGB image. Vis. Comput. 36(10–12), 2227–2239 (2020)

    Article  Google Scholar 

  12. Cui, H., Shen, S., Gao, W., Wang, Z.: Progressive large-scale structure-from-motion with orthogonal msts. In: 2018 International conference on 3D vision (3D). IEEE, pp. 79–88 (2018)

  13. Schonberger, J.L., Frahm, J.-M.: Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104–4113 (2016)

  14. Li, G., Wu, Z., Liu, Y., Zhang, H., Nie, Y., Mao, A.: 3d hand reconstruction from a single image based on biomechanical constraints. Vis. Comput. 37, 2699–2711 (2021)

    Article  Google Scholar 

  15. Li, Z., Zuo, W., Wang, Z., Zhang, L.: Robust 3d reconstruction from uncalibrated small motion clips. Vis. Comput. (2022). https://doi.org/10.1007/s00371-021-02090-w

    Article  Google Scholar 

  16. Loh, Y.P., Chan, C.S.: Getting to know low-light images with the exclusively dark dataset. Comput. Vis. Image Underst. 178, 30–42 (2019). https://doi.org/10.1016/j.cviu.2018.10.010

    Article  Google Scholar 

  17. Anaya, J., Barbu, A.: Renoir - a dataset for real low-light noise image reduction. J. Vis. Commun. Image Represent. 51, 144–154 (2018)

    Article  Google Scholar 

  18. Yin, L.-j., Chen, Q., Gu, G.-h., Gong, S.-x.: Monte carlo simulation and implementation of photon counting image based on apd. J. Nanjing Univer. Sci. Technol. (Nat. Sci.) 34(5), 649–652 (2010)

  19. Wang, X., Yin, L., Gao, M., Wang, Z., Shen, J., Zou, G.: Denoising method for passive photon counting images based on block-matching 3d filter and non-subsampled contourlet transform. Sensors 19(11), 2462 (2019)

    Article  Google Scholar 

  20. Li, Y., Yin, L., Wang, Z., Pan, J., Gao, M., Zou, G., Liu, J., Wang, L.: Bayesian regularization restoration algorithm for photon counting images. Appl. Intell. 51(8), 5898–5911 (2021)

    Article  Google Scholar 

  21. Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: A papier-mâché approach to learning 3d surface generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 216–224 (2018)

  22. Saito, S., Simon, T., Saragih, J., Joo, H.: Pifuhd: multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 84–93 (2020)

  23. Wu, J., Zhang, C., Zhang, X., Zhang, Z., Freeman, W.T., Tenenbaum, J.B.: Learning shape priors for single-view 3d completion and reconstruction. In: Proceedings of the European conference on computer vision (ECCV), pp. 646–662 (2018)

  24. Han, X., Li, Z., Huang, H., Kalogerakis, E., Yu, Y.: High-resolution shape completion using deep neural networks for global structure and local geometry inference. In: Proceedings of the IEEE international conference on computer vision, pp. 85–93 (2017)

  25. Zhang, X., Feng, Y., Li, S., Zou, C., Wan, H., Zhao, X., Guo, Y., Gao, Y.: View-guided point cloud completion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15890–15899 (2021)

  26. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460–4470 (2019)

  27. Nguyen, A.-D., Choi, S., Kim, W., Lee, S.: Graphx-convolution for point cloud deformation in 2d-to-3d conversion. In: Proceedings of the IEEE/CVF International conference on computer vision, pp. 8628–8637 (2019)

  28. Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605–613 (2017)

  29. Mandikal, P., Navaneet, K., Agarwal, M., Babu, R.V.: 3d-lmnet: latent embedding matching for accurate and diverse 3d point cloud reconstruction from a single image. arXiv preprint arXiv:1807.07796 (2018)

  30. Shin, D., Fowlkes, C.C., Hoiem, D.: Pixels, voxels, and views: a study of shape representations for single view 3d object shape prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3061–3069 (2018)

  31. Smith, E.J., Meger, D.: Improved adversarial systems for 3d object generation and reconstruction. In: Conference on robot learning. PMLR, pp. 87–96 (2017)

  32. Doersch, C.: Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 (2016)

  33. Sun, X., Wu, J., Zhang, X., Zhang, Z., Zhang, C., Xue, T., Tenenbaum, J.B., Freeman, W.T.: Pix3d: dataset and methods for single-image 3d shape modeling. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2974–2983 (2018)

  34. Qiu, S., Anwar, S., Barnes, N.: Geometric back-projection network for point cloud classification. IEEE Trans. Multim. 24, 1943–1955 (2021)

    Article  Google Scholar 

  35. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 7794–7803 (2018)

  36. Tatarchenko, M., Dosovitskiy, A., Brox, T.: Octree generating networks: efficient convolutional architectures for high-resolution 3d outputs. In: Proceedings of the IEEE International conference on computer vision, pp. 2088–2096 (2017)

  37. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: a deep representation for volumetric shapes. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 1912–1920 (2015)

  38. Lai, K., Bo, L., Fox, D.: Unsupervised feature learning for 3d scene labeling. In: 2014 IEEE International conference on robotics and automation (ICRA). IEEE, pp. 3050–3057 (2014)

  39. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  40. Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., et al.: Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018)

  41. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Proceedings of the european conference on computer vision (ECCV), pp. 3–19 (2018)

  42. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) (2018)

  43. Guo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R.R., Hu, S.-M.: Pct: point cloud transformer. Comput. Vis. Media 7(2), 187–199 (2021)

    Article  Google Scholar 

  44. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3146–3154 (2019)

  45. Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W.: Ccnet: criss-cross attention for semantic segmentation. In: Proceedings of the IEEE/CVF International conference on computer vision, pp. 603–612 (2019)

  46. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30 (2017)

  47. Gomez-Donoso, F., Garcia-Garcia, A., Garcia-Rodriguez, J., Orts-Escolano, S., Cazorla, M.: Lonchanet: a sliced-based cnn architecture for real-time 3d object recognition. In: 2017 International joint conference on neural networks (IJCNN). IEEE, pp. 412–418 (2017)

  48. Wang, C., Pelillo, M., Siddiqi, K.: Dominant set clustering and pooling for multi-view 3d object recognition. arXiv preprint arXiv:1906.01592 (2019)

  49. Brock, A., Lim, T., Ritchie, J.M., Weston, N.: Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236 (2016)

  50. Arsalan Soltani, A., Huang, H., Wu, J., Kulkarni, T.D., Tenenbaum, J.B.: Synthesizing 3d shapes via modeling multi-view depth maps and silhouettes with deep generative networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1511–1519 (2017)

  51. Arvind, V., Costa, A., Badgeley, M., Cho, S., Oermann, E.: Wide and deep volumetric residual networks for volumetric image classification. arXiv preprint arXiv:1710.01217 (2017)

  52. Xu, M., Zhang, J., Zhou, Z., Xu, M., Qi, X., Qiao, Y.: Learning geometry-disentangled representation for complementary understanding of 3d object point cloud. In: Proceedings of the AAAI conference on artificial intelligence, vol. 35, pp. 3056–3064 (2021)

  53. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. (tog) 38(5), 1–12 (2019)

  54. Xiang, T., Zhang, C., Song, Y., Yu, J., Cai, W.: Walk in the cloud: learning curves for point clouds shape analysis. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 915–924 (2021)

  55. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708 (2017)

  56. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (2015)

  57. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \(<\) 0.5 mb model size. arXiv preprint arXiv:1602.07360 (2016)

  58. Xie, S., Liu, S., Chen, Z., Tu, Z.: Attentional shapecontextnet for point cloud recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4606–4615 (2018)

  59. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141 (2018)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) (62101310) and Natural Science Foundation of Shandong Province, China (ZR2020MF127).

Author information

Authors and Affiliations

  1. Shandong University of Technology, School of Electrical and Electronic Engineering, Zibo, 255000, Shandong, China

    Yulin Deng, Liju Yin, Xiaoning Gao, Hui Zhou, Zhenzhou Wang & Guofeng Zou

Authors
  1. Yulin Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liju Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoning Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hui Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhenzhou Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guofeng Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YD did conceptualization, methodology, software, writing reviewing and editing. LY done visualization, investigation, supervision. XG performed data curation, software, and validation. HZ contributed to writing—original draft preparation. ZW and GZ gave software.

Corresponding author

Correspondence to Liju Yin.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Consent for publication

We undersigned declare that this manuscript is original, has not been published before and is not currently being considered for publication elsewhere. We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us. We understand that the corresponding author is the sole contact for the editorial process. She is responsible for communicating with the other authors about progress, submissions of revisions and final approval of proofs. We guarantee the availability of data and materials. Thank you for your consideration. I am looking forward to your assessment of our manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deng, Y., Yin, L., Gao, X. et al. EC-FBNet: embeddable converged front- and back-end network for 3D reconstruction in low-light-level environment. Vis Comput 40, 4441–4456 (2024). https://doi.org/10.1007/s00371-023-03091-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03091-7

Keywords