Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Multi-view 3D Objects Localization from Street-Level Scenes

  • Conference paper
  • First Online:
Image Analysis and Processing – ICIAP 2022 (ICIAP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13232))

Included in the following conference series:

Abstract

This paper presents a method to localize street-level objects in 3D from images of an urban area. Our method processes 3D sparse point clouds reconstructed from multi-view images and leverages 2D instance segmentation to find all objects within the scene and to generate for each object the corresponding cluster of 3D points and matched 2D detections. The proposed approach is robust to changes in image sizes, viewpoint changes, and changes in the object’s appearance across different views. We validate our approach on challenging street-level crowd-sourced images from the Mapillary platform, showing a significant improvement in the mean average precision of object localization for the available Mapillary annotations. These results showcase our method’s effectiveness in localizing objects in 3D, which could potentially be used in applications such as high-definition map generation of urban environments. The code is publicly available (https://github.com/IIT-PAVIS/Multi-view-3D-Objects-Localization-from-Street-level-Scenes).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Mapillary Python SDK, mapillary api v4. https://github.com/mapillary/mapillary-python-sdk. Accessed 15 Dec 2021

  2. Mapillary, the street-level imagery platform that scales and automates mapping. https://www.mapillary.com/. Accessed 15 Jul 2021

  3. OpenStreetMap. openstreetmap. Accessed 15 Jul 2021

  4. Ahmed, S.M., Chew, C.M.: Density-based clustering for 3d object detection in point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10608–10617 (2020)

    Google Scholar 

  5. Jocher, G., et al.: ultralytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations (April 2021). https://doi.org/10.5281/zenodo.4679653

  6. Blender Online Community: Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam (2016). http://www.blender.org

  7. Branson, S., Wegner, J.D., Hall, D., Lang, N., Schindler, K., Perona, P.: From Google Maps to a fine-grained catalog of street trees. ISPRS J. Photogramm. Remote. Sens. 135, 13–30 (2018). https://doi.org/10.1016/j.isprsjprs.2017.11.008

    Article  Google Scholar 

  8. Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for autonomous driving. In: Proceedings of the IEEE CVPR, pp. 1907–1915 (2017)

    Google Scholar 

  9. Crocco, M., Rubino, C., Del Bue, A.: Structure from motion with objects. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4141–4149 (2016)

    Google Scholar 

  10. Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, vol. 96, pp. 226–231 (1996)

    Google Scholar 

  11. Gay, P., Rubino, C., Bansal, V., Del Bue, A.: Probabilistic structure from motion with objects (PSfMO). In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3075–3084 (2017)

    Google Scholar 

  12. Hanif, M.S., Ahmad, S., Khurshid, K.: On the improvement of foreground-background model-based object tracker. IET Comput. Vis. 11(6), 488–496 (2017)

    Article  Google Scholar 

  13. Hebbalaguppe, R., Garg, G., Hassan, E., Ghosh, H., Verma, A.: Telecom inventory management via object recognition and localisation on Google Street View images. In: 2017 IEEE WACV, pp. 725–733. IEEE (2017)

    Google Scholar 

  14. Krylov, V.A., Kenny, E., Dahyot, R.: Automatic discovery and geotagging of objects from street view imagery. Remote Sens. 10(5), 661 (2018)

    Article  Google Scholar 

  15. Ku, J., Mozifian, M., Lee, J., Harakeh, A., Waslander, S.L.: Joint 3d proposal generation and object detection from view aggregation. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–8. IEEE (2018)

    Google Scholar 

  16. Liu, C.J., Ulicny, M., Manzke, M., Dahyot, R.: Context aware object geotagging. arXiv preprint arXiv:2108.06302 (2021)

  17. Liu, W.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  18. Nassar, A.S., D’Aronco, S., Lefèvre, S., Wegner, J.D.: GeoGraph: graph-based multi-view object detection with geometric cues end-to-end. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12352, pp. 488–504. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58571-6_29

    Chapter  Google Scholar 

  19. Nassar, A.S., Lefèvre, S., Wegner, J.D.: Simultaneous multi-view instance detection with learned geometric soft-constraints. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6559–6568 (2019)

    Google Scholar 

  20. Nicholson, L., Milford, M., Sünderhauf, N.: QuadricSLAM: dual quadrics from object detections as landmarks in object-oriented slam. IEEE Robot. Autom. Lett. 4(1), 1–8 (2018)

    Article  Google Scholar 

  21. Qi, C.R., Chen, X., Litany, O., Guibas, L.J.: ImVoteNet: boosting 3d object detection in point clouds with image votes. In: Proceedings of the IEEE/CVF CVPR, pp. 4404–4413 (2020)

    Google Scholar 

  22. Qi, C.R., Litany, O., He, K., Guibas, L.J.: Deep Hough voting for 3d object detection in point clouds. In: Proceedings of the IEEE/CVF ICCV, pp. 9277–9286 (2019)

    Google Scholar 

  23. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE CVPR, pp. 652–660 (2017)

    Google Scholar 

  24. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413 (2017)

  25. Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  26. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural. Inf. Process. Syst. 28, 91–99 (2015)

    Google Scholar 

  27. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104–4113 (2016)

    Google Scholar 

  28. Wang, Y., Chao, W.L., Garg, D., Hariharan, B., Campbell, M., Weinberger, K.Q.: Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In: Proceedings of the IEEE/CVF Conference on CVPR, pp. 8445–8453 (2019)

    Google Scholar 

  29. Wegner, J.D., Branson, S., Hall, D., Schindler, K., Perona, P.: Cataloging public objects using aerial and street-level images-urban trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6014–6023 (2016)

    Google Scholar 

  30. Yang, S., Scherer, S.: CubeSLAM: monocular 3d object detection and slam without prior models. arXiv preprint arXiv:1806.00557 (2018)

  31. Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3d object detection and tracking. In: Proceedings of the IEEE/CVF Conference on CVPR, pp. 11784–11793 (2021)

    Google Scholar 

  32. You, Y., et al.: Pseudo-LiDAR++: accurate depth for 3d object detection in autonomous driving. arXiv preprint arXiv:1906.06310 (2019)

  33. Zhang, C., Fan, H., Li, W.: Automated detecting and placing road objects from street-level images. Comput. Urban Sci. 1(1), 1–18 (2021). https://doi.org/10.1007/s43762-021-00019-6

    Article  Google Scholar 

  34. Zhang, W., Witharana, C., Li, W., Zhang, C., Li, X., Parent, J.: Using deep learning to identify utility poles with crossarms and estimate their locations from Google Street View images. Sensors 18(8), 2484 (2018)

    Article  Google Scholar 

  35. Zhang, Z., Sun, B., Yang, H., Huang, Q.: H3DNet: 3D object detection using hybrid geometric primitives. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 311–329. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_19

    Chapter  Google Scholar 

  36. Zhao, J., Zhang, X.N., Gao, H., Yin, J., Zhou, M., Tan, C.: Object detection based on hierarchical multi-view proposal network for autonomous driving. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE (2018)

    Google Scholar 

Download references

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 870743.

Author information

Authors and Affiliations

  1. Pattern Analysis and Computer Vision (PAVIS), Istituto Italiano di Tecnologia (IIT), Genova, Italy

    Javed Ahmad, Matteo Toso, Matteo Taiana, Stuart James & Alessio Del Bue

  2. Department of Electrical, Electronic and Telecommunications Engineering, and Naval Architecture (DITEN), Universita degli Studi di Genova, Genoa, Italy

    Javed Ahmad

Authors
  1. Javed Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matteo Toso
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matteo Taiana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stuart James
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alessio Del Bue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Javed Ahmad .

Editor information

Editors and Affiliations

  1. Boston University, Boston, MA, USA

    Stan Sclaroff

  2. National Research Council, Lecce, Italy

    Cosimo Distante

  3. National Research Council, Lecce, Italy

    Marco Leo

  4. University of Catania, Catania, Italy

    Giovanni M. Farinella

  5. Technische Universität München, Garching, Germany

    Federico Tombari

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 18804 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ahmad, J., Toso, M., Taiana, M., James, S., Del Bue, A. (2022). Multi-view 3D Objects Localization from Street-Level Scenes. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13232. Springer, Cham. https://doi.org/10.1007/978-3-031-06430-2_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06430-2_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06429-6

  • Online ISBN: 978-3-031-06430-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation