Skip to main content
SpringerLink
Log in

Spatio-temporal Data Association for Object-augmented Mapping

  • Regular Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Traditionally, visual SLAM methods make use of visual features for mapping and localization. However, the resulting map may lack important semantic information, such as the objects (and their locations) present in the location. Since the same objects may be detected several times during the mapping phase, data association becomes a critical issue: objects viewed from different angles and in different time instants must be fused together into a single instance on the map. In this paper, we propose Spatio-temporal Data Association (STDA) for object-augmented mapping. It is based on expected similarities between consecutive frames (temporal association) and similar non-consecutive frames (spatial association). The experiments suggest that our system is capable of correctly fusing together multiple views of several objects, resulting in only one false positive association in more than 130 detected objects across several datasets. The results are competitive with the state-of-the-art. We also generated object location ground truth annotations for 3 simulated environments to foster further comparison. Finally, the annotated map was used for an object fetching task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Availability of data and material

Our code will be made public on a GitHub repository when the paper is published.

References

  1. Salim, A., Eray, E.: Performance evaluation of the grid-based FastSLAM in V-REP using MATLAB. In: 14th International Conference on Advanced Trends in Radioelecrtronics, Telecommunications and Computer Engineering (TCSET), pp 276–281. IEEE (2018)

  2. Castle, R.O., Murray, D.W.: Keyframe-based recognition and localization during video-rate parallel tracking and mapping. Image Vis. Comput. 29(8), 524–532 (2011)

    Article  Google Scholar 

  3. Civera, J., Gálvez-López, D., Riazuelo, L., Tardós, J.D., Montiel, J.M.M.: Towards semantic SLAM using a monocular camera. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 1277–1284. IEEE (2011)

  4. De Gregorio, D., Cavallari, T., Di Stefano, L.: Skimap++: Real-time mapping and object recognition for robotics. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp 660–668 (2017)

  5. Gálvez-López, D., Salas, M., Tardós, J.D., Montiel, J.M.M.: Real-time monocular object SLAM. Robot. Auton. Syst. 75, 435–449 (2016)

    Article  Google Scholar 

  6. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 3354–3361. IEEE (2012)

  7. Helton, J.C., Davis, F.J., Johnson, J.D.: A comparison of uncertainty and sensitivity analysis results obtained with random and Latin hypercube sampling. Reliab. Eng. Syst. Saf. 89(3), 305–330 (2005)

    Article  Google Scholar 

  8. Hosseinzade, M., Li, K., Latif, Y., Reid, I.: Real-time monocular object-model aware sparse SLAM. In: International Conference on Robotics and Automation, pp 7123–7129. IEEE (2019)

  9. Iqbal, A., Gans, N.R.: Data association and localization of classified objects in visual SLAM. In: Journal of Intelligent & Robotic Systems (2020)

  10. Koenig, N., Howard, A.: Design and use paradigms for gazebo, an open-source multi-robot simulator. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 3, pp 2149–2154. IEEE (2004)

  11. Labbé, M., Michaud, F.: RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. J. Field Robot. 36(2), 416–446 (2019)

    Article  Google Scholar 

  12. Mao, M., Zhang, H., Li, S., Zhang, B.: Semantic-RTAB-map (SRM): A semantic SLAM system with CNNs on depth images. Math. Found. Comput. 2(1), 29 (2019)

    Article  Google Scholar 

  13. Mu, B., Liu, S.-Y., Paull, L., Leonard, J., How, J.P.: Slam with objects using a nonparametric pose graph. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 4602–4609. IEEE (2016)

  14. Mur-Artal, R., Tardós, J.D.: ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGBD cameras. IEEE Trans. Robot. 33(5), 1255–1262 (2017)

    Article  Google Scholar 

  15. Nistér, D., Naroditsky, O., Bergen, J.: Visual odometry. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition., vol. 1, pp I–I. IEEE (2004)

  16. Pire, T., Corti, J., Grinblat, G.: Online object detection and localization on stereo visual SLAM system. In: Journal of Intelligent & Robotic Systems, pp 1–10 (2019)

  17. Pire, T., Fischer, T., Castro, G., De Cristóforis, P., Civera, J., Berlles, J.J.: S-PTAM: Stereo parallel tracking and mapping. Robot. Auton. Syst. 93, 27–42 (2017)

    Article  Google Scholar 

  18. Rasouli, A., Tsotsos, J.K.: The effect of color space selection on detectability and discriminability of colored objects. arXiv:1702.05421 (2017)

  19. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv:1804.02767(2018)

  20. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99 (2015)

  21. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: An efficient alternative to SIFT or SURF. In: International conference on computer vision, pp 2564–2571. IEEE (2011)

  22. Salas-Moreno, R.F., Newcombe, R., Strasdat, H., Kelly, P.H.J., Davison, A.J.: SLAM++: Simultaneous localisation and mapping at the level of objects. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1352–1359 (2013)

  23. Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D SLAM systems. In: Proc. of the International Conference on Intelligent Robot Systems (IROS) (2012)

  24. Vasudevan, S., Gächter, S., Nguyen, V., Siegwart, R.: Cognitive maps for mobile robots—an object based approach. Robot. Auton. Syst. 55(5), 359–371 (2007)

    Article  Google Scholar 

  25. Vincent, J., Labbé, M, Lauzon, J.-S., Grondin, F., Comtois-Rivet, P.-M., Michaud, F.: Dynamic object tracking and masking for visual SLAM. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 4974–4979 (2020)

  26. Yang, S., Scherer, S.: CubeSLAM: Monocular 3-D object SLAM. IEEE Trans. Robot. 35(4), 925–938 (2019)

    Article  Google Scholar 

  27. Zhang, L., Wei, L., Shen, P., Wei, W., Zhu, G., Song, J.: Semantic SLAM based on object detection and improved octomap. IEEE Access 6, 75545–75559 (2018)

    Article  Google Scholar 

  28. Zhong, F., Wang, S., Zhang, Z., Wang, Y.: Detect-SLAM: Making object detection and SLAM mutually beneficial. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp 1001–1010. IEEE (2018)

  29. Zitnick, C., Dollár, P.: Edge boxes: Locating object proposals from edges. In: European Conference on Computer Vision, pp 391–405. Springer (2014)

Download references

Acknowledgments

We would like to thank Mathieu Labbé, the creator of RTAB-Map, for his Herculean effort in maintaining the project, and for always responding quickly and accurately to our queries. We also thank PAL Robotics team for their support related to TIAGo (both in the real world and in the simulation). We also thank the Brazilian research agencies CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico), and FACEPE (Fundação de Amparo à Ciência e Tecnologia do Estado de Pernambuco) for financial support for this research.

Funding

This paper was supported by CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and FACEPE (Fundação de Amparo à Ciência e Tecnologia do Estado de Pernambuco).

Author information

Authors and Affiliations

  1. Centro de Informática, Universidade Federal de Pernambuco, Recife, Brazil

    Felipe D. B. de Oliveira, Marcondes R. da Silva Jr. & Aluizio F. R. Araújo

Authors
  1. Felipe D. B. de Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcondes R. da Silva Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aluizio F. R. Araújo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

– Felipe Duque-Belfort: devised the main idea, wrote most of the code, wrote the text of the article.

– Marcondes R. S. Júnior: wrote part of the code, developed the deep learning-related subjects.

– Aluizio F. R. Araujo: responsible for overall supervision and orientation, English corrections and helped with the experimental setup.

Corresponding author

Correspondence to Felipe D. B. de Oliveira.

Ethics declarations

Conflict of Interests

No conflicts of interest to be reported.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Oliveira, F.D.B., da Silva, M.R. & Araújo, A.F.R. Spatio-temporal Data Association for Object-augmented Mapping. J Intell Robot Syst 103, 1 (2021). https://doi.org/10.1007/s10846-021-01445-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-021-01445-8

Keywords

Search

Navigation