Skip to main content
SpringerLink
Log in

Data Association and Localization of Classified Objects in Visual SLAM

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Maps generated by many visual Simultaneous Localization and Mapping algorithms consist of geometric primitives such as points, lines or planes. These maps offer a topographic representation of the environment, but they contain no semantic information about the environments. Object classifiers leveraging advances in machine learning are highly accurate and reliable, capable of detecting and classifying thousands of objects. Classifiers can be incorporated into a SLAM pipeline to add semantic information to a scene. Frequently, this semantic information is conducted for each frame of the image, but semantic labeling is not persistent over time. In this work, we present a nonparametric statistical approach to perform matching/association of objects detected over consecutive image frames. These associated classified objects are then localized in the accrued map using an unsupervised clustering method. We test our approach on multiple data sets, and it shows strong performance in terms of objects correctly associated from frame to frame. We also have tested our algorithm on three data sets in our lab environment using tag markers to demonstrate the accuracy of classified object localization process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al.: Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv:1603.04467 (2016)

  2. Angeli, A., Filliat, D, Doncieux, S, Meyer, J.: Fast and incremental method for loop-closure detection using bags of visual words. IEEE Trans. Robot. 24, 1027–1037 (2008)

    Article  Google Scholar 

  3. Ataer-Cansizoglu, E., Taguchi, Y.: Object Detection and Tracking in Rgb-D Slam via Hierarchical Feature Grouping. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4164–4171 (2016)

  4. Berkhin, P.: A survey of clustering data mining techniques. Grouping multidimensional data, pp 25–71. Springer, Berlin (2006)

    Book  Google Scholar 

  5. Bhattacharyya, A.: On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math. Soc., 99–109 (1943)

  6. Bowman, S.L., Atanasov, N., Daniilidis, K., Pappas, G.J.: Probabilistic data association for semantic slam. In: IEEE International Conference on Robotics and Automation, pp. 1722–1729 (2017)

  7. Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., Leonard, J.J.: Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Trans. Robot. 32, 1309–1332 (2016)

    Article  Google Scholar 

  8. Sander, C.M.: The hdbscan clustering library. http://hdbscan.readthedocs.io/.

  9. Campello, R.J., Moulavi, D., Sander, J.: Density-based clustering based on hierarchical density estimates. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining, Springer. pp. 160–172 (2013)

  10. Castle, R.O., Gawley, D.J., Klein, G., Murray, D.W.: Towards simultaneous recognition, localization and mapping for hand-held and wearable cameras. In: IEEE International Conference on Robotics and Automation, pp. 4102–4107 (2007)

  11. Castle, R.O., Klein, G., Murray, D.W.: Combining monoslam with object recognition for scene augmentation using a wearable camera. Image Vis. Comput. 28, 1548–1556 (2010)

    Article  Google Scholar 

  12. Choudhary, S., Trevor, A.J., Christensen, H.I., Dellaert, F.: Slam with object discovery, modeling and mapping. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 1018–1025 (2014)

  13. Civera, J., Galvez-Lopez, D., Riazuelo, L., Tardes, J.D., Montiel, J.M.M.: Towards semantic slam using a monocular camera. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1277–1284 (2011)

  14. Collobert, R., Bengio, S., Mariéthoz, J.: Torch: a modular machine learning software library. Technical Report. Idiap (2002)

  15. Durrant-Whyte, H., Bailey, T.: Simultaneous localization and mapping: part I. IEEE Robot. Autom. Mag. 13, 99–110 (2006)

    Article  Google Scholar 

  16. Ekvall, S., Jensfelt, P., Kragic, D.: Integrating active mobile robot object recognition and slam in natural environments. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5792–5797 (2006)

  17. Frintrop, S., Jensfelt, P.: Attentional landmarks and active gaze control for visual slam. IEEE Trans. Robot. 24, 1054–1065 (2008)

    Article  Google Scholar 

  18. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587 (2014)

  19. Glocker, B., Izadi, S., Shotton, J., Criminisi, A.: Real-Time Rgb-D Camera Relocalization. In: 2013 IEEE International Symposium on Mixed and Augmented Reality, pp. 173–179 (2013)

  20. Goncalves, L., di Bernardo, E., Benson, D., Svedman, M., Ostrowski, J., Karlsson, N., Pirjanian, P.: A visual front-end for simultaneous localization and mapping. In: IEEE International Conference on Robotics and Automation, pp. 44–49 (2005)

  21. Grinvald, M., Furrer, F., Novkovic, T., Chung, J.J., Cadena, C., Siegwart, R., Nieto, J.: Volumetric instance-aware semantic mapping and 3d object discovery. IEEE Robot. Autom. Lett. 4, 3037–3044 (2019)

    Article  Google Scholar 

  22. Higgins, J.J.: Introduction to modern nonparametric statistics (2003)

  23. Holz, D., Ichim, A.E., Tombari, F., Rusu, R.B., Behnke, S.: Registration with the point cloud library: a modular framework for aligning in 3-d. IEEE Robot. Autom. Mag. 22, 110–124 (2015)

    Article  Google Scholar 

  24. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (2017)

  25. Huynh, D.Q.: Metrics for 3d rotations: Comparison and analysis. J. Math. Imaging Vis. 35, 155–164 (2009)

    Article  MathSciNet  Google Scholar 

  26. Iqbal, A., Gans, N.R.: Localization of classified objects in slam using nonparametric statistics and clustering. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 161–168 (2018)

  27. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. In: ACM International Conference on Multimedia, pp. 675–678. ACM (2014)

  28. Lai, K., Bo, L., Ren, X., Fox, D.: A Large-Scale Hierarchical Multi-View RGB-D Object Dataset. In: 2011 IEEE International Conference on Robotics and Automation, pp. 1817–1824 (2011)

  29. Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollȧr, P., Zitnick, C.L.: Microsoft COCO: common objects in context. arXiv:1405.0312(2014)

  30. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440 (2015)

  31. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  32. Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, 50–60 (1947)

  33. McCormac, J., Handa, A., Davison, A., Leutenegger, S.: Semanticfusion: Dense 3D semantic mapping with convolutional neural networks. In: IEEE International Conference on Robotics and Automation. pp. 4628–4635 (2017)

  34. Mu, B., Liu, S.Y., Paull, L., Leonard, J., How, J.P.: SLAM with Objects using a Nonparametric Pose Graph. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4602–4609 (2016)

  35. Munoz-Salinas, R., Marin-Jimenez, M. J., Yeguas-Bolivar, E., Medina-Carnicer, R.: Mapping and localization from planar markers. Pattern Recogn. 73, 158–171 (2018)

    Article  Google Scholar 

  36. Mur-Artal, R., Tardós, J.D.: Probabilistic Semi-Dense Mapping from Highly Accurate Feature-Based Monocular Slam. In: Robotics, Science and Systems (2015)

  37. Nicholson, L., Milford, M., Sünderhauf, N.: Quadricslam: Dual quadrics from object detections as landmarks in object-oriented slam. IEEE Robot. Autom. Lett. 4, 1–8 (2018)

    Article  Google Scholar 

  38. Pillai, S.: Monocular slam supported object recognition. arXiv:1506.01732 (2015)

  39. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp. 91–99 (2015)

  40. Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H.J., Davison, A.J.: SLAM++: Simultaneous localisation and mapping at the level of objects. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1352–1359 (2013)

  41. Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 573–580 (2012)

  42. Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A Benchmark for the Evaluation of RGB-D SLAM Systems. In: Proceedings of the International Conference on Intelligent Robot Systems (2012)

  43. Sünderhauf, N., Pham, T.T., Latif, Y., Milford, M., Reid, I.: Meaningful maps with object-oriented semantic mapping. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 5079–5085 (2017)

  44. Thrun, S., et al.: Robotic mapping: A survey. ExplorArtif Intell Millennium 1:1

  45. Uijlings, J.R., Van De Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition. Int J Comput Vis 104, 154–171 (2013)

    Article  Google Scholar 

  46. Zender, H., Mozos, O.M., Jensfelt, P., Kruijff, G.J., Burgard, W.: Conceptual spatial representations for indoor mobile robots. Robot. Auton. Syst. 56, 493–502 (2008)

    Article  Google Scholar 

Download references

Acknowledgments

This research has been funded by the Advanced Driver Assistance System (ADAS) group at Texas Instruments (TI) in Dallas, TX. We would also like to thank Viraj Mavani for his support and guidance in capturing the UTD dataset and helping generate the results for the project.

Author information

Authors and Affiliations

  1. University of Texas at Arlington Research Institute, University of Texas at Arlington, Fort Worth, TX, 76118, USA

    Asif Iqbal & Nicholas R. Gans

Authors
  1. Asif Iqbal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas R. Gans
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicholas R. Gans.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the Advanced Driver Assistance System (ADAS) group at Texas Instruments (TI) in Dallas, TX

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iqbal, A., Gans, N.R. Data Association and Localization of Classified Objects in Visual SLAM. J Intell Robot Syst 100, 113–130 (2020). https://doi.org/10.1007/s10846-020-01189-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-020-01189-x

Keywords

Search

Navigation