Skip to main content

Advertisement

Springer Nature Link
Log in

Remote object navigation for service robots using hierarchical knowledge graph in human-centered environments

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Remote object navigation (RON), defined as navigating to a remote object that is invisible in the current view, is an inevitable and extremely challenging task for a service robot, particularly when facing unstructured or dynamic human-centered environments. How to apply object-level semantic knowledge about the scene (called scene knowledge graph, SKG) to assist robots in cognition of the environment has become a hot research topic in robot intelligence. In this paper, we propose a knowledge-based RON method to skillfully combine the hierarchical knowledge in SKG and the probability-based navigation strategy. In detail, we first develop an automated pipeline to construct a novel SKG from massive visual data in real indoor environments. Then we propose a reasoner to derive the probabilistic representation of the hierarchical knowledge contained in the SKG. Additionally, a two-stage navigator composed of global path planning and local search strategy is applied as a distance-aware task planner to reduce the navigation path cost. The experimental results in real-world scenarios indicate that the proposed method has efficient performance and robustness on RON task compared to other approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Liu B, Xiao X, Stone P (2021) A lifelong learning approach to mobile robot navigation. IEEE Robot Autom Lett 6(2):1090–1096. https://doi.org/10.1109/LRA.2021.3056373

    Article  Google Scholar 

  2. Thrun S (1998) Learning metric-topological maps for indoor mobile robot navigation. Artif Intell 99(1):21–71

    Article  Google Scholar 

  3. Grisettiyz G, Stachniss C, Burgard W (2005) Improving grid-based slam with rao-blackwellized particle filters by adaptive proposals and selective resampling. In: Proceedings of the 2005 IEEE international conference on robotics and automation, pp 2432–2437. IEEE

  4. Wu P, Kong L, Gao S (2012) Holography map for home robot: an object-oriented approach. Intel Serv Robot 5(3):147–157

    Article  Google Scholar 

  5. Chaplot DS, Gandhi DP, Gupta A, Salakhutdinov RR (2020) Object goal navigation using goal-oriented semantic exploration. Adv Neural Inf Process Syst 33

  6. Nüchter A, Hertzberg J (2008) Towards semantic maps for mobile robots. Robot Auton Syst 56(11):915–926

    Article  Google Scholar 

  7. Ruiz-Sarmiento J-R, Galindo C, Gonzalez-Jimenez J (2015) Exploiting semantic knowledge for robot object recognition. Knowl-Based Syst 86:131–142

    Article  Google Scholar 

  8. Grinvald M, Furrer F, Novkovic T, Chung JJ, Cadena C, Siegwart R, Nieto J (2019) Volumetric instance-aware semantic mapping and 3d object discovery. IEEE Robot Autom Lett 4(3):3037–3044

    Article  Google Scholar 

  9. Wang Z, Tian G, Shao X (2020) Home service robot task planning using semantic knowledge and probabilistic inference. Knowl-Based Syst 204:106174

    Article  Google Scholar 

  10. Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Thirty-first AAAI conference on artificial intelligence

  11. Fabian M, Gjergji K, Gerhard W et al (2007) Yago: a core of semantic knowledge unifying wordnet and wikipedia. In: 16th international world wide web conference, WWW, pp 697–706

  12. Anousaki G, Kyriakopoulos KJ (1999) Simultaneous localization and map building for mobile robot navigation. IEEE Robot Autom Mag 6(3):42–53

    Article  Google Scholar 

  13. Leonard JJ, Durrant-Whyte HF, Cox IJ (1992) Dynamic map building for an autonomous mobile robot. Int J Robot Res 11(4):286–298

    Article  Google Scholar 

  14. Bosse M, Zlot R (2008) Map matching and data association for large-scale two-dimensional laser scan-based slam. Int J Robot Res 27(6):667–691

    Article  Google Scholar 

  15. Hess W, Kohler D, Rapp H, Andor D (2016) Real-time loop closure in 2D lidar slam. In: 2016 IEEE international conference on robotics and automation (ICRA), pp 1271–1278. IEEE

  16. Taheri H, Xia ZC (2021) Slam; definition and evolution. Eng Appl Artif Intell 97:104032

    Article  Google Scholar 

  17. LaValle SM, Kuffner JJ, Donald B et al (2001) Rapidly-exploring random trees: progress and prospects. Algorithmic Comput Robot New Dir 5:293–308

    MATH  Google Scholar 

  18. Kavraki LE, Svestka P, Latombe J-C, Overmars MH (1996) Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Trans Robot Autom 12(4):566–580

    Article  Google Scholar 

  19. Yamauchi B (1997) A frontier-based approach for autonomous exploration. In: Proceedings 1997 IEEE international symposium on computational intelligence in robotics and automation CIRA’97.’Towards new computational principles for robotics and automation’, pp 146–151. IEEE

  20. Chaplot DS, Sathyendra KM, Pasumarthi RK, Rajagopal D, Salakhutdinov R (2018) Gated-attention architectures for task-oriented language grounding. In: Thirty-second AAAI conference on artificial intelligence

  21. Yang S, Li G, Yu Y (2020) Graph-structured referring expression reasoning in the wild. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9952–9961

  22. Vasudevan AB, Dai D, Van Gool L (2021) Talk2nav: long-range vision-and-language navigation with dual attention and spatial memory. Int J Comput Vis 129(1):246–266

    Article  Google Scholar 

  23. Gupta S, Davidson J, Levine S, Sukthankar R, Malik J (2017) Cognitive mapping and planning for visual navigation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2616–2625

  24. Henriques JF, Vedaldi A (2018) Mapnet: an allocentric spatial memory for mapping environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8476–8484

  25. Singh NH, Thongam K (2019) Neural network-based approaches for mobile robot navigation in static and moving obstacles environments. Intel Serv Robot 12(1):55–67

    Article  Google Scholar 

  26. Wu Y, Wu Y, Tamar A, Russell S, Gkioxari G, Tian Y (2018) Learning and planning with a semantic model. arXiv preprint arXiv:1809.10842

  27. Savva M, Kadian A, Maksymets O, Zhao Y, Wijmans E, Jain B, Straub J, Liu J, Koltun V, Malik J, et al. (2019) Habitat: a platform for embodied ai research. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9339–9347

  28. Shen B, Xia F, Li C, Martín-Martín R, Fan L, Wang G, Buch S, D’Arpino C, Srivastava S, Tchapmi LP, et al. (2020) igibson, a simulation environment for interactive tasks in large realistic scenes. arXiv preprint arXiv:2012.02924

  29. Qi Y, Wu Q, Anderson P, Wang X, Wang WY, Shen C, Hengel Avd (2020) Reverie: remote embodied visual referring expression in real indoor environments. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9982–9991

  30. Morad SD, Mecca R, Poudel RP, Liwicki S, Cipolla R (2021) Embodied visual navigation with automatic curriculum learning in real environments. IEEE Robot Autom Lett 6(2):683–690

    Article  Google Scholar 

  31. Giuliari F, Castellini A, Berra R, Del Bue A, Farinelli A, Cristani M, Setti F, Wang Y (2021) Pomp++: Pomcp-based active visual search in unknown indoor environments. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 1523–1530. IEEE

  32. Zhang S, Song X, Bai Y, Li W, Chu Y, Jiang S (2021) Hierarchical object-to-zone graph for object navigation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 15130–15140

  33. Liang Y, Chen B, Song S (2021) Sscnav: confidence-aware semantic scene completion for visual semantic navigation. In: 2021 IEEE international conference on robotics and automation (ICRA), pp 13194–13200. IEEE

  34. Quillan R (1963) A notation for representing conceptual information: an application to semantics and mechanical english paraphrasing

  35. Ruiz-Sarmiento J-R, Galindo C, Gonzalez-Jimenez J (2017) Building multiversal semantic maps for mobile robot operation. Knowl-Based Syst 119:257–272

    Article  Google Scholar 

  36. Lorbach M, Höfer S, Brock O (2014) Prior-assisted propagation of spatial information for object search. In: 2014 IEEE/RSJ international conference on intelligent robots and systems, pp 2904–2909. IEEE

  37. Kim U-H, Park J-M, Song T-J, Kim J-H (2019) 3-d scene graph: a sparse and semantic representation of physical environments for intelligent agents. IEEE Trans Cybern 50(12):4921–4933

  38. Armeni I, He Z-Y, Gwak J, Zamir AR, Fischer M, Malik J, Savarese S (2019) 3d scene graph: a structure for unified semantics, 3d space, and camera. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 5664–5673

  39. Zhang S, Hao A, Qin H, et al. (2021) Knowledge-inspired 3d scene graph prediction in point cloud. Adv Neural Inf Process Syst 34

  40. Zeng Z, Röfer A, Jenkins OC (2020) Semantic linking maps for active visual object search. In: 2020 IEEE international conference on robotics and automation (ICRA), pp 1984–1990. IEEE

  41. Yang W, Wang X, Farhadi A, Gupta A, Mottaghi R (2018) Visual semantic navigation using scene priors. arXiv preprint arXiv:1810.06543

  42. Ke L, Li X, Bisk Y, Holtzman A, Gan Z, Liu J, Gao J, Choi Y, Srinivasa S (2019) Tactical rewind: self-correction via backtracking in vision-and-language navigation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6741–6749

  43. Mantelli M, Pittol D, Maffei R, Torresen J, Prestes E, Kolberg M (2021) Semantic active visual search system based on text information for large and unknown environments. J intell Robot Syst 101(2):1–23

    Article  Google Scholar 

  44. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28:91–99

    Google Scholar 

  45. Howard RA (1960) Dynamic programming and Markov processes

  46. Chang A, Dai A, Funkhouser T, Halber M, Niessner M, Savva M, Song S, Zeng A, Zhang Y (2017) Matterport3d: learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158

  47. Smouse PE, Long JC (1992) Matrix correlation analysis in anthropology and genetics. Am J Phys Anthropol 35(S15):187–213

    Article  Google Scholar 

  48. Kolve E, Mottaghi R, Han W, VanderBilt E, Weihs L, Herrasti A, Gordon D, Zhu Y, Gupta A, Farhadi A (2017) Ai2-thor: an interactive 3d environment for visual AI. arXiv preprint arXiv:1712.05474

  49. Qiu Y, Pal A, Christensen HI (2020) Learning hierarchical relationships for object-goal navigation. arXiv preprint arXiv:2003.06749

  50. Zhou X, Wang D, Krähenbühl P (2019) Objects as points. arXiv preprint arXiv:1904.07850

  51. Hart PE, Nilsson NJ, Raphael B (1968) A formal basis for the heuristic determination of minimum cost paths. IEEE Trans Syst Sci Cybern 4(2):100–107

    Article  Google Scholar 

  52. Ge Z, Liu S, Wang F, Li Z, Sun J (2021) Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430

Download references

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (52172376) and the Project Fund of the GENERAL ADMINISTRATION OF CUSTOMS.P.R.CHINA (2021HK261). Thanks for the support of the China Electronic Port Data Center Beijing Branch. The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation.

Author information

Authors and Affiliations

  1. School of Transportation Science and Engineering, Beihang University, Xueyuan Road, Beijing, 100191, China

    Yongwei Li, Xiang Huo & Xinkai Wu

  2. Beijing Robint Technology Co. Ltd., Xueyuan Road, Beijing, 100191, China

    Yalong Ma

Authors
  1. Yongwei Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yalong Ma
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xiang Huo
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xinkai Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Conceptualization, methodology, data curation, formal analysis and writing—original draft preparation were performed by YL. Software and visualization were performed by YM. Resources and writing—reviewing and editing were performed by XH. Supervision and writing—reviewing and editing were performed by XW. The first draft of the manuscript was written by YL, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Xinkai Wu.

Ethics declarations

Competing interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Ma, Y., Huo, X. et al. Remote object navigation for service robots using hierarchical knowledge graph in human-centered environments. Intel Serv Robotics 15, 459–473 (2022). https://doi.org/10.1007/s11370-022-00428-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-022-00428-4

Keywords