Skip to main content
Springer Nature Link
Log in

Deep Learning Methods Integration for Improving Natural Interaction Between Humans and an Assistant Mobile Robot in the Context of Autonomous Navigation

  • Conference paper
  • First Online:
Progress in Artificial Intelligence (EPIA 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13566))

Included in the following conference series:

  • 1664 Accesses

Abstract

This paper describes a full navigation architecture which includes a set of available Deep Learning-based modules, focused on “speech to text” and “text to speech” translation, and face recognition, for enabling natural interaction between a smart mobile assistant robot and its human users, in a context of autonomous navigation. The system is novel because it allows complex spoken commands to be syntactically analyzed in Spanish and transformed into motion plans, ready to be executed by the robot, by using the well-known Navigation stack included in the ROS ecosystem. A novel computationally efficient approach (to semantically label the free space from raw data provided by a low cost laser scanner device), enables the generation of a labelled polygonal map, which enhances fixed and mobile obstacle avoidance and local planning in real indoor environments. Robot configuration, learned maps, and Deep Learning models adapted to each scenario are safely and privately stored in Google Cloud, allowing the robot to adapt its behavior to different users and settings. The tests which demonstrate the performance of the system, both in simulation and real environments, are also described.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Miller, D.P.: Assistive robotics: an overview. In: Mittal, V.O., Yanco, H.A., Aronis, J., Simpson, R. (eds.) Assistive Technology and Artificial Intelligence, vol. 1458, pp. 126–136. Springer, Heidelberg (2006). https://doi.org/10.1007/BFb0055975

    Chapter  Google Scholar 

  2. Brose, S.W., et al.: The role of assistive robotics in the lives of persons with disability. Am. J. Phys. Med. Rehabil. 89(6), 509–521 (2010)

    Article  Google Scholar 

  3. Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics, Intelligent Robotics and Autonomous Agents Series. MIT Press, Cambridge (2006)

    MATH  Google Scholar 

  4. Tzafestas, S.G.: Mobile robot control and navigation: a global overview. J. Intell. Rob. Syst. 91(1), 35–58 (2018). https://doi.org/10.1007/s10846-018-0805-9

    Article  Google Scholar 

  5. Ravankar, A., Ravankar, A.A., Kobayashi, Y., Hoshino, Y., Peng, C.-C.: Path smoothing techniques in robot navigation: state-of-the-art, current and future challenges. Sensors 18(9), 3170 (2018). https://doi.org/10.3390/s18093170

    Article  Google Scholar 

  6. Chandra, R.: Precise localization for achieving next-generation autonomous navigation: State-of-the-art, taxonomy and future prospects. Comput. Commun. 160, 351–374 (2020). https://doi.org/10.1016/j.comcom.2020.06.007

    Article  Google Scholar 

  7. Crespo, J., Castillo, J.C., Mozos, O.M., Barber, R.: Semantic information for robot navigation: a survey. Appl. Sci. 10(2), 497, 1–28 (2020). https://doi.org/10.3390/app10020497

  8. Yasuda, Y.D.V., Martins, L.E.G., Cappabianco, F.A.M.: Autonomous visual navigation for mobile robots: a systematic literature review. ACM Comput. Surv. 53(1), 1–34 (2020). https://doi.org/10.1145/3368961

    Article  Google Scholar 

  9. Zhu, K., Zhang, T.: Deep reinforcement learning based mobile robot navigation: a review. IEEE Tsinghua Sci. Technol. 26(5), 674–691 (2021). https://doi.org/10.26599/TST.2021.9010012

    Article  Google Scholar 

  10. Shishehgar, M., Kerr, D., Blake, J.: A systematic review of research into how robotic technology can help older people. Smart Health. 7–8, 1–18 (2018). https://www.sciencedirect.com/science/article/pii/S2352648316300149

  11. Tesla Autopilot. https://www.tesla.com/es_ES/autopilot. Accessed 12 Apr 2022

  12. Vaussard, F., et al.: Lessons learned from robotic vacuum cleaners entering the home ecosystem. Robot. Auton. Syst. 62(3), 376–391 (2014). https://doi.org/10.1016/j.robot.2013.09.014

    Article  Google Scholar 

  13. Robaczewski, A., Bouchard, J., Bouchard, K., Gaboury, S.: Socially assistive robots: the specific case of the NAO. Int. J. Soc. Robot. 13(4), 795–831 (2020). https://doi.org/10.1007/s12369-020-00664-7

    Article  Google Scholar 

  14. Pandey, A.K., Gelin, R.: A mass-produced sociable humanoid robot: pepper: the first machine of its kind. IEEE Robot. Autom. Mag. 25(3), 40–48 (2018). https://doi.org/10.1109/MRA.2018.2833157

    Article  Google Scholar 

  15. Këpuska, V., Bohouta, G.: Next-generation of virtual personal assistants (Microsoft Cortana, Apple Siri, Amazon Alexa and Google Home). In: Proceedings of IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), pp. 99–103. IEEE, Las Vegas (2018). https://doi.org/10.1109/CCWC.2018.8301638

  16. Gil, H.: The elderly and the digital inclusion: a brief reference to the initiatives of the European Union and Portugal. MOJ Gerontol. Geriatr. 4(6), 213–221 (2019)

    Article  Google Scholar 

  17. Beaunoyer, E., Dupéré, S., Guitton, M.J.: COVID-19 and digital inequalities: reciprocal impacts and mitigation strategies. Comput. Hum. Behav. 111 (2020). https://doi.org/10.1016/j.chb.2020.106424

  18. AAL Programme. Ageing Well in the Digital World. http://www.aal-europe.eu/about/. Accessed 12 Apr 2022

  19. engage. Enabling social robots. https://engage-aal-project.eu/. Accessed 12 Apr 2022

  20. ReMember-Me. http://www.aal-europe.eu/projects/remember-me/. Accessed 12 Apr 2022

  21. AgeWell. http://www.aal-europe.eu/projects/agewell/. Accessed 12 Apr 2022

  22. eWare. http://www.aal-europe.eu/projects/eware/. Accessed 12 Apr 2022

  23. CAMI. http://www.aal-europe.eu/projects/cami/. Accessed 12 Apr 2022

  24. ASSAM. http://www.aal-europe.eu/projects/assam/. Accessed 12 Apr 2022

  25. ExCITE. http://www.aal-europe.eu/projects/excite/. Accessed 12 Apr 2022

  26. ALIAS. http://www.aal-europe.eu/projects/alias/. Accessed 12 Apr 2022

  27. DOMEO. http://www.aal-europe.eu/projects/domeo/. Accessed 12 Apr 2022

  28. Hannun, A., et al.: Deep speech: scaling up end-to-end speech recognition. arXiv:1412.5567 (2014). https://doi.org/10.48550/arXiv.1412.5567

  29. DeepSpeech. https://github.com/mozilla/DeepSpeech. Accessed 15 Apr 2022

  30. DeepSpeech-polyglot. https://gitlab.com/Jaco-Assistant/Scribosermo. Accessed 15 Apr 2022

  31. Festival. https://www.cstr.ed.ac.uk/projects/festival/. Accessed 15 Apr 2022

  32. CMU Flite. http://cmuflite.org/. Accessed 15 Apr 2022

  33. Pico Text-to-Speech. https://www.openhab.org/addons/voice/picotts/. Accessed 15 Apr 2022

  34. Text to Speech (TTS) library for Python 2 and 3, pyttsx3. https://pypi.org/project/pyttsx3/. Accessed 15 Apr 2022

  35. Talkey Documentation. https://talkey.readthedocs.io/en/latest/. Accessed 15 Apr 2022

  36. spaCy. https://spacy.io/. Accessed 15 Apr 2022

  37. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 815–823. IEEE. Boston (2015). https://doi.org/10.1109/CVPR.2015.7298682

  38. Pavón, N., Ferruz, J., Ollero, A.: Describing the environment using semantic labelled polylines from 2D laser scanned raw data: application to autonomous navigation. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2010), pp. 3257–3262, IEEE, Taipei (2010). https://doi.org/10.1109/IROS.2010.5650846

  39. Pavon-Pulido, N., Lopez-Riquelme, J.A., Pinuaga-Cascales, J.J., Ferruz-Melero, J., Morais Dos Santos, R.: Cybi: a smart companion robot for elderly people: improving teleoperation and telepresence skills by combining cloud computing technologies and fuzzy logic. In: Proceedings of IEEE ICARSC, pp. 198–203, IEEE, Vila Real (2015). https://doi.org/10.1109/ICARSC.2015.40

Download references

Acknowledgements

This work has been partially supported by the project 21668/PDC/21 Program Proof of Concept included in the “Programa Séneca 2021” Region of Murcia, Spain.

Author information

Authors and Affiliations

  1. Technical University of Cartagena, Campus Muralla del Mar, Antiguo Hospital Real de Marina de Cartagena, C/Dr. Fleming s/n, 30202, Cartagena, Spain

    Roberto Oterino-Bono, Nieves Pavón-Pulido, Juan Antonio López-Riquelme, Marta Jiménez-Muñoz & Jorge J. Feliu-Batlle

  2. Clinical and Experimental Neuroscience (NiCE), School of Medicine, Campus Mare Nostrum, University of Murcia, Murcia, Spain

    Jesús Damián Blasco-García & María Trinidad Herrero

Authors
  1. Roberto Oterino-Bono
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nieves Pavón-Pulido
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jesús Damián Blasco-García
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan Antonio López-Riquelme
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marta Jiménez-Muñoz
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jorge J. Feliu-Batlle
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. María Trinidad Herrero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nieves Pavón-Pulido .

Editor information

Editors and Affiliations

  1. ISEP/GECAD, Polytechnic Institute of Porto, Porto, Portugal

    Goreti Marreiros

  2. IST/INESC-ID, University of Lisbon, Lisbon, Portugal

    Bruno Martins

  3. IST/INESC-ID, University of Lisbon, Porto Salvo, Portugal

    Ana Paiva

  4. CISUC, University of Coimbra, Coimbra, Portugal

    Bernardete Ribeiro

  5. IST/INESC-ID, University of Lisbon, Porto Salvo, Portugal

    Alberto Sardinha

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Oterino-Bono, R. et al. (2022). Deep Learning Methods Integration for Improving Natural Interaction Between Humans and an Assistant Mobile Robot in the Context of Autonomous Navigation. In: Marreiros, G., Martins, B., Paiva, A., Ribeiro, B., Sardinha, A. (eds) Progress in Artificial Intelligence. EPIA 2022. Lecture Notes in Computer Science(), vol 13566. Springer, Cham. https://doi.org/10.1007/978-3-031-16474-3_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16474-3_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16473-6

  • Online ISBN: 978-3-031-16474-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics