Skip to main content
SpringerLink
Log in

Improving robot’s perception of uncertain spatial descriptors in navigational instructions by evaluating influential gesture notions

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Human-friendly interactive features are preferred for service robots used in emerging areas of robotic applications such as caretaking, health care, assistance, education and entertainment since they are intended to be operated by non-expert users. Humans prefer to use voice instructions, responses, and suggestions in their daily interactions. Such voice instructions and responses often include uncertain spatial descriptors such as “little” and “far”, which have no definitive quantitative meaning. Service robots involve direct interactions with human users through voice communication. Therefore, the ability to effectively quantify the meaning of such uncertain spatial descriptors is necessary for human-friendly service robots. This paper proposes a novel method to quantify the uncertain spatial descriptors in navigational instructions based on the current environmental setting and the influential notions conveyed by the pointing gestures that accompany voice instructions. The uncertain spatial descriptors are quantified by a fuzzy inference system that evaluates the spatial parameters of the current environment and the influential notions conveyed by pointing gestures, if available. According to the obtained experimental results, the proposed method is capable of improving the quantification ability of uncertain spatial descriptors by robots.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Aaltonen I, Arvola A, Heikkila P, Lammi H (2017) Hello pepper, may i tickle you?: children’s and adults’ responses to an entertainment robot at a shopping mall. In: Proceedings of the companion of the 2017 ACM/IEEE international conference on human–robot interaction, ACM, pp 53–54

  2. Arkin RC (1998) Behavior-based robotics. MIT Press, Cambridge

    Google Scholar 

  3. Bethel CL, Murphy RR (2010) Review of human studies methods in HRI and recommendations. Int J Soc Robot 2(4):347–359

    Article  Google Scholar 

  4. Boboc RG, Dumitru AI, Antonya C (2015) Point-and-command paradigm for interaction with assistive robots. Int J Adv Robot Syst 12(6):75

    Article  Google Scholar 

  5. Colonnesi C, Stams GJJ, Koster I, Noom MJ (2010) The relation between pointing and language development: a meta-analysis. Dev Rev 30(4):352–366

    Article  Google Scholar 

  6. Di Nuovo A, Broz F, Wang N, Belpaeme T, Cangelosi A, Jones R, Esposito R, Cavallo F, Dario P (2018) The multi-modal interface of robot-era multi-robot services tailored for the elderly. Intell Serv Robot 11(1):109–126. https://doi.org/10.1007/s11370-017-0237-6

    Article  Google Scholar 

  7. Ellis PD (2010) The essential guide to effect sizes: statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press, Cambridge

    Book  Google Scholar 

  8. Fasola J, Matarić MJ (2013) Using spatial semantic and pragmatic fields to interpret natural language pick-and-place instructions for a mobile service robot. In: International conference on social robotics. Springer, Berlin, pp 501–510

  9. Fischinger D, Einramhof P, Papoutsakis K, Wohlkinger W, Mayer P, Panek P, Hofmann S, Koertner T, Weiss A, Argyros A et al (2016) Hobbit, a care robot supporting independent living at home: first prototype and lessons learned. Robot Autonom Syst 75:60–78

    Article  Google Scholar 

  10. Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Autonom Syst 42(3):143–166

    Article  Google Scholar 

  11. Frennert S, Östlund B (2014) Review: seven matters of concern of social robots and older people. Int J Soc Robot 6(2):299–310

    Article  Google Scholar 

  12. Hemachandra S, Duvallet F, Howard TM, Roy N, Stentz A, Walter MR (2015) Learning models for following natural language directions in unknown environments. In: 2015 IEEE international conference on robotics and automation (ICRA), IEEE, pp 5608–5615

  13. Huang S, Tanioka T, Locsin R, Parker M, Masory O (2011) Functions of a caring robot in nursing. In: 2011 7th International conference natural language processing and knowledge engineering, pp 425–429

  14. Hunt A (2000) Jspeech grammar format. W3C Note, June

  15. Jayasekara AGBP, Watanabe K, Kiguchi K, Izumi K (2010) Interpretation of fuzzy voice commands for robots based on vocal cues guided by user’s willingness. In: 2010 IEEE/RSJ international conference intelligent robots and systems, pp 778–783

  16. Jayawardena C, Watanabe K, Izumi K (2007) Controlling a robot manipulator with fuzzy voice commands using a probabilistic neural network. Neural Comput Appl 16(2):155–166

    Article  Google Scholar 

  17. Katsamanis A, Pitsikalis V, Theodorakis S, Maragos P (2017) Multimodal gesture recognition. In: The handbook of multimodal-multisensor interfaces. Association for Computing Machinery and Morgan & Claypool, pp 449–487

  18. Kawamura K, Bagchi S, Park T (1994) An intelligent robotic aid system for human services. NASA Conf. Publication, NASA, pp 413–413

  19. Kita S (2003) Pointing: where language, culture, and cognition meet. Psychology Press, London

    Book  Google Scholar 

  20. Kleanthous S, Christophorou C, Tsiourti C, Dantas C, Wintjens R, Samaras G, Christodoulou E (2016) Analysis of elderly users’ preferences and expectations on service robot’s personality, appearance and interaction. In: International springer, conference human aspects of IT for the aged population, pp 35–44

  21. Koceski S, Koceska N (2016) Evaluation of an assistive telepresence robot for elderly healthcare. J Med Syst 40(5):1–7. https://doi.org/10.1007/s10916-016-0481-x

    Article  Google Scholar 

  22. Kollar T, Tellex S, Roy D, Roy N (2010) Toward understanding natural language directions. In: 2010 5th ACM/IEEE international conference on human–robot interaction (HRI), IEEE, pp 259–266

  23. Kopp S, Bergmann K (2017) Using cognitive models to understand multimodal processes: the case for speech and gesture production. In: The Handbook of multimodal-multisensor interfaces. Association for Computing Machinery and Morgan & Claypool, pp 239–276

  24. Kruijff GJM, Zender H, Jensfelt P (2007) Situated dialogue and spatial organization: what, whereâ and why? Int J Adv Robot Syst 4(1):16

    Article  Google Scholar 

  25. Lalanne D, Nigay L, Robinson P, Vanderdonckt J, Ladry JF, et al (2009) Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 international conference on multimodal interfaces, ACM, pp 153–160

  26. Lin CT, Kan MC (1998) Adaptive fuzzy command acquisition with reinforcement learning. IEEE Trans Fuzzy Syst 6(1):102–121

    Article  Google Scholar 

  27. Liu R, Zhang X (2016) Fuzzy context-specific intention inference for robotic caregiving. Int J Adv Robot Syst 13(5):1–14

    Google Scholar 

  28. Matuszek C, Bo L, Zettlemoyer L, Fox D (2014) Learning from unscripted deictic gesture and language for human–robot interactions. In: 28th AAAI conference on artificial intelligence, pp 2556–2563

  29. Mavridis N (2015) A review of verbal and non-verbal human-robot interactive communication. Robot Autonom Syst 63:22–35

    Article  MathSciNet  Google Scholar 

  30. McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, Chicago

    Google Scholar 

  31. Muthugala MAVJ, Jayasekara AGBP (2016a) Interpretation of uncertain information in mobile service robots by analyzing surrounding spatial arrangement based on occupied density variation. In: 2016 IEEE/RSJ International conference on intelligent robots and systems (IROS), IEEE, pp 1517–1523

  32. Muthugala MAVJ, Jayasekara AGBP (2016b) MIRob: an intelligent service robot that learns from interactive discussions while handling uncertain information in user instructions. In: 2016 Moratuwa engineering research conference (MERCon), IEEE, pp 397–402

  33. Muthugala MAVJ, Jayasekara AGBP (2018) A review of service robots coping with uncertain information in natural language instructions. IEEE Access PP(99):1–1. https://doi.org/10.1109/ACCESS.2018.2808369

  34. Muthugala MAVJ, Srimal PHDAS, Jayasekara AGBP (2017) Enhancing interpretation of ambiguous voice instructions based on the environment and the user’s intention for improved human-friendly robot navigation. Appl Sci 7(8):821

    Article  Google Scholar 

  35. Neßelrath R, Moniri MM, Feld M (2016) Combining speech, gaze, and micro-gestures for the multimodal control of in-car functions. In: 2016 12th International conference on intelligent environments (IE), IEEE, pp 190–193

  36. Nobe S (2000) Where do most spontaneous representational gestures actually occur with respect to speech. Lang Gesture 2:186

    Article  Google Scholar 

  37. Palopoli L, Argyros A, Birchbauer J, Colombo A, Fontanelli D, Legay A, Garulli A, Giannitrapani A, Macii D, Moro F et al (2015) Navigation assistance and guidance of older adults across complex public spaces: the dali approach. Intell Serv Robot 8(2):77–92

    Article  Google Scholar 

  38. Pulasinghe K, Watanabe K, Izumi K, Kiguchi K (2004) Modular fuzzy-neuro controller driven by spoken language commands. IEEE Trans Syst Man Cybern B, Cybern 34(1):293–302

    Article  Google Scholar 

  39. Reich-Stiebert N, Eyssel F (2015) Learning with educational companion robots? toward attitudes on education robots, predictors of attitudes, and application potentials for education robots. Int J Soc Robot 7(5):875–888

    Article  Google Scholar 

  40. Robinson H, MacDonald B, Broadbent E (2014) The role of healthcare robots for older people at home: a review. Int J Soc Robot 6(4):575–591

    Article  Google Scholar 

  41. Schiffer S, Ferrein A (2016) Decision-theoretic planning with fuzzy notions in golog. Int J Uncert Fuzz Knowl Based Syst 24:123–143

    Article  MathSciNet  Google Scholar 

  42. Schiffer S, Ferrein A, Lakemeyer G (2012) Caesar: an intelligent domestic service robot. Intell Serv Robot 5(4):259–273

    Article  Google Scholar 

  43. Skubic M, Perzanowski D, Blisard S, Schultz A, Adams W, Bugajska M, Brock D (2004) Spatial language for human–robot dialogs. IEEE Trans Syst Man Cybern C Appl Rev 34(2):154–167

    Article  Google Scholar 

  44. Tapus A, Tapus C, Mataric MJ (2008) User–robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intell Serv Robot 1(2):169–183

    Article  Google Scholar 

  45. Whitney D, Eldon M, Oberlin J, Tellex S (2016) Interpreting multimodal referring expressions in real time. In: 2016 IEEE International conference robotics and automation (ICRA), IEEE, pp 3331–3338

  46. Zhang Y, Hu Y, Zhang P, Zhang W (2014) Development of personal assistant system with human computer interaction. Int J Hum Comput Interact (IJHCI) 5(3):40

    MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors acknowledge the commitment of the volunteers, who have participated in the experiments.

Author information

Author notes

  1. M. A. Viraj J. Muthugala

    Present address: Engineering Product Development Pillar, Singapore University of Technology and Design, Singapore, Singapore

  2. P. H. D. Arjuna S. Srimal

    Present address: Department of Electrical and Computer Engineering, University of Manitoba, Winnipeg, Canada

Authors and Affiliations

  1. Intelligent Service Robotics Group, Department of Electrical Engineering, University of Moratuwa, Moratuwa, 10400, Sri Lanka

    M. A. Viraj J. Muthugala, P. H. D. Arjuna S. Srimal & A. G. Buddhika P. Jayasekara

Authors
  1. M. A. Viraj J. Muthugala
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. H. D. Arjuna S. Srimal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. G. Buddhika P. Jayasekara
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. A. Viraj J. Muthugala.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the University of Moratuwa Senate Research Grant No. SRC/CAP/2017/03.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 9157 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Muthugala, M.A.V.J., Srimal, P.H.D.A.S. & Jayasekara, A.G.B.P. Improving robot’s perception of uncertain spatial descriptors in navigational instructions by evaluating influential gesture notions. J Multimodal User Interfaces 15, 11–24 (2021). https://doi.org/10.1007/s12193-020-00328-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-020-00328-w

Keywords

Search

Navigation