Skip to main content
Springer Nature Link
Log in

The impact of human–robot multimodal communication on mental workload, usability preference, and expectations of robot behavior

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Multimodal communication between humans and autonomous robots is essential to enhance effectiveness of human–robot team performance in complex, novel environments, such as in military intelligence, surveillance, and reconnaissance operations in urban settings. It is imperative that a systematic approach be taken to evaluate the factors that each modality contributes to the user’s ability to perform successfully and safely. This paper addresses the effects of unidirectional speech and gesture methods of communication on perceived workload, usability preferences, and expectations of robot behavior while commanding a robot teammate to perform a spatial-navigation task. Each type of communication was performed alone or simultaneously. Results reveal that although the speech-alone condition elicited the lowest level of perceived workload, the usability preference and expectations of robot behavior after interacting through each communication condition was the same. Further, workload ratings between the gesture and speech-gesture conditions were similar indicating systems that employ gesture communication could also support speech communication with little to no additional subjectively perceived cognitive burden on the user. Findings also reveal that workload alone should not be used as a sole determining factor of communication preference during system and task evaluation and design. Additionally, perceived workload did not seem to negatively impact the level of expectations regarding the robot’s behavior. Recommendations for future human–robot communication evaluation are provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Phillips E, Ososky S, Grove J, Jenstch F (2011) From tools to teammates: toward the development of appropriate mental models for intelligent robots. In: Proceedings from the human factors and ergonomics society 55th annual meeting, pp 1491–1495

  2. Redden E, Elliott l, Barnes M (2013) Robots: the new team members. In: Coovert M, Thompson L (eds) The psychology of workplace technology. Society of industrial organizational psychology frontiers series, Routledge Press

  3. Barber D, Lackey S, Reinerman-Jones L, Hudson I (2013) Visual and tactile interfaces for bi-directional human robot communication. SPIE defense, security, and sensing. International society for optics and photonics, pp 87410U–87410U

  4. Main D (2011) Robot first responders could map out a building before humans enter. Popular Mechanics. Retrieved 10 Oct 2014. http://www.popularmechanics.com/technology/engineering/robots/robot-first-responders-could-map-out-a-building-before-humans-enter

  5. Gleeson B, Maclean K, Haddadi A, Croft E, Alcazar J (2013) Gestures for industry: intuitive human–robot communication from human observation. In: Proceedings from the 8th ACM/IEEE intenational conference on human–robot interaction. IEEE Press, Piscataway, pp 349–356

  6. Harriott C, Zhang T, Adams J (2011) Evaluating the applicability of current models of workload to peer-based human–robot teams. In: Proceedings from the 6th international conference on human-robot interaction, New York, ACM, pp 45–52

  7. Medicherla H, Sekmen A (2007) Human–robot interaction via voice-controllable intelligent user interface. Robotica 25(5):521–527

    Article  Google Scholar 

  8. Neisser U (1976) Cognition and reality. W.H. Freeman, San Francisco

    Google Scholar 

  9. Lackey SJ, Barber DJ, Reinerman-Jones L, Badler N, Hudson I (2011) Defining next-generation multi-modal communication in human–robot interaction. Human factors and ergonomics society conference. HFES, Las Vegas

  10. Oviatt S (2012) Multimodal interfaces. In: Jacko J (ed) Handbook of human–computer interaction, 3rd edn. Lawrence Erlbaum, Mahwah

    Google Scholar 

  11. Barber D, Reinerman-Jones L, Matthews G (2014) Toward a tactile language for human–robot interaction: two studies of tacton learning and performance. Hum Factors. doi:10.1177/0018720814548063

  12. Jaimes A, Sebe N (2007) Multimodal human–computer interaction: a survey. Comput Vis Image Underst 108(1):116–134

    Article  Google Scholar 

  13. Alonso-Martin F, Castro-González Á, Gorostiza J, Salichs M (2013) Multidomain voice activity detection during human–robot interaction. In: Herrmann MPG, Bremmer P, Spiers A, Leonards U (eds) Social robotics. Springer International Publishing, Bristol, pp 64–73

    Chapter  Google Scholar 

  14. Kamm C, Walker M, Rabiner L (1997) The role of speech processing in human–computer intelligent communication. Speech Commun 23:263–278

    Article  Google Scholar 

  15. Breazeal C, Aryananda L (2002) Recognition of affective communicative intent in robot-directed speech. Auton Robots 12:83–104

    Article  MATH  Google Scholar 

  16. Beidel E (2011) www.NationalDefenseMagazine.org. Retrieved 14 Jan 2015, from Army shift focus to dismounted Soldiers. http://www.nationaldefensemagazine.org/archive/2011/April/Pages/ArmyShiftsFocustoDismountedSoldiers.aspx

  17. Tikhanoff V, Cangelosi A, Metta G (2011) Integration of speech and action in humanoid robots: iCub simulation experiments. IEEE Trans Auton Mental Dev 3(1):17–29

    Article  Google Scholar 

  18. Erickson D, DeWees M, Lewis J, Matson E (2012) Communication for task completion with heterogeneous robots. In Kim J-H, Matson E, Myung H, Xu P (eds) Robot intelligence technology and applications 2012, vol 208, pp 873–882

  19. Pourmehr S, Monajjemi V, Vaughn R, Mori G. (2013) “You two! Take off!”: creating, modifying and commanding groups of robots using face engagement and indirect speech in voice commands. In: Proceedings from IEEE/RSJ international conference on intelligent robots and systems 2013, pp 137–142. IEEE

  20. Stiefelhagen R, Ekenel HK, Fügen C, Gieselmann P, Holzapfel H, Kraft F, Nickel K, Voit M, Waibel A (2007) Enabling multimodal human–robot interaction for the Karlsruhe humanoid robot. IEEE Trans Robot 23(5):840–851

    Article  Google Scholar 

  21. Pettitt R, Redden E, Carsten C (2009) Scalablity of robotic controllers: speech-based robotic controller evaluation (ARL-TR-4858). US Army Research Laboratory, Aberdeen Proving Ground

    Google Scholar 

  22. Pettitt R, Carstens C, Elliot L (2014) Speech-based robotic control for dismounted soldiers: evaluation of visual display options. Army Research Laboratory, Department of Defense, USA

  23. Kennedy W, Bugajska M, Marge M, Adams W, Fransen B, Perzanowski D, Schultz AC, Trafton G (2007) Spatial representation and reasoning for human–robot collaboration. In: Proceedings of the 22nd conference on artificial intelligence. AAAI Press, Vancouver, pp 1554–1559

  24. Harris J, Barber D (2014) Speech and gesture interfaces for squad level human robot teaming. In: Proceedings from SPIE 9084: Unmanned Systems Technology XVI, 90840B. Baltimore

  25. Redden E, Carstens C, Pettitt R (2010) Intuitive speech-based robotic control. Army Research Lab, Department of Defense, USA

  26. Jung S-W, Sung K-W, Park M-Y, Kang E-U, Hwang W-J, Won J-D et al (2013) A study on robust control of mobile robot by voice command. In: Proceedings of the 13th international conference on control, automation and systems (ICCAS), pp 657–659. Gwangju. doi:10.1109/ICCAS.2013.6703950

  27. Pigeon S, Swail C, Geoffrois E, Bruckner G, van Leeuwen D, Teixeira C et al (2005) Use of speech and language technology in military environments. Res Technol Organ

  28. Ekman P (2004) Emotional and conversational nonverbal signals. In: Larrazabal M, Miranda L (eds) Language, knowledge, and representation. Kluwer Academic Publishers, Amsterdam, pp 39–50

    Chapter  Google Scholar 

  29. Mehrabian A (1972) Nonverbal communication. Aldine Transaction, New Brunswick

    Google Scholar 

  30. Ekman P, Friesen W (1968) Nonverbal behavior in psychotherapy research. In: Shlien J (ed) Research in psychotherapy conference. American Psychological Association, Washington, pp 179–216

    Chapter  Google Scholar 

  31. Efron D (1941) Gesture and environment. King’s Crown, New York

    Google Scholar 

  32. Stajonov A (2009) Gesture-based human–computer interaction [Doctoral dissertation]. Jacobs University, Bremen

    Google Scholar 

  33. Burger B, Ferrané I, Lerasle F, Infantes G (2012) Two-handed gesture recognition and fusion with speech to command a robot. Auton Robot 32(1):129–147

    Article  Google Scholar 

  34. Barber D, Abich J IV, Phillips E, Talone A, Jentsch F, Hill S (2015) Field assessment of multimodal communication for dismounted human–robot teams. In: Proceedings of the 59th human factors and ergonomics society 2014, Los Angeles

  35. Boonpinon N, Sudsng A (2008) Formation control for multi-robot teams using a data glove. In: Proceedings from the IEEE conference on robotics, automation, and mechatronics Chengdu, IEEE pp 525–531

  36. Hill S, Barber D, Evans A (2015) Achieving the vision of effective Soldier–robot teaming: recent work in multimodal communication. In: HRI ’15 Extended Abstracts. Tenth annual ACM/IEEE international conference on human–robot interaction, ACM, Portland, pp 177–178

  37. Rogalla O, Ehrenmann M, Zöllner R, Becher R, Dillmann R (2002) Using gesture and speech control for commanding a robot assistant. In: Proceedings of the 11th IEEE international workshop on robot and human interactive communication, pp 454–459. IEEE

  38. Cheng L, Sun Q, Su H, Cong Y (2012). Design and implementation of human–robot interactive demonstrationn system based on Kinect. In: IEEE 24th Chinese: control and decision conference, pp 971–975. IEEE

  39. Tran N, Phan H, Dinh V, Ellen J, Berg B, Lum J et al (2009) Wireless data glove for gesture-based robotic control. In Jacko J (ed) Human–computer interaction. Novel interaction methods and techniques: 13th international conference, HCI International 2009 Springer, Berlin, vol 5611, pp 271–280

  40. Wachs J, Kölsch M, Stern H, Edan Y (2011) Vision-based hand-gesture applications. Commun ACM 54(2):60–71

    Article  Google Scholar 

  41. Dumas B, Lalanne D, Oviatt S (2009) Multimodal interfaces: a survey of principles, models, and frameworks. In: Lalanne D, Kohlas J (eds) Human machine interaction: lecture notes in computer science. Springer, Berlin Heidelberg, Berlin, pp 3–26

    Chapter  Google Scholar 

  42. Sarter N (2006) Multiple-resource theory as a basis for multimodal interface design: success stories, qualifications, and research needs. In: Kramer A, Wiegmann D, Kirlik A (eds) Attention: from theory to practice. Oxford University Press, New York, pp 187–195

    Chapter  Google Scholar 

  43. Wickens C (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3(2):159–177

    Article  Google Scholar 

  44. Vitense H, Jacko J, Emery V (2003) Multimodal feedback: an assessment of performance and mental workload. Ergonomics 46(1–3):68–87

    Article  Google Scholar 

  45. Perzanowski D, Adams W, Schultz A, Marsh E (2000) Towards seamless integration in a multi-modal interface. Naval Research Lab, Department of Defense, USA

  46. Perzanowski D, Schultz AC, Adams W, Marsh E (2000) Using a natural language and gesture interface for unmanned vehicles. In: AeroSense 2000. International society for optics and photonics, Bellingham, Washington, USA, pp 341–347

  47. Hart S, Staveland L (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock P, Meshkati N (eds) Human mental workload. North Holland Press, Amsterdam

    Google Scholar 

  48. Matthews G, Reinerman-Jones L, Barber D, Abich J IV (2014). The psychometrics of mental workload: multiple measures are sensitive, but divergent. Hum Factors

  49. Abich J IV, Matthews G, Reinerman-Jones L, Barber D (2015) Predicting performance and workload from baseline and concurrent task measures. In: Proceedings of the human factors and ergonomics society annual meeting, vol 59, Sage CA: Los Angeles, CA: SAGE Publications, pp 1676–1680

  50. Pauzié A, Manzano J (2007) Evaluation of driver mental workload facing new in-vehicle information and communication technology. In: Proceedings of the 20th enhanced safety of vehicles conference, Lyon

  51. Pitt I, Edwards A (1997) An improved auditory interface for the exploration of lists. In: Proceedings of the fifth ACM international conference on multimedia, Seattle, pp. 51–61

  52. White T, Kehring K, Glumm M (2009) Effects of unimodal and multimodal cues about threat locations on target acquisition and workload. Mil Psychol 21:497–512

    Article  Google Scholar 

  53. Bolt R (1980) Put-that-there: voice and gesture at the graphics interface. Comput Graph 14(3):262–270

    Article  Google Scholar 

  54. Leventhal L, Barnes J (2008) Usability engineering: process, products, and examples. Pearson Prentice Hall, Upper Saddle River

    Google Scholar 

  55. Lingard L, Reznick R, Espin S, Regehr G, DeVito I (2002) Team communications in the operating room: talk patterns, sites of tension, and implications for novices. Acad Med 77(3):232–237

    Article  Google Scholar 

  56. Lohse M (2010) Investigating the influence of situations and expectations on user behavior: empirical analyses in human–robot interaction [Doctoral dissertation]. Univeristy of Bielefeld, Bielefeld

    Google Scholar 

  57. Robins B, Dautenhahn K, Boekhorst Rt, Billard A (2004) Effects of repeated exposure to a humanoid robot on children with autism. In: Proceedings from the universal access and assistive technology conference, pp. 225–236

  58. Komatsu T, Kurosawa R, Yamada S (2012) How does the difference bewteen user’s expectations and perceptions about a robotic agent affect their behavior? Int J Soc Robot 4(2):109–116

    Article  Google Scholar 

  59. Olson J, Roese N, Zanna M (1996) Expectancies. In: Higgins E, Kruglanski A (eds) Social psychology: handbook of basic principles. Guilford Press, New York, pp 211–238

    Google Scholar 

  60. Norman D (1999) Affordances, conventions and design. Interactions 6(3):38–43

    Article  Google Scholar 

  61. Heckhausen H (1977) Achievement motivation and its constructs: a cognitive model. Motiv Emot 1(4):283–329

    Article  Google Scholar 

  62. Roese NJ, Sherman JW (2007) Expectancy. In: Kruglanski AW, Higgins ET (eds) Social psychology. Handbook of basic principles. Guilford Press, New York, pp 91–115

    Google Scholar 

  63. Kelley H (1950) The warm–cold variable in first impressions of persons. J Personal 18:431–439

    Article  Google Scholar 

  64. Hancock P (2009) Mind, machine, and morality. Ashgate, Chichester

    Google Scholar 

  65. Lin P, Bekey G, Abney K (2008) Autonomous military robotics: risk, ethics, and design. Department of Navy, Office of Naval Research, USA

  66. Zajonc R (1968) Attitudinal effects of mere exposure. J Person Soc Psychol 9(2p2):1–27. doi:10.1037/h0025848

  67. Fechner G (1876) Vorschule der aesthetik. Breitkopf & Härtel, Leipzig

    Google Scholar 

  68. Titchener E (1910) Textbook of psychology. Macmillan, New York

    Google Scholar 

  69. Robins B, Dautenhahn K, te Boekhorst R, Billard A (2004) Effects of repeated exposure to a humanoid robot on children with autism. In: Keates S, Clarkson J, Langdon P, Robinson P (eds) Designing a more inclusive world. Springer, London, pp 225–236

    Chapter  Google Scholar 

  70. Takayama L, Pantofaru C (2009) Influences on proxemic behaviors in human–robot interaction. In: Proceedings of the international conference on intelligent robots and systems (IROS 2009), St. Louis, MO, pp 5495–5502

  71. Bartneck C, Suzuki T, Kanda T, Nomura T (2007) The influence of people’s culture and prior experiences with Aibo on their attitude towards robots. AI Soc 21(1):217–230

    Google Scholar 

  72. Derryberry D, Reed M (2002) Anxiety-related attentional biases and their regulation by attentional control. J Abnorm Psychol 111(2):225–236. doi:10.1037//0021-843X.111.2.225

    Article  Google Scholar 

  73. Ekstrom R, French J, Harman H (1979) Cognitive factors: their identification and replication. Multivar Behav Res Monogr 79(2)

  74. Brooke J (1996) SUS—a quick and dirty usability scale. Usability Eval Ind 189(194):4–7

    Google Scholar 

  75. Lohse M (2011) Bridging the gap between users’ expectations and system evaluations. In: 20th IEEE international symposium on robot and human interactive communication, Altanta, pp 485–490

  76. Abich J IV, Barber D, Reinerman-Jones L (2015) Experimental environments for dismounted human–robot multimodal communications. In: Shumaker R, Lackey S (eds) Virtual, augmented and mixed reality: 7th international conference, VAMR 2015, Held as Part of HCI International 2015, vol 9179. Springer International Publishing, Los Angeles, CA, USA, pp 165–173

  77. Frick-Horbury D (2002) The effects of hand gestures on verbal recall as a function of high- and low-verbal skill levels. Gen Psychol 129(2):137–147

  78. Frick-Horbury D (2002) The use of hand gestures as self-generated cues for recall of verbally associated targets. Am J Psychol 115(1):1–20

    Article  Google Scholar 

  79. Krauss R, Chen Y, Chawla P (1996) Nonverbal behavior and nonverbal communication: what do conversational hand gestures tell us? In: Zanna M (ed) Advances in experimental social psychology. Academic Press, Tampa

    Google Scholar 

  80. Krauss R, Hadar U (2001) The role of speech-related arm/hand gestures in word retrieval. In: Campbell R, Messing L (eds) Gestures, speech, and sign. Oxford University Press, Oxford, pp 93–116

    Google Scholar 

  81. Hall E (1990) The hidden dimension. Anchor Books, New York

    Google Scholar 

  82. Graham J, Argyle M (1975) A cross-cultural study of the communication of exta-verbal meaning by gestures. Int J Psychol 10(1):57–67

    Article  Google Scholar 

  83. Beatty MJ, Friedland MH (1990) Public speaking state anxiety as a function of selected situational and predispositional variables. Commun Educ 39(2):142–147

    Article  Google Scholar 

  84. Kagitcibasi C (2005) Autonomy and relatedness in cultural context. J Cross Cult Psychol 36(4):403–422

    Article  Google Scholar 

Download references

Acknowledgements

This research was sponsored by the U.S. Army Research Laboratory (ARL) and was accomplished under Cooperative Agreement Number W911NF-10-2-0016. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of ARL or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.

Author information

Authors and Affiliations

  1. Institute for Simulation and Training, University of Central Florida, 3100 Technology Parkway, Suite 333, Orlando, FL, 32826, USA

    Julian Abich IV

  2. Institute for Simulation and Training, University of Central Florida, 3100 Technology Parkway, Suite 306-B, Orlando, FL, 32826, USA

    Daniel J. Barber

Authors
  1. Julian Abich IV
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Daniel J. Barber
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Julian Abich IV.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abich, J., Barber, D.J. The impact of human–robot multimodal communication on mental workload, usability preference, and expectations of robot behavior. J Multimodal User Interfaces 11, 211–225 (2017). https://doi.org/10.1007/s12193-016-0237-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-016-0237-4

Keywords