Skip to main content
SpringerLink
Log in

Systematic Literature Review on the User Evaluation of Teleoperation Interfaces for Professional Service Robots

  • Conference paper
  • First Online:
HCI in Business, Government and Organizations (HCII 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14039))

Included in the following conference series:

  • 1144 Accesses

Abstract

Remote-controlled robots can perform common tasks, thereby minimizing the exposure to the risks faced by employees in hazardous occupations. The focus of the review in this paper is on the user evaluation of teleoperation interfaces of service robots used in organizational/professional settings. Our study is motivated by the lack of review of user evaluation methods and metrics specific to teleoperation interfaces rather than an evaluation of the robot itself. Grounded in a user-centered approach to design, the three research questions in this review were guided by the three major factors that form the design space for product and service development (Context, Technology, User). The findings of this article are categorized based on areas of application or context of use, technologies used to operate the interfaces, and tools and techniques used to assess user needs as well as user perception of developed solutions. Based on the reported findings, the article provides suggestions for future research that aims at designing interfaces for human-robot collaborations in the workplace.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. ISO 8373:2012(en) Robots and robotic devices — Vocabulary. https://www.iso.org/obp/ui/#iso:std:iso:8373:ed-2:v1:en:term:2.10. Accessed 27 Jan 2023

  2. DeLone, W.H., McLean, E.R.: Measuring e-commerce success: applying the DeLone & McLean information systems success model. International Journal of Electronic Commerce, pp. 31–47 (2004)

    Google Scholar 

  3. Lindgaard, G., Dudek, C.: What is this evasive beast we call user satisfaction? Interacting with Computers, pp. 429–452 (2003)

    Google Scholar 

  4. Menachemi, N., Burkhardt, J., Shewchuk, R., Burke, D., Brooks, R.G.: Hospital information technology and positive financial performance: a different approach to finding an ROI. J. Healthcare Manage. 51(1), 40–59 (2006)

    Google Scholar 

  5. Hurtienne, J., Blessing, L.: Design for intuitive use - testing image schema theory for user interface design. In: DS 42: Proceedings of ICED 2007, the 16th International Conference on Engineering Design, pp. 829–830. Paris (2007)

    Google Scholar 

  6. Djamasbi, S., Strong, D.: User experience-driven innovation in smart and connected worlds. AIS Trans. Human-Comp. Interaction 11(4), 215–231 (2019). https://doi.org/10.17705/1thci.00121

  7. Alrefaei, D., et al.: Using eye tracking to measure user engagement with a shared decision aid. In: 17th edition of Augmented Cognition, 25th International Conference on Human Computer Interaction, (Forthcoming)

    Google Scholar 

  8. Larkin, C., et al.: ReachCare mobile apps for patients experiencing suicidality in the emergency department: development and usability testing using mixed methods. JMIR Formative Res. 7, e41422 (2023). https://formative.jmir.org/2023/1/e41422. https://doi.org/10.2196/41422

  9. Wang, X.V., Wang, L.: A literature survey of the robotic technologies during the COVID-19 pandemic. J. Manufactur. Syst. 60, 823–836 (2021)

    Google Scholar 

  10. Demir, K.A., Döven, G., Sezen, B.: Industry 5.0 and human-robot co-working. Procedia Comput. Sci. 158, 688–695 (2019)

    Google Scholar 

  11. Sugianto, L.-F., Tojib, D.R., Burstein, F.: A practical measure of employee satisfaction with B2E portals. ICIS 2007 Proceedings, AIS, Montreal (2007)

    Google Scholar 

  12. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology: toward a unified view. MIS Quarterly, pp. 425–478 (2003)

    Google Scholar 

  13. Coronado, E., Kiyokawa, T., Ricardez, G.A., Ramirez-Alpizar, I.G., Venture, G., Yamayobe, N.: Evaluating quality in human-robot interaction: a systematic search and classification of performance and human-centered factors, measures and metrics towards an industry 5.0. Journal of Manufacturing Systems, pp. 392–410 (2022)

    Google Scholar 

  14. Ste-Croix, C., Bray-Miners, J., Morton, A.: Human-Robot Interaction Literature Review (2012)

    Google Scholar 

  15. Rodriguez, D., Perez, C., Jagersand, M., Figueroa, P.: A comparison of smartphone interfaces for teleoperation of robot arms. In: 2017 XLIII Latin American Computer Conference (CLEI), pp. 1–8. IEEE, Cordoba (2017)

    Google Scholar 

  16. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 6(7), e1000097 (2009). https://doi.org/10.1371/journal.pmed.1000097

  17. Leitner, J., Luciw, M., Förster, A., Schmidhuber, J.: Teleoperation of a 7 DOF humanoid robot arm using human arm accelerations and EMG signals. In: 12th International Symposium on Artificial Intelligence, vol. 20, Robotics and Automation in Space (i-SAIRAS), Montreal, Canada (2014)

    Google Scholar 

  18. Singh, G., Bermúdez i Badia, S., Ventura, R., Silva, J.L.: Physiologically attentive user interface for robot teleoperation: real time emotional state estimation and interface modification using physiology, facial expressions and eye movements. In: 11th International Joint Conference on Biomedical Engineering Systems and Technologies, pp. 294–302. SCITEPRESS-Science and Technology Publications (2018)

    Google Scholar 

  19. Almeida, L., Menezes, P., Dias, J.: Interface transparency issues in teleoperation. Appl. Sci. 10(18), 6232 (2020)

    Article  Google Scholar 

  20. Adamides, G., et al.: Design and development of a semi-autonomous agricultural vineyard sprayer: human–robot interaction aspects. J. Field Robotics 34(8), 1407–1426 (2017). https://doi.org/10.1002/rob.21721

    Article  Google Scholar 

  21. Szafir, D., Mutlu, B., Fong, T.: Designing planning and control interfaces to support user collaboration with flying robots. Int. J. Robotics Res. 36(5–7), 514–542 (2017)

    Article  Google Scholar 

  22. Michaud, F., et al.: Exploratory design and evaluation of a homecare teleassistive mobile robotic system. Mechatronics 20(7), 751–766 (2010)

    Google Scholar 

  23. Nejatimoharrami, F., Faina, A., Jovanovic, A., St-Cyr, O., Chignell, M., Stoy, K.: UI Design for an engineering process: programming experiments on a liquid handling robot. In: 2017 First IEEE International Conference on Robotic Computing (IRC), pp. 196–203. IEEE (2017)

    Google Scholar 

  24. Latif, H.O., Sherkat, N., Lotfi, A.: TeleGaze: Teleoperation through eye gaze. In: 2008 7th IEEE International Conference on Cybernetic Intelligent Systems, pp. 1–6. IEEE (2008)

    Google Scholar 

  25. Awde, A., Boudaoud, M., Macioce, M., Régnier, S., Clévy, C.: A microrobotic approach for the intuitive assembly of industrial electrooptical sensors based on closed-loop light feeling. IEEE/ASME Trans. Mechatron. 27(6), 5462–5471 (2022)

    Article  Google Scholar 

  26. Lima, A.T., Rocha, F.A.S., Torre, M.P., Azpúrua, H., Freitas, G.M.: Teleoperation of an ABB IRB 120 robotic manipulator and BarrettHand BH8–282 using a Geomagic Touch X haptic device and ROS. In 2018 Latin American Robotic Symposium. In: 2018 Brazilian Symposium on Robotics (SBR) and 2018 Workshop on Robotics in Education (WRE), pp. 188–193. IEEE (2018)

    Google Scholar 

  27. Fidêncio, A., et al.: Metodologia para Avaliac¸ ˜ao de Interfaces de Teleoperac¸ ˜ao, XIII Simp´osio Brasileiro de Automac¸ ˜ao Inteligente (2017)

    Google Scholar 

  28. Witmer, B.G., Singer, M.J., Measuring presence in virtual environments: a presence questionnaire. Presence Teleoperators virtual Environ. 7 (3), 225e240 (1998)

    Google Scholar 

  29. Mavridis, N., Giakoumidis, N., Machado, E.L.: A novel evaluation framework for teleoperation and a case study on natural human-arm-imitation through motion capture. Int. J. Soc. Robot. 4, 5–18 (1998)

    Article  Google Scholar 

  30. Ainasoja, A.E., Pertuz, S., Kämäräinen, J.K.: Smartphone Teleoperation for Self-Balancing Telepresence Robots (2019)

    Google Scholar 

  31. Antuvan, C.W., Ison, M., Artemiadis, P.: Embedded human control of robots using myoelectric interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 22(4), 820–827 (2014)

    Article  Google Scholar 

  32. Pryor, W., et al.: Experimental evaluation of teleoperation interfaces for cutting of satellite insulation. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 4775–4781. IEEE (2019)

    Google Scholar 

  33. Macchini, M., Havy, T., Weber, A., Schiano, F., Floreano, D.: Hand-worn haptic interface for drone teleoperation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 10212–10218. IEEE (2020)

    Google Scholar 

  34. Schmidt, L., Hegenberg, J., Cramar, L.: User studies on teleoperation of robots for plant inspection. Industrial Robot: An Int. J. 41(1), 6–14 (2014)

    Article  Google Scholar 

  35. Adamides, G., et al.: HRI usability evaluation of interaction modes for a teleoperated agricultural robotic sprayer. Appl. Ergon. 62, 237–246 (2017). https://doi.org/10.1016/j.apergo.2017.03.008

    Article  Google Scholar 

  36. Schwarzer, R., Jerusalem, M.: Self-efficacy measurement: Generalized self-efficacy scale (GSES). In: Weinman, J., Wright, S., Johnston, M. (Eds.), Measures in health psychology: A user’s portfolio, pp. 35–37. NFER-Nelson, Windsor, England (1995)

    Google Scholar 

  37. Hegarty, M., Richardson, A.E., Montello, D.R., Lovelace, K., Subbiah, I.: Development of a self-report measure of environmental spatial ability. Intelligence 30(5), 425447 (2002)

    Google Scholar 

  38. Vorderer, P., et al.: Development of the MEC spatial presence questionnaire (MEC SPQ). unpublished report to the European Community on Project Presence: MEC (IST-200137661), Hannover, Munich, Helsinki, Porto, Zurich (2004)

    Google Scholar 

  39. Wang, Z., Giannopoulos, E., Slater, M., Peer, A.: Handshake: realistic human-robot interaction in haptic enhanced virtual reality. Presence 20(4), 371–392 (2011)

    Article  Google Scholar 

  40. Orlosky, J., Theofilis, K., Kiyokawa, K., Nagai, Y.: Effects of throughput delay on perception of robot teleoperation and head control precision in remote monitoring tasks. PRESENCE: Virtual and Augmented Reality 27(2), 226–241 (2018)

    Google Scholar 

  41. Nenna, F., Orso, V., Zanardi, D., Gamberini, L.: The virtualization of human–robot interactions: a user-centric workload assessment. Virtual Reality, pp. 1–19 (2022)

    Google Scholar 

  42. Slater, M., Usoh, M., Steed, A.: Depth of presence in virtual environments. Presence-Teleoper. Virtual Environ. 3, 130–144 (1994)

    Article  Google Scholar 

  43. Usoh, M., Catena, E., Arman, S., Slater, M.: Using presence questionnaires in reality. Presence Teleoper. Virtual Environ. 9, 497–503 (2000)

    Article  Google Scholar 

  44. Reason, J.T., Brand, J.J.: Motion Sickness. Academic press (1975)

    Google Scholar 

  45. Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)

    Article  Google Scholar 

  46. Blitch, J.G.: A neurophysiological examination of multi-robot control during NASA’s extreme environment mission operations project. In: Advances in Human Factors in Robots and Unmanned Systems: Proceedings of the AHFE 2016 International Conference on Human Factors in Robots and Unmanned Systems, pp. 341–351. Walt Disney World®, Springer International Publishing, Florida, USA (2017). https://doi.org/10.1007/978-3-319-41959-6_28

  47. Adams, S., Kane, R., Bates, R.: Validation of the China lake situational awareness scale with 3D SART and S-CAT. Naval Air Warfare Center Weapons Division (452330D), China Lake, CA (1998)

    Google Scholar 

  48. Chicaiza, F.A., Slawiñski, E., Salinas, L.R., Mut, V.A.: Evaluation of path planning with force feedback for bilateral teleoperation of unmanned rotorcraft systems. J. Intell. Robotic Syst. 105(2), (2022). https://doi.org/10.1007/s10846-022-01651-y

  49. Gatsoulis, Y., Virk, G.S., Dehghani-Sanij, A.A.: On the measurement of situation awareness for effective human-robot interaction in teleoperated systems. J. Cognitive Eng. Decision Making 4(1), 69–98 (2010)

    Article  Google Scholar 

  50. Brooke, J.: SUS-A quick and dirty usability scale. Usability Eva. Ind. 189(194), 4–7 (1996)

    Google Scholar 

  51. Bangor, A., Kortum, P.T., Miller, J.T.,: An empirical evaluation of the system usability scale. Intl. J. HumaneComputer Interact. 24(6), 574e594 (2008)

    Google Scholar 

  52. Valero-Gomez, A., de la Puente, P.: Usability evaluation of a pda interface for exploration mobile robots. IFAC Proc. 44(1), 1120–1125 (2011)

    Article  Google Scholar 

  53. Drury, J.L., Keyes, B., Yanco, H.A.: Lassoing hri: analyzing situation awareness in map-centric and videocentric interfaces. In: Proceedings of the Second ACM SIGCHI/SIGART Conference on Human-Robot Interaction, pp. 279–286 (2007)

    Google Scholar 

  54. Lin, T.C., Krishnan, A.U., Li, Z.: Intuitive, efficient and ergonomic tele-nursing robot interfaces: Design evaluation and evolution. ACM Trans. Human-Robot Interaction (THRI) 11(3), 1–41 (2022)

    Article  Google Scholar 

  55. Jacinto-Villegas, J.M., et al.: A novel wearable haptic controller for teleoperating robotic platforms. IEEE Robotics Auto. Lett. 2(4), 2072–2079 (2017), Article 7962162. https://doi.org/10.1109/LRA.2017.2720850

  56. Lewis, J.R.: IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Technical Report. IBM—Human Factors Group, Boca Raton, FL, USA (1993)

    Google Scholar 

  57. Cruz-Ramirez, S.R., Ishizuka, Y., Mae, Y., Takubo, T., Arai, T.: Dismantling interior facilities in buildings by human robot collaboration. In: 2008 IEEE International Conference on Robotics and Automation, pp. 2583–2590. IEEE (2008)

    Google Scholar 

  58. Chauhan, M., Deshpande, N., Caldwell, D.G., Mattos, L.S.: Design and modeling of a three-degree-of-freedom articulating robotic microsurgical forceps for trans-oral laser microsurgery. J. Med. Dev. 13(2) (2019)

    Google Scholar 

  59. Sanguino, T.M., Márquez, J.A., Carlson, T., Millán, J.D.: Improving skills and perception in robot navigation by an augmented virtuality assistance system. J. Intell. Robotic Syst. 76, 255–266 (2014)

    Google Scholar 

  60. Alonso, R., Bonini, A., Reforgiato Recupero, D., Spano, L.D.: Exploiting virtual reality and the robot operating system to remote-control a humanoid robot. Multimedia Tools Appl. 81(11), 15565-15592 (2022)

    Google Scholar 

  61. Nakayama, A., Ruelas, D., Savage, J., Bribiesca, E.: Teleoperated service robot with an immersive mixed reality interface. Инфopмaтикa и aвтoмaтизaция 20(6), 1187–1223 (2021)

    Google Scholar 

  62. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Advances in Psychology, 52, pp. 139–183. North-Holland (1988)

    Google Scholar 

  63. Kent, D., Saldanha, C., Chernova, S.: Leveraging depth data in remote robot teleoperation interfaces for general object manipulation. Int. J. Robotics Res. 39(1), 39–53 (2022)

    Article  Google Scholar 

  64. Glas, D.F., Kanda, T., Ishiguro, H., Hagita, N.: Field trial for simultaneous teleoperation of mobile social robots. In: Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, pp. 149–156 (2009)

    Google Scholar 

  65. Doisy, G., Ronen, A., Edan, Y.: Comparison of three different techniques for camera and motion control of a teleoperated robot. Appl. Ergon. 58, 527–534 (2017)

    Article  Google Scholar 

  66. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. Paper presented at the Proceedings of the human factors and ergonomics society annual meeting (2006)

    Google Scholar 

  67. Okishiba, S., et al.: Tablet interface for direct vision teleoperation of an excavator for urban construction work. Automation Const. 102, 17–26 (2019)

    Google Scholar 

  68. Gholami, S., Garate, V.R., De Momi, E., Ajoudani, A.: A probabilistic shared-control framework for mobile robots. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11473–11480. IEEE (2020)

    Google Scholar 

  69. Kikuchi, T., Takano, T., Yamaguchi, A., Ikeda, A., Abe, I.: Haptic interface with twin-driven mr fluid actuator for teleoperation endoscopic surgery system. Actuators, 10(10), 245 (2021). https://doi.org/10.3390/act10100245

  70. Bhat, R., Pandey, V., Rao, A.K., Chandra, S.: An evaluation of cognitive and neural correlates for indirect vision driving and rover teleoperation. In: 2017 2nd International Conference on Man and Machine Interfacing (MAMI), pp. 1–5. IEEE (2017)

    Google Scholar 

  71. Randelli, G., Venanzi, M., Nardi, D.: Evaluating tangible paradigms for ground robot teleoperation. In: 2011 RO-MAN, pp. 389–394. IEEE (2011)

    Google Scholar 

  72. Quintero, C.P., Dehghan, M., Ramirez, O., Ang, M.H., Jagersand, M.: Flexible virtual fixture interface for path specification in tele-manipulation. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 5363–5368. IEEE (2017)

    Google Scholar 

  73. Jevtić, A., Colomé, A., Alenyà, G., Torras, C.: User evaluation of an interactive learning framework for single-arm and dual-arm robots. In: Agah, A., Cabibihan, J.-J., Howard, A.M., Salichs, M.A., He, H. (eds.) ICSR 2016. LNCS (LNAI), vol. 9979, pp. 52–61. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47437-3_6

    Chapter  Google Scholar 

  74. Siddhartha, B., Chavan, A.P., Uma, B.V.: An electronic smart jacket for the navigation of visually impaired society. Materials Today: Proc. 5(4), 10665–10669 (2018)

    Google Scholar 

  75. Shojaeizadeh, M., Djamasbi, S., Paffenroth, R.C., Trapp, A.C.: Detecting task demand via an eye tracking machine learning system. Decis. Support Syst. 116, 91–101 (2019)

    Article  Google Scholar 

  76. Alrefaei, D., et al..: Impact of anxiety on information processing among young adults: an exploratory eye-tracking study. In: Proceedings of the 56th Hawaii International Conference on System Sciences, pp. 6321–6330. Hawaii (2023)

    Google Scholar 

  77. Norouzi Nia, J., Varzgani, F., Djamasbi, S., Tulu, B., Lee, C., Muehlschlegel, S.: Visual hierarchy and communication effectiveness in medical decision tools for surrogate-decision-makers of critically ill traumatic brain injury patients. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) HCII 2021. LNCS (LNAI), vol. 12776, pp. 210–220. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78114-9_15

    Chapter  Google Scholar 

  78. Jain, P., Djamasbi, S., Wyatt, J.: Creating value with proto-research persona development. In: HCI in Business, Government and Organizations. Information Systems and Analytics: 6th International Conference, HCIBGO 2019. Held as Part of the 21st HCI International Conference, HCII 2019. Proceedings, Part II 21, pp. 72–82. Springer International Publishing, Orlando, FL, USA (2019). https://doi.org/10.1007/978-3-030-22338-0_6

  79. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 318–340 (1989)

    Article  Google Scholar 

  80. Igbaria, M., Tan, M.: The consequences of information technology acceptance on subsequent individual performance. Inf. Manage. 32(3), 113–121 (1997)

    Article  Google Scholar 

  81. UserZoom: The State of UX 2022. UserZoom (2022)

    Google Scholar 

  82. Forrester Research Inc.: The Total Economic Impact of IBM’s Design Thinking Practice. IBM (2018)

    Google Scholar 

Download references

Acknowledgement

The research was sponsored by the DEVCOM Analysis Center and was accomplished under Cooperative Agreement Number W911NF-22–2-0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

Author information

Authors and Affiliations

  1. Worcester Polytechnic Institute, Worcester, MA, 01609, USA

    Gaayathri Sankar, Soussan Djamasbi, Zhi Li & Jing Xiao

  2. Army DEVCOM Analysis Center (DAC), Aberdeen Proving Ground, MD, 21005, USA

    Norbou Buchler

Authors
  1. Gaayathri Sankar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Soussan Djamasbi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Norbou Buchler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gaayathri Sankar .

Editor information

Editors and Affiliations

  1. City University of Hong Kong, Kowloon, Hong Kong

    Fiona Nah

  2. City University of Hong Kong, Kowloon, Hong Kong

    Keng Siau

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sankar, G., Djamasbi, S., Li, Z., Xiao, J., Buchler, N. (2023). Systematic Literature Review on the User Evaluation of Teleoperation Interfaces for Professional Service Robots. In: Nah, F., Siau, K. (eds) HCI in Business, Government and Organizations. HCII 2023. Lecture Notes in Computer Science, vol 14039. Springer, Cham. https://doi.org/10.1007/978-3-031-36049-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36049-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36048-0

  • Online ISBN: 978-3-031-36049-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation