Abstract
The field of Socially Assistive Robot (SAR) tutoring has extensively explored both subjective and objective usability metrics for seated tablet-based human-robot interactions. As SAR tutoring introduces kinesthetic mixed reality environments where students can move around and physically manipulate virtual objects, usability metrics for such interactions need to be re-evaluated. This paper applies standard usability metrics from seated 2D interactions to a kinesthetic mixed reality environment and validates those metrics with post-interaction survey data. Using data from a pilot study (\(n=9\)) conducted with a mixed reality SAR tutor, three commonly-used metrics of usability for seated 2D tutoring interfaces were collected: performance, manipulation time, and gaze. The strength of each usability metric was compared to subjective survey-based scores measured with the System Usability Scale (SUS). The results show that usability scores were correlated with the gaze metric but not with the manipulation time or performance metrics. The findings provide interesting implications for the design and evaluation of kinesthetic mixed reality robot tutoring environments.
K. Mahajan and T. Groechel—Equal contribution.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Caleb-Solly, P., Dogramadzi, S., Huijnen, C.A., Heuvel, H.: Exploiting ability for human adaptation to facilitate improved human-robot interaction and acceptance. Inf. Soc. 34(3), 153–165 (2018)
Cho, H., Powell, D., Pichon, A., Kuhns, L.M., Garofalo, R., Schnall, R.: Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation. Int. J. Med. Inform. 129, 366–373 (2019)
Clabaugh, C., Matarić, M.: Escaping Oz: autonomy in socially assistive robotics. Ann. Rev. Control Robot. Auton. Syst. 2, 33–61 (2019)
Clabaugh, C.E., et al.: Long-term personalization of an in-home socially assistive robot for children with autism spectrum disorders. Front. Robot. AI 6, 110 (2019)
Dey, A., Billinghurst, M., Lindeman, R.W., Swan II, J.E.: A systematic review of usability studies in augmented reality between 2005 and 2014. In: 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 49–50. IEEE (2016)
Feingold-Polak, R., Elishay, A., Shahar, Y., Stein, M., Edan, Y., Levy-Tzedek, S.: Differences between young and old users when interacting with a humanoid robot: a qualitative usability study. Paladyn, J. Behav. Robot. 9(1), 183–192 (2018)
Groechel, T., Kuo, C., Dasgupta, R., Wathieu, A.: interaction-lab/movetocode: Doi release (2020). https://doi.org/10.5281/zenodo.3924514
Holden, R.J., et al.: Usability and feasibility of consumer-facing technology to reduce unsafe medication use by older adults. Res. Soc. Adm. Pharm. 16(1), 54–61 (2020)
Ibrahim, R.H., Hussein, D.A.: Assessment of visual, auditory, and kinesthetic learning style among undergraduate nursing students. Int. J. Adv. Nurs. Stud. 5, 1–4 (2016)
Ichinco, M., Harms, K.J., Kelleher, C.: Towards understanding successful novice example user in blocks-based programming. J. Vis. Lang. Sent. Syst. 3, 101–118 (2017)
Kaya, A., Ozturk, R., Altin Gumussoy, C.: Usability measurement of mobile applications with system usability scale (SUS). In: Calisir, F., Cevikcan, E., Camgoz Akdag, H. (eds.) Industrial Engineering in the Big Data Era. LNMIE, pp. 389–400. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-03317-0_32
Olde Keizer, R.A.C.M., et al.: Using socially assistive robots for monitoring and preventing frailty among older adults: a study on usability and user experience challenges. Health Technol. 9(4), 595–605 (2019). https://doi.org/10.1007/s12553-019-00320-9
Lee, W.H., Lee, H.K.: The usability attributes and evaluation measurements of mobile media AR (augmented reality). Cogent Arts Hum. 3(1), 1241171 (2016)
Linek, S.B.: Order effects in usability questionnaires. J. Usab. Stud. 12(4), 164–182 (2017)
Malik, N.A., Hanapiah, F.A., Rahman, R.A.A., Yussof, H.: Emergence of socially assistive robotics in rehabilitation for children with cerebral palsy: a review. Int. J. Adv. Rob. Syst. 13(3), 135 (2016)
Menges, R., Tamimi, H., Kumar, C., Walber, T., Schaefer, C., Staab, S.: Enhanced representation of web pages for usability analysis with eye tracking. In: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, pp. 1–9 (2018)
Papadopoulos, I., Lazzarino, R., Miah, S., Weaver, T., Thomas, B., Koulouglioti, C.: A systematic review of the literature regarding socially assistive robots in pre-tertiary education. Comput. Educ. 155, 103924 (2020). https://doi.org/10.1016/j.compedu.2020.103924. http://www.sciencedirect.com/science/article/pii/S0360131520301238
Piech, C., Sahami, M., Huang, J., Guibas, L.: Autonomously generating hints by inferring problem solving policies. In: Proceedings of the Second (2015) ACM Conference on Learning@ Scale, pp. 195–204 (2015)
Pino, M., Boulay, M., Jouen, F., Rigaud, A.S.: “Are we ready for robots that care for us?" Attitudes and opinions of older adults toward socially assistive robots. Front. Aging Neurosci. 7, 141 (2015)
Pranoto, H., Tho, C., Warnars, H.L.H.S., Abdurachman, E., Gaol, F.L., Soewito, B.: Usability testing method in augmented reality application. In: 2017 International Conference on Information Management and Technology (ICIMTech), pp. 181–186. IEEE (2017)
Rodriguez, R.G., Monteoliva, J.M., Pattini, A.E.: A comparative field usability study of two lighting measurement protocols. Int. J. Hum. Factors Ergon. 5(4), 323–343 (2018)
Roscoe, R.D., Allen, L.K., Weston, J.L., Crossley, S.A., McNamara, D.S.: The writing pal intelligent tutoring system: usability testing and development. Comput. Compos. 34, 39–59 (2014)
Shackel, B.: Usability-context, framework, definition, design and evaluation. Interact. Comput. 21(5–6), 339–346 (2009)
Sonderegger, A., Schmutz, S., Sauer, J.: The influence of age in usability testing. Appl. Ergon. 52, 291–300 (2016)
Stein, G., Lédeczi, A.: Mixed reality robotics for stem education. In: 2019 IEEE Blocks and Beyond Workshop (B&B), pp. 49–53 (2019)
Vázquez, C., Xia, L., Aikawa, T., Maes, P.: Words in motion: kinesthetic language learning in virtual reality. In: 2018 IEEE 18th International Conference on advanced learning technologies (ICALT), pp. 272–276. IEEE (2018)
Wang, J., Antonenko, P., Celepkolu, M., Jimenez, Y., Fieldman, E., Fieldman, A.: Exploring relationships between eye tracking and traditional usability testing data. Int. J. Hum.-Comput. Interact. 35(6), 483–494 (2019)
Williams, T., Hirshfield, L., Tran, N., Grant, T., Woodward, N.: Using augmented reality to better study human-robot interaction. In: Chen, J.Y.C., Fragomeni, G. (eds.) HCII 2020. LNCS, vol. 12190, pp. 643–654. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49695-1_43
Williams, T., et al.: Virtual, augmented, and mixed reality for human-robot interaction (VAM-HRI). In: Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2020, pp. 663–664. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3371382.3374850
Xefteris, S., Palaigeorgiou, G.: Mixing educational robotics, tangibles and mixed reality environments for the interdisciplinary learning of geography and history (2019)
Yang, F.C.O.: The design of AR-based virtual educational robotics learning system. In: 2019 8th International Congress on Advanced Applied Informatics (IIAI-AAI), pp. 1055–1056. IEEE (2019)
Acknowledgements
This work was supported by NSF NRI 2.0 grant for “Communicate, Share, Adapt: A Mixed Reality Framework for Facilitating Robot Integration and Customization” (NSF IIS-1925083). We would also like to thank Matthew Rueben for all of his assistance.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Mahajan, K., Groechel, T., Pakkar, R., Cordero, J., Lee, H., Matarić, M.J. (2020). Adapting Usability Metrics for a Socially Assistive, Kinesthetic, Mixed Reality Robot Tutoring Environment. In: Wagner, A.R., et al. Social Robotics. ICSR 2020. Lecture Notes in Computer Science(), vol 12483. Springer, Cham. https://doi.org/10.1007/978-3-030-62056-1_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-62056-1_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62055-4
Online ISBN: 978-3-030-62056-1
eBook Packages: Computer ScienceComputer Science (R0)