Skip to main content
SpringerLink
Log in

Adapting Usability Metrics for a Socially Assistive, Kinesthetic, Mixed Reality Robot Tutoring Environment

  • Conference paper
  • First Online:
Social Robotics (ICSR 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12483))

Included in the following conference series:

Abstract

The field of Socially Assistive Robot (SAR) tutoring has extensively explored both subjective and objective usability metrics for seated tablet-based human-robot interactions. As SAR tutoring introduces kinesthetic mixed reality environments where students can move around and physically manipulate virtual objects, usability metrics for such interactions need to be re-evaluated. This paper applies standard usability metrics from seated 2D interactions to a kinesthetic mixed reality environment and validates those metrics with post-interaction survey data. Using data from a pilot study (\(n=9\)) conducted with a mixed reality SAR tutor, three commonly-used metrics of usability for seated 2D tutoring interfaces were collected: performance, manipulation time, and gaze. The strength of each usability metric was compared to subjective survey-based scores measured with the System Usability Scale (SUS). The results show that usability scores were correlated with the gaze metric but not with the manipulation time or performance metrics. The findings provide interesting implications for the design and evaluation of kinesthetic mixed reality robot tutoring environments.

K. Mahajan and T. Groechel—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Caleb-Solly, P., Dogramadzi, S., Huijnen, C.A., Heuvel, H.: Exploiting ability for human adaptation to facilitate improved human-robot interaction and acceptance. Inf. Soc. 34(3), 153–165 (2018)

    Article  Google Scholar 

  2. Cho, H., Powell, D., Pichon, A., Kuhns, L.M., Garofalo, R., Schnall, R.: Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation. Int. J. Med. Inform. 129, 366–373 (2019)

    Article  Google Scholar 

  3. Clabaugh, C., Matarić, M.: Escaping Oz: autonomy in socially assistive robotics. Ann. Rev. Control Robot. Auton. Syst. 2, 33–61 (2019)

    Article  Google Scholar 

  4. Clabaugh, C.E., et al.: Long-term personalization of an in-home socially assistive robot for children with autism spectrum disorders. Front. Robot. AI 6, 110 (2019)

    Article  Google Scholar 

  5. Dey, A., Billinghurst, M., Lindeman, R.W., Swan II, J.E.: A systematic review of usability studies in augmented reality between 2005 and 2014. In: 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 49–50. IEEE (2016)

    Google Scholar 

  6. Feingold-Polak, R., Elishay, A., Shahar, Y., Stein, M., Edan, Y., Levy-Tzedek, S.: Differences between young and old users when interacting with a humanoid robot: a qualitative usability study. Paladyn, J. Behav. Robot. 9(1), 183–192 (2018)

    Article  Google Scholar 

  7. Groechel, T., Kuo, C., Dasgupta, R., Wathieu, A.: interaction-lab/movetocode: Doi release (2020). https://doi.org/10.5281/zenodo.3924514

  8. Holden, R.J., et al.: Usability and feasibility of consumer-facing technology to reduce unsafe medication use by older adults. Res. Soc. Adm. Pharm. 16(1), 54–61 (2020)

    Article  Google Scholar 

  9. Ibrahim, R.H., Hussein, D.A.: Assessment of visual, auditory, and kinesthetic learning style among undergraduate nursing students. Int. J. Adv. Nurs. Stud. 5, 1–4 (2016)

    Google Scholar 

  10. Ichinco, M., Harms, K.J., Kelleher, C.: Towards understanding successful novice example user in blocks-based programming. J. Vis. Lang. Sent. Syst. 3, 101–118 (2017)

    Google Scholar 

  11. Kaya, A., Ozturk, R., Altin Gumussoy, C.: Usability measurement of mobile applications with system usability scale (SUS). In: Calisir, F., Cevikcan, E., Camgoz Akdag, H. (eds.) Industrial Engineering in the Big Data Era. LNMIE, pp. 389–400. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-03317-0_32

    Chapter  Google Scholar 

  12. Olde Keizer, R.A.C.M., et al.: Using socially assistive robots for monitoring and preventing frailty among older adults: a study on usability and user experience challenges. Health Technol. 9(4), 595–605 (2019). https://doi.org/10.1007/s12553-019-00320-9

    Article  Google Scholar 

  13. Lee, W.H., Lee, H.K.: The usability attributes and evaluation measurements of mobile media AR (augmented reality). Cogent Arts Hum. 3(1), 1241171 (2016)

    Google Scholar 

  14. Linek, S.B.: Order effects in usability questionnaires. J. Usab. Stud. 12(4), 164–182 (2017)

    Google Scholar 

  15. Malik, N.A., Hanapiah, F.A., Rahman, R.A.A., Yussof, H.: Emergence of socially assistive robotics in rehabilitation for children with cerebral palsy: a review. Int. J. Adv. Rob. Syst. 13(3), 135 (2016)

    Article  Google Scholar 

  16. Menges, R., Tamimi, H., Kumar, C., Walber, T., Schaefer, C., Staab, S.: Enhanced representation of web pages for usability analysis with eye tracking. In: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, pp. 1–9 (2018)

    Google Scholar 

  17. Papadopoulos, I., Lazzarino, R., Miah, S., Weaver, T., Thomas, B., Koulouglioti, C.: A systematic review of the literature regarding socially assistive robots in pre-tertiary education. Comput. Educ. 155, 103924 (2020). https://doi.org/10.1016/j.compedu.2020.103924. http://www.sciencedirect.com/science/article/pii/S0360131520301238

  18. Piech, C., Sahami, M., Huang, J., Guibas, L.: Autonomously generating hints by inferring problem solving policies. In: Proceedings of the Second (2015) ACM Conference on Learning@ Scale, pp. 195–204 (2015)

    Google Scholar 

  19. Pino, M., Boulay, M., Jouen, F., Rigaud, A.S.: “Are we ready for robots that care for us?" Attitudes and opinions of older adults toward socially assistive robots. Front. Aging Neurosci. 7, 141 (2015)

    Article  Google Scholar 

  20. Pranoto, H., Tho, C., Warnars, H.L.H.S., Abdurachman, E., Gaol, F.L., Soewito, B.: Usability testing method in augmented reality application. In: 2017 International Conference on Information Management and Technology (ICIMTech), pp. 181–186. IEEE (2017)

    Google Scholar 

  21. Rodriguez, R.G., Monteoliva, J.M., Pattini, A.E.: A comparative field usability study of two lighting measurement protocols. Int. J. Hum. Factors Ergon. 5(4), 323–343 (2018)

    Article  Google Scholar 

  22. Roscoe, R.D., Allen, L.K., Weston, J.L., Crossley, S.A., McNamara, D.S.: The writing pal intelligent tutoring system: usability testing and development. Comput. Compos. 34, 39–59 (2014)

    Article  Google Scholar 

  23. Shackel, B.: Usability-context, framework, definition, design and evaluation. Interact. Comput. 21(5–6), 339–346 (2009)

    Article  Google Scholar 

  24. Sonderegger, A., Schmutz, S., Sauer, J.: The influence of age in usability testing. Appl. Ergon. 52, 291–300 (2016)

    Article  Google Scholar 

  25. Stein, G., Lédeczi, A.: Mixed reality robotics for stem education. In: 2019 IEEE Blocks and Beyond Workshop (B&B), pp. 49–53 (2019)

    Google Scholar 

  26. Vázquez, C., Xia, L., Aikawa, T., Maes, P.: Words in motion: kinesthetic language learning in virtual reality. In: 2018 IEEE 18th International Conference on advanced learning technologies (ICALT), pp. 272–276. IEEE (2018)

    Google Scholar 

  27. Wang, J., Antonenko, P., Celepkolu, M., Jimenez, Y., Fieldman, E., Fieldman, A.: Exploring relationships between eye tracking and traditional usability testing data. Int. J. Hum.-Comput. Interact. 35(6), 483–494 (2019)

    Article  Google Scholar 

  28. Williams, T., Hirshfield, L., Tran, N., Grant, T., Woodward, N.: Using augmented reality to better study human-robot interaction. In: Chen, J.Y.C., Fragomeni, G. (eds.) HCII 2020. LNCS, vol. 12190, pp. 643–654. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49695-1_43

    Chapter  Google Scholar 

  29. Williams, T., et al.: Virtual, augmented, and mixed reality for human-robot interaction (VAM-HRI). In: Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2020, pp. 663–664. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3371382.3374850

  30. Xefteris, S., Palaigeorgiou, G.: Mixing educational robotics, tangibles and mixed reality environments for the interdisciplinary learning of geography and history (2019)

    Google Scholar 

  31. Yang, F.C.O.: The design of AR-based virtual educational robotics learning system. In: 2019 8th International Congress on Advanced Applied Informatics (IIAI-AAI), pp. 1055–1056. IEEE (2019)

    Google Scholar 

Download references

Acknowledgements

This work was supported by NSF NRI 2.0 grant for “Communicate, Share, Adapt: A Mixed Reality Framework for Facilitating Robot Integration and Customization” (NSF IIS-1925083). We would also like to thank Matthew Rueben for all of his assistance.

Author information

Authors and Affiliations

  1. Viterbi School of Engineering, University of Southern California, Los Angeles, CA, USA

    Kartik Mahajan, Thomas Groechel, Roxanna Pakkar, Julia Cordero, Haemin Lee & Maja J. Matarić

Authors
  1. Kartik Mahajan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Groechel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roxanna Pakkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Julia Cordero
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haemin Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Maja J. Matarić
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kartik Mahajan .

Editor information

Editors and Affiliations

  1. Pennsylvania State University, University Park, PA, USA

    Alan R. Wagner

  2. University of Nevada Reno, Reno, NV, USA

    David Feil-Seifer

  3. University of Denver, Denver, CO, USA

    Kerstin S. Haring

  4. University of Naples Federico II, Naples, Italy

    Silvia Rossi

  5. Colorado School of Mines, Golden, CO, USA

    Thomas Williams

  6. Wichita State University, Wichita, KS, USA

    Hongsheng He

  7. National University of Singapore, Singapore, Singapore

    Shuzhi Sam Ge

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mahajan, K., Groechel, T., Pakkar, R., Cordero, J., Lee, H., Matarić, M.J. (2020). Adapting Usability Metrics for a Socially Assistive, Kinesthetic, Mixed Reality Robot Tutoring Environment. In: Wagner, A.R., et al. Social Robotics. ICSR 2020. Lecture Notes in Computer Science(), vol 12483. Springer, Cham. https://doi.org/10.1007/978-3-030-62056-1_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62056-1_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62055-4

  • Online ISBN: 978-3-030-62056-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation