Skip to main content
Springer Nature Link
Log in

Assessment of a Novel Virtual Environment for Examining Cognitive-Motor Processes During Execution of Action Sequences in a Human-Robot Teaming Context

  • Conference paper
  • First Online:
Augmented Cognition (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14694))

Included in the following conference series:

  • 390 Accesses

Abstract

With the development of advanced AI and robotic systems, there is a growing interest in examining human-robot teaming. While the vast majority of human-robot studies has focused on technological developments, only a limited body of work has considered employing neurophysiological data and real-world activities to examine human cognitive-motor processes in such a teaming context. Although human-robot teaming can be examined using physical systems, virtual environments also offer numerous advantages such as versatility, scalability, and cost-effectiveness. Therefore, here we propose and assess a novel virtual environment (VTEAM) through which human cognitive-motor processes can be examined when individuals perform alone or with a robotic teammate, sequential tasks that have similar features to real-world activities. This new experimental platform allows synchronous behavioral and neurophysiological (EEG) data collection to provide a more comprehensive examination of human cognitive-motor behavior. VTEAM was evaluated by assessing its usability, as well as the resulting team performance and human perception of the workload and of the robotic teammate. The findings revealed appropriate levels of usability and workload when individuals operated VTEAM to complete two tasks alone or with the robotic teammate. When engaged, the robotic teammate - which individuals perceived as likeable, intelligent and safe - was able to improve task performance, suggesting that this platform can robustly assess human-robot teaming. Thus, this novel experimental platform appears to be appropriate for investigating human cognitive-motor processes when individuals perform and learn action sequences alone or collaboratively with a robotic teammate.

J. Jayashankar, A. L. Packy and A. Teymourlouei are co-first authors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Gevins, A., Smith, M.E., McEvoy, L., Yu, D.: High-resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice. Cereb. Cortex 7(4), 374–385 (1997)

    Article  Google Scholar 

  2. Gentili, R.J., et al.: Combined assessment of attentional reserve and cognitive effort under various levels of challenge with a dry EEG system. Psychophys 55(6), e13059 (2018)

    Article  Google Scholar 

  3. Miller, M.W., et al.: The effects of team environment on cerebral cortical processes and attentional reserve. Sport Exerc. Perform. Psychol. 3(1), 61–74 (2014)

    Google Scholar 

  4. Mehta, R.K., Parasuraman, R.: Neuroergonomics: a review of applications to physical and cognitive work. Front. Hum. Neurosci. 7, 889 (2013)

    Article  Google Scholar 

  5. Dehais, F., Lafont, A., Roy, R., Fairclough, S.: A neuroergonomics approach to mental workload, engagement and human performance. Front. Neurosci. 14, 268 (2020)

    Article  Google Scholar 

  6. Ahmad, M.I., Bernotat, J., Lohan, K., Eyssel, F. Trust and cognitive load during human-robot interaction, arXiv:1909.05160 (2019)

  7. Ehrlich, S.K., Cheng, G.: A feasibility study for validating robot actions using EEG-based error-related potentials. Int J of Soc Robotics 11, 271–283 (2019)

    Article  Google Scholar 

  8. de Visser, E.J., Beatty, P.J., Estepp, J.R., Kohn, S., Abubshait, A., Fedota, J.R., McDonald, C.G. Learning from the slips of others: neural correlates of trust in automated Agents. Front Hum Neurosci, 12, (2018)

    Google Scholar 

  9. Henschel, A., Hortensius, R., Cross, E.S.: Social cognition in the age of human-robot interaction. Trends Neurosci. 43(6), 373–384 (2020)

    Article  Google Scholar 

  10. Hauge, T.C., et al.: A novel application of Levenshtein distance for assessment of high-level motor planning underlying performance during learning of complex motor sequences. J. Mot. Learn. Dev. 8(1), 67–86 (2019)

    Article  Google Scholar 

  11. Hauge, T.C., Katz, G.E., Davis, G.P., Huang, D.W., Reggia, J.A., Gentili, R.J.: High-level motor planning assessment during performance of complex action sequences in humans and a humanoid robot. Int. J. Soc. Robot. 13, 981–998 (2021)

    Article  Google Scholar 

  12. Katz, G., Huang, D.W., Hauge, T., Gentili, R., Reggia, J. A novel parsimonious cause-effect reasoning algorithm for robot imitation and plan recognition. IEEE Trans Cognit Dev Syst PP(99):1–17 (2017)

    Google Scholar 

  13. Welsh, M.C., Huizinga, M.: Tower of Hanoi disk-transfer task: influences of strategy knowledge and learning on performance. Learn. Individ. Differ. 15(4), 283–298 (2005)

    Article  Google Scholar 

  14. Wulf, G., Shea, C.H.: Principles derived from the study of simple skills do not generalize to complex skill learning. Psychon. Bull. Rev. 9(2), 185–211 (2002)

    Article  Google Scholar 

  15. Wong, M., et al.: A remote synthetic testbed for human-robot teaming: an iterative design process. Proc Hum Factor Ergonom Soc Ann Meet 65(1), 781–785 (2021)

    Article  Google Scholar 

  16. Raimondo, F.R., et al.: Trailblazing roblox virtual synthetic testbed development for human-robot teaming studies. Proc Hum Factor Ergonom Soc Ann Meet 66(1), 812–816 (2022)

    Article  Google Scholar 

  17. El Makrini, I., Merckaert, K., Lefeber, D. Vanderborght, B. Design of a collaborative architecture for human-robot assembly tasks. Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1624–1629 (2017)

    Google Scholar 

  18. Natarajan, M., et al.: Human-robot teaming: grand challenges. Curr Robot Rep 4, 81–100 (2023)

    Article  Google Scholar 

  19. Cian, L., Dreossi, T., Dovier, A. Modeling and solving the rush hour puzzle. In: 37th Italian Conference on Computational Logic, June 29 - July 1, 2022, Bologna, Italy (2022)

    Google Scholar 

  20. Huang, D.W., Katz, G.E., Langsfeld, J.D., Gentili, R.J., Reggia, J.A. A virtual demonstrator environment for robot imitation learning. In: IEEE international conference on technologies for practical robot applications (TePRA), Woburn, MA, USA, pp 1–6 (2015)

    Google Scholar 

  21. Huang, D.W., Katz, G.E., Langsfeld, J.D., Oh, H., Gentili, R.J., Reggia J.A. An object-centric paradigm for robot programming by demonstration. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) Foundations of Augmented Cognition. AC 2015. LNCS, vol 9183. Springer, Cham (2015)

    Google Scholar 

  22. Shaver, A.A., Peri, N., Mezebish, R., Matthew, G., Berson, A., Gaskins, C., Davis, G.P., Katz, G.E., Samuel, I., Reinhard, M.J., Costanzo, M.E., Reggia, J.A., Purtilo, J., Gentili, R.J. Assessment of a novel virtual environment for examining human cognitive-motor performance during execution of action sequences. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Augmented Cognition. HCII 2022. Lecture Notes in Computer Science, vol 13310. Springer, Cham (2022)

    Google Scholar 

  23. Flake, G.W., Baum, E.B.: Rush hour is pspace-complete, or “why you should generously tip parking lot attendants.” Theoret. Comput. Sci. 270, 895–911 (2002)

    Article  MathSciNet  Google Scholar 

  24. Pereira, A.G., Ritt, M., Buriol, L.S.: Pull and pushpull are PSPACE-complete. Theoret. Comput. Sci. 628, 50–61 (2016)

    Article  MathSciNet  Google Scholar 

  25. Lewis, J.R.: Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction 14(3–4), 463–488 (2002)

    Article  Google Scholar 

  26. Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Intl. Int J Hum-Comput Int 24(6), 574–594 (2008)

    Google Scholar 

  27. Sauro, J. Dumas, J. S. Comparison of three one-question, post-task usability questionnaires. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘09). Association for Computing Machinery, New York, NY, USA, 1599–1608 (2009)

    Google Scholar 

  28. Kortum, P.T., Bangor, A.: Usability ratings for everyday products measured with the system usability scale. Int J Hum-Comput Int 29(2), 67–76 (2013)

    Google Scholar 

  29. Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Usability Stud. 4(3), 114–123 (2009)

    Google Scholar 

  30. Sauro, J., Lewis, J.R.: Quantifying the user experience: practical statistics for user research, 2nd edn. Morgan Kaufmann, Cambridge (2016)

    Google Scholar 

  31. Barnum, C.N.: Usability testing essentials, 2nd edn. Morgan Kaufmann, Cambridge (2021)

    Google Scholar 

  32. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In proceedings of the human factors and ergonomics society annual meeting 50(9), 904–908 (2006)

    Article  Google Scholar 

  33. Shuggi, I.M., Oh, H., Shewokis, P.A., Gentili, R.J.: Mental workload and motor performance dynamics during practice of reaching movements under various levels of task difficulty. Neuroscience 360, 166–179 (2017)

    Article  Google Scholar 

  34. Grier, R.A.: How high is high? A meta-analysis of NASA-TLX global workload scores. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 59(1), 1727–1731 (2015)

    Article  Google Scholar 

  35. Bartneck, C., Kulić, D., Croft, E., Zoghbi, S.: Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J of Soc Robotics 1, 71–81 (2009)

    Article  Google Scholar 

  36. Kerzel, M., Strahl, E., Magg, S., Navarro-Guerrero, N., Heinrich S., Wermter, S. NICO — Neuro-inspired companion: A developmental humanoid robot platform for multimodal interaction. Proc. 26th IEEE International Symposium on Robot and Human Interactive Communication (ROMAN), pp. 113–120 (2017)

    Google Scholar 

  37. Maroto-Gómez, M., Castro-González, Á., Malfaz, M., Salichs, M.Á.: A biologically inspired decision-making system for the autonomous adaptive behavior of social robots. Complex Intell. Syst. 9, 6661–6679 (2023)

    Article  Google Scholar 

Download references

Acknowledgment

Work on VLEARN and VTEAM was supported in part by funding from ONR to RG, JR, and GK (N000141912044) and to JP (N000142112821).

Author information

Authors and Affiliations

  1. Department of Kinesiology, University of Maryland, College Park, MD, USA

    Jayesh Jayashankar, Anna L. Packy, Alexandra A. Shaver & Rodolphe J. Gentili

  2. Neuroscience and Cognitive Science Program, University of Maryland, College Park, MD, USA

    Jayesh Jayashankar, Anna L. Packy, James A. Reggia, James Purtilo & Rodolphe J. Gentili

  3. Department of Computer Science, University of Maryland, College Park, MD, USA

    Arya Teymourlouei

  4. Electrical Engineering and Computer Science, Syracuse University, Syracuse, NY, USA

    Garrett E. Katz

  5. Institute for Advanced Computer Studies, University of Maryland, College Park, MD, USA

    James A. Reggia

  6. Maryland Robotics Center, University of Maryland, College Park, MD, USA

    James A. Reggia & Rodolphe J. Gentili

Authors
  1. Jayesh Jayashankar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna L. Packy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arya Teymourlouei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexandra A. Shaver
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Garrett E. Katz
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. James A. Reggia
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. James Purtilo
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rodolphe J. Gentili
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to James Purtilo or Rodolphe J. Gentili .

Editor information

Editors and Affiliations

  1. Soar Technology Inc., Orlando, FL, USA

    Dylan D. Schmorrow

  2. Katmai Government Services, Orlando, FL, USA

    Cali M. Fidopiastis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jayashankar, J. et al. (2024). Assessment of a Novel Virtual Environment for Examining Cognitive-Motor Processes During Execution of Action Sequences in a Human-Robot Teaming Context. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Augmented Cognition. HCII 2024. Lecture Notes in Computer Science(), vol 14694. Springer, Cham. https://doi.org/10.1007/978-3-031-61569-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-61569-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-61568-9

  • Online ISBN: 978-3-031-61569-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics