Skip to main content
SpringerLink
Log in

Review of Human Studies Methods in HRI and Recommendations

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

This article provides an overview on planning, designing, and executing human studies for Human-Robot Interaction (HRI) that leads to ten recommendations for experimental design and study execution. Two improvements are described, using insights from the psychology and social science disciplines. First is to use large sample sizes to better represent the populations being investigated to have a higher probability of obtaining statistically significant results. Second is the application of three or more methods of evaluation to have reliable and accurate results, and convergent validity. Five primary methods of evaluation exist: self-assessments, behavioral observations, psychophysiological measures, interviews, and task performance metrics. The article describes specific tools and procedures for operationalizing these improvements, as well as suggestions for recruiting participants. A recent large-scale, complex, controlled human study in HRI using 128 participants and four methods of evaluation is presented to illustrate planning, design, and execution choices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bartneck C, Kulic D, Croft E, Zoghbi S (2008) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot 2009(1):71–81

    Google Scholar 

  2. Bethel CL (2009) Robots without faces: non-verbal social human-robot interaction. Dissertation, University of South Florida

  3. Bethel CL, Bringes C, Murphy RR (2009) Non-facial and non-verbal affective expression in appearance-constrained robots for use in victim management: robots to the rescue! In: 4th ACM/IEEE international conference on human-robot interaction (HRI2009), San Diego. ACM, New York

    Google Scholar 

  4. Bethel CL, Salomon K, Burke JL, Murphy RR (2007) Psychophysiological experimental design for use in human-robot interaction studies. In: The 2007 international symposium on collaborative technologies and systems (CTS 2007). IEEE, Orlando

    Google Scholar 

  5. Bethel CL, Salomon K, Murphy RR (2009) Preliminary results: Humans find emotive non-anthropomorphic robots more calming. In: 4th ACM/IEEE international conference on human-robot interaction (HRI2009), San Diego, CA

  6. Bethel CL, Salomon K, Murphy RR, Burke JL (2007) Survey of psychophysiology measurements applied to human-robot interaction. In: 16th IEEE international symposium on robot and human interactive communication, Jeju Island, South Korea. IEEE, New York

    Google Scholar 

  7. Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25:49–59

    Article  Google Scholar 

  8. Burke JL, Murphy RR, Riddle DR, Fincannon T (2004) Task performance metrics in human-robot interaction: taking a systems approach. In: Performance metrics for intelligent systems, Gaithersburg, MD

  9. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Earlbaum, Hillsdale

    MATH  Google Scholar 

  10. Dautenhahn K, Walters M, Woods S, Koay KL, Nehaniv CL, Sisbot A, Alami R, Siméon T (2006) How may i serve you? A robot companion approaching a seated person in a helping context. In: 1st ACM SIGCHI/SIGART conference on human-robot interaction (HRI2006). ACM Press, New York, pp 172–179

    Chapter  Google Scholar 

  11. Elara MR, Wijesoma S, Acosta Calderon CA, Zhou C (2009) Experimenting false alarm demand for human robot interactions in humanoid soccer robots. Int J Soc Robot 2009(1):171–180

    Article  Google Scholar 

  12. Elmes DG, Kantowitz BH, Roediger HL III (2006) Research methods in psychology, 8th edn. Thomson-Wadsworth, Belmont

    Google Scholar 

  13. Faul F, Erdfelder E, Lang AG, Buchner A (2007) G*power 3: A flexible statistical power analysis program for social, behavioral, and biomedical sciences. Behav Res Meth 39(2):175–191

    Google Scholar 

  14. Goodwin CJ (2003) Research in psychology-methods and design. Wiley, Hoboken

    Google Scholar 

  15. Itoh K, Miwa H, Nukariya Y, Zecca M, Takanobu H, Roccella S, Carrozza MC, Dario P, Atsuo T (2006) Development of a bioinstrumentation system in the interaction between a human and a robot. In: International conference of intelligent robots and systems, Beijing, China, pp. 2620–2625

  16. Johnson B, Christensen L (2004) Educational research quantitative, qualitative, and mixed approaches, 2nd edn. Pearson Education, Boston

    Google Scholar 

  17. Kidd CD, Breazeal C (2005) Human-robot interaction experiments: Lessons learned. In: Proceeding of AISB’05 symposium robot companions: hard problems and open challenges in robot-human interaction, Hatfield, Hertfordshire, pp. 141–142

    Google Scholar 

  18. Kulić D, Croft E (2006) Physiological and subjective responses to articulated robot motion. Robotica 15(1) 13–27. doi:10.1017/S0263574706002955

    Google Scholar 

  19. Lazar J, Feng JH, Hochheiser H (2010) Research methods in human-computer interaction. Wiley, New York

    Google Scholar 

  20. Liu C, Rani P, Sarkar N (2006) Affective state recognition and adaptation in human-robot interaction: a design approach. In: International conference on intelligent robots and systems (IROS 2006), Beijing, China, pp. 3099–3106

  21. Moshkina L, Arkin RC (2005) Human perspective on affective robotic behavior: a longitudinal study. In: IEEE/RSJ international conference on intelligent robots and systems (IROS 2005), pp. 2443–2450

  22. Murphy RR, Riddle D, Rasmussen E (2004) Robot-assisted medical reachback: a survey of how medical personnel expect to interact with rescue robots. In: 13th IEEE international workshop on robot and human interactive communication (RO-MAN 2004), pp. 301–306

  23. Mutlu B, Hodgins JK, Forlizzi J (2006) A storytelling robot: Modeling and evaluation of human-like gaze behavior. In: 2006 IEEE-RAS international conference on humanoid robots (HUMANOIDS’06), IEEE, Genova, Italy

  24. Mutlu B, Osman S, Forlizzi J, Hodgins JK, Kiesler S (2006) Task structure and user attributes as elements of human-robot interaction design. In: 15th IEEE international workshop on robot and human interactive communication (RO-MAN 2006). IEEE, University of Hertfordshire, Hatfield

  25. Olsen DR, Goodrich MA (2003) Metrics for evaluating human-robot interactions. In: Performance metrics for intelligent systems workshop

  26. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intel 23(10):1175–1191

    Article  Google Scholar 

  27. Preece J, Rogers Y, Sharp H (2007) Interaction design-beyond human-computer interaction, 2nd edn. Wiley, New York

    Google Scholar 

  28. Rani P, Sarkar N, Smith CA, Kirby LD (2004) Anxiety detecting robotic system—towards implicit human-robot collaboration. Robotica 22(1):85–95

    Article  Google Scholar 

  29. Riddle DR, Murphy RR, Burke JL (2005) Robot-assisted medical reachback: using shared visual information. In: IEEE international workshop on robot and human interactive communication (ROMAN 2005), IEEE, Nashville, TN, pp. 635–642

  30. Schweigert WA (1994) Research methods & statistics for psychology. Brooks/Cole Publishing Company, Pacific Grove, CA

    Google Scholar 

  31. Shaughnessy JJ, Zechmeister EB (1994) Research methods in psychology. McGraw-Hill, New York

    Google Scholar 

  32. Steinfeld A, Fong T, Kaber D, Lewis M, Scholtz J, Schultz A, Goodrich M (2006) Common metrics for human-robot interaction. In: 1st ACM SIGCHI/SIGART conference on human-robot interaction, Salt Lake City, Utah, USA. ACM, New York

    Google Scholar 

  33. Stevens JP (1999) Intermediate statistics a modern approach, 2nd edn. Erlbaum, Mahwah

    MATH  Google Scholar 

  34. Watson D, Clark LA, Tellegen A (1988) Development and validation of brief measures of positive and negative affect: the Panas scales. J Pers Soc Psychol 54(6):1063–1070

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Yale University, New Haven, CT, 06511, USA

    Cindy L. Bethel

  2. Center for Robot-Assisted Search and Rescue, Texas A & M University, College Station, TX, 77843, USA

    Robin R. Murphy

Authors
  1. Cindy L. Bethel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robin R. Murphy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cindy L. Bethel.

Additional information

This material is based upon work supported by the National Science Foundation under Grant # 0937060 to the Computing Research Association for the CIFellows Project, a National Science Foundation Graduate Research Fellowship Award Number DGE-0135733, ARL Number W911NF-06-2-0041, IEEE Robotics and Automation Society Graduate Fellowship, and a Microsoft HRI grant.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bethel, C.L., Murphy, R.R. Review of Human Studies Methods in HRI and Recommendations. Int J of Soc Robotics 2, 347–359 (2010). https://doi.org/10.1007/s12369-010-0064-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-010-0064-9

Keywords

Search

Navigation