Skip to main content
SpringerLink
Log in

Methodological Issues in Scenario-Based Evaluation of Human–Robot Interaction

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Scenarios have been widely used for evaluating human–robot interactions (HRIs). However, little has been reported on systematic utilization of different types of media for deploying HRI scenarios. This study investigates the methodological issues in scenario-based HRI evaluation, focusing on the effect of scenario media on user attitudes toward robots. Two experiments are designed to examine how scenario media may influence the elder adults’ attitudes towards social robots. Different types of scenario media, including text, video, interactive video, and live interaction, were compared systematically with respect to established evaluation criteria. The results showed that the characteristics of scenario media influenced users’ acceptance of robots and affected their attitudes. The outcome of the study helps designers to select scenario media for deploying contextual information of HRI.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Forlizzi J (2007) How robotic products become social products: an ethnography study of cleaning in the home. In: Proceedings HRI’07, Arlington, pp 127–136

  2. Broadbent E, Stafford R, MacDonald B (2009) Acceptance of healthcare robots for older population: review and future directions. I J Soc Robot 1:319–330

    Article  Google Scholar 

  3. Lee MK, Kiesler S, Forlizzi J (2010) Receptionist or information kiosk: how do people talk with a robot? CSCW 2010. Savannah, pp 31–40

  4. Heerink M, Kröse B, Evers V, Wielinga B (2008) The influence of social presence on acceptance of a companion robot by older people. J Phys Agents 2(2):33–40

    Google Scholar 

  5. Walters ML, Lohse M, Hanheide M, Wrede B, Syrdal DS, Koay KL, Green A, Hüttenrauch H, Dautenhahn K, Sagerer G, Severinson-Eklundh K (2011) Evaluating the robot personality and verbal behavior of domestic robots using video-based studies. Adv Robot 25:2233–2254

    Article  Google Scholar 

  6. Weiss A, Tscheligi M (2010) Special issue on robots for future societies: evaluating social acceptance and societal impact of robots. I J Soc Robot 2(4):345–346

    Article  Google Scholar 

  7. Bethel CL, Murphy RR (2010) Review of human studies methods in HRI and recommendations. I J Soc Robot 2(4):347–359

    Article  Google Scholar 

  8. Young JE, Sung J-Y, Voida A, Sharlin E, Igarashi T, Christensen HI, Grinter RE (2011) Evaluating human–robot interaction: focusing on the holistic interaction experience. I J Soc Robot 3(1):53–67

    Article  Google Scholar 

  9. Dautenhahn K (2007) Methodology and themes of human–robot interaction: a growing research field. I J Adv Robot Syst 4(1):103–108

    Google Scholar 

  10. Salvini P, Laschi C, Dario P (2010) Design for acceptability: improving robots’ coexistence in human society. I J Soc Robot 2(4):451–460

    Article  Google Scholar 

  11. Woods S, Walters M, Koay KL, Dautenhahn K (2006) Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In: 9th international workshop on advanced motion control (AMC’06), Istanbul, pp 750–755

  12. Weiss A, Wurhofer D, Bernhaupt R, Almaninger M, Tscheligi M (2010) A methodological adaptation for heuristic evaluation of HRI. In: Proceedings of the 19th IEEE international symposium on robot and human interactive communication, Principe di Piemonte -Viareggio, pp 1–6

  13. Carroll JM (2000) Making use: scenario-based design of human–computer interactions. MIT Press, Cambridge

    Google Scholar 

  14. Rosson MB, Carroll JM (2002) Usability engineering: scenario-based development of human–computer interaction. Morgan Kaufmann, San Francisco

    Google Scholar 

  15. Xu Q, Ng JSL, Cheong YL, Tan OY, Wong JB, Chee BTT, Park T (2012) The role of social context in human–robot interaction evaluation. In: Southeast Asian Network of Ergonomics Societies (SEANES), Langkawi

  16. Xu Q, Helander M (2009) Scenario-based affective design for vehicle ecosystems. In: 17th congress of the international ergonomics association, Beijing

  17. Clarkson E, Arkin RC (2007) Applying heuristic evaluation to human–robot interaction systems. In: Proceedings of the twentieth international Florida artificial intelligence research society conference, Key West, pp 44–49

  18. Woods SN, Walters ML, Koay KL, Dautenhahn K (2006) Methodological issues in HRI: a comparison of live and video- based methods in robot to human approach direction trials. In: 15th IEEE international symposium on robot and human interactive communication (RO-MAN 06), Hatfield, pp 51–58

  19. Xu Q, Ng JSL, Cheong YL, Tan OY, Wong JB, Chee BTT, Park T (2012) Effect of scenario media on elder adults’ evaluation of human robot interaction. In: 21st IEEE international symposium on robot and human interactive communication, Paris, pp 455–462

  20. Rolland C, Achour CB, Cauvet C, Ralyté J, Sutcliffe A, Maiden N, Jarke M, Haumer P, Pohl K, Dubois E, Heymans P (1998) A proposal for scenario classification framework. Requir Eng 3:23–47

    Article  Google Scholar 

  21. Sung J, Grinter RE, Christensen HI (2010) Domestic robot ecology: an initial framework to unpack long-term acceptance of robots at home. I J Soc Robot 2(4):417–429

    Article  Google Scholar 

  22. Forlizzi J, DiSalvo C (2006) Service robots in the domestic environment: a study of the roomba vacuum in the home. In: 1st ACM SIGCHI/SIGART conference on human–robot interaction (HRI’06), Salt Lake City, pp 258–265

  23. Sabelli AM, Kanda T, Hagita N (2011) A conversational robot in an elderly care center: an ethnographic study. HRI’11, Lausanne, pp 37–44

  24. Yamazaki K, Yamazaki A, Okada M, Kuno Y, Kobayashi Y, Hoshi Y, Pitsch K, Luff P, vom Lehn D, Heath C (2009) Revealing gauguin: engaging visitors in robot guide’s explanation in an art museum. CHI 2009, Boston, pp 1437–1446

  25. Mutlu B, Forlizzi J (2008) Robots in organizations: the role of workflow, social, and environmental factors in human–robot interaction. HRI’08, Amsterdam, pp 287–294

  26. Coradeschi S, Kristoffersson A, Loutfi A, von Rump S, Cesta A, Cortellessa G, Gonzalez J (2011) Towards a methodology for longitudinal evaluation of social robotic telepresence for elderly. In: HRI 2011 1st workshop on social robotic telepresence, Laussanne.

  27. Paiva A, Dias J, Sobral D, Aylett R, Woods S, Hall L, Zoll C (2004) Caring for agents that care: building empathic relations with synthetic agents. In: AAMAS autonomous agents and multi agent systems, New York City, pp 194–201

  28. Wainer J, Feil-Seifer DJ, Shell D, Mataric M (2006) The role of physical embodiment in human–robot interaction. In: International workshop on robot and human interaction communication (RO-MAN06), Hatfield, pp 117–122.

  29. Powers A, Kiesler S, Fussell S, Torrey C (2007) Comparing a computer agent with a humanoid robot. In: Proceedings of the HRI’07 Arlington, pp 145–152.

  30. Looije R, Neerincx MA, Cnossen F (2010) Persuasive robotic assistant for health self-management of older adults: design and evaluation of social behaviors. Int J Hum-Comput Stud 68(6):386– 397

    Article  Google Scholar 

  31. Bainbridge WA, Hart JW, Kim ES, Scassellati B (2010) The benefits of interactions with physically present robots over video-displayed agents. I J Soc Robot 3(1):41–52

    Article  Google Scholar 

  32. Riek LD, Rabinowitch TC, Bremner P, Pipe AG, Fraser M, Robinson P (2010) Cooperative gestures: Effective signaling for humanoid robots. In: 5th ACM/IEEE international conference on human–robot interaction, pp 61–68

  33. McCurdy M, Connors C, Pyrzak G, Kanefsky B, Vera A (2006) Breaking the fidelity barrier: an examination of our current characterization of prototypes and an example of a mixed-fidelity success. CHI 2006, Montréal, pp 1233–1242

  34. Walker M, Takayama L, Landay JA (2002) High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. In: Proceedings of the human factors and ergonomics society 46th annual meeting, Baltimore, pp 661–665

  35. Heerink M (2010) Assessing acceptance of assistive social robots by aging adults. University of Amsterdam, Amsterdam

    Google Scholar 

  36. Nomura T, Suzuki T, Kanda T, Kato K (2006) Altered attitudes of people toward robots: investigation through the negative attitudes toward robots scale. In: Proceedings of AAAI-06 workshop on human implications of human–robot interaction, pp 29–35

  37. Syrdal DS, Dautenhahn K, Koay KL, Walters ML (2009) The negative attitudes towards robots scale and reactions to robot behaviour in a live human–robot interaction study. In: New frontiers in HRI symposium, Edinburgh

  38. Tsui KM, Desai M, Yanco HA (2010) Using the “negative attitude toward robots scale” with telepresence robots. In: Performance metrics for intelligent systems workshop (PerMIS’10), Baltimore

  39. Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified view. MIS Q 27:425–478

    Google Scholar 

  40. De Ruyter B, Saini P, Markopoulos P, van Breemen A (2005) Assessing the effects of building social intelligence in a robotic interface for the home. Interact Comput 17:522–541

    Article  Google Scholar 

  41. Heerink M, Kröse B, Evers V, Wielinga B (2010) Assessing acceptance of assistive social agent technology by older adults: the Almere model. I J Soc Robot 2:361–375

    Article  Google Scholar 

  42. Secord PF, Backman CW (1964) Social psychology. McGraw Hill, New York

    Google Scholar 

  43. Ajzen I (1991) The theory of planned behavior. Organ Behavi Hum Dec Process 50:179–211

    Article  Google Scholar 

  44. Connell JB, Mendelsohn GA, Robins RW, J. C, (2001) Effects of communication medium on interpersonal perceptions: don’t hang up the telephone yet!. GROUP’01, Boulder, pp 117–124

  45. Xu Q, Zhou F, Jiao J (2011) Affective-cognitive modeling for user experience with modular colored fuzzy petri nets. J Comput Inf Sci Eng 11(1):011004

    Article  Google Scholar 

  46. Crites SLJ, Fabrigar LR, Petty RE (1994) Measuring the affective and cognitive properties of attitudes: conceptual and methodological issues. Personal Soc Psychol Bull 20:619–634

    Article  Google Scholar 

  47. Kirby R, Forlizzi J, Simmons R (2010) Affective social robots. Robot Auton Syst 58:322–332

    Article  Google Scholar 

  48. Broadbent E, Tamagawa R, Patience A, Knock B, Kerse N, Day K, MacDonald BA (2012) Attitudes towards health-care robots in a retirement village. Aust J Ageing 31(2):115–120

    Article  Google Scholar 

  49. Ajzen I (1985) From intentions to actions: a theory of planned behavior. In: Kuhl J, Beckmann J (eds) Action control., SSSP Springer series in social psychologySpringer, Berlin Heidelberg, pp 11–39

    Chapter  Google Scholar 

  50. Chiou J-S (1998) The effects of attitude, subjective norm, and perceived behavioral control on consumers’ purchase intentions: the moderating effects of product knowledge and attention to social comparison information. Proc Natl Sci Counc 9:298–308

    Google Scholar 

  51. Taylor S, Todd P (1995) Decomposition and crossover effects in the theory of planned behavior: a study of consumer adoption intentions. Int J Res Market 12:137–156

    Article  Google Scholar 

  52. Lim YK, Pangam A, Periyasami S, Aneja S (2006) Comparative analysis of high- and low-fidelity prototypes for more valid usability evaluations of mobile devices. NordiCHI 2006, Oslo, pp 291–300

  53. Buxton B (2007) Sketching user experiences: getting the design right and the right design. Morgan Kaufmann, San Francisco

    Google Scholar 

  54. Davidoff S, Lee MK, Dey AK, Zimmerman J (2007) Rapidly exploring application design through speed dating. UbiComp 2007, Innsbruck, pp 429–446

Download references

Acknowledgments

The work was supported by Singapore Agency for Science, Technology and Research (A*STAR) Science and Engineering Research Council (SERC) Thematic Strategic Research Programme (TSRP) grant on Human Factors Engineering (#092 153 0090).

Author information

Authors and Affiliations

  1. Institute for Infocomm Research, Agency for Science, Technology and Research, 1 Fusionopolis Way, #21-01 Connexis, Singapore , 138632, Singapore

    Qianli Xu, Jamie Ng, Odelia Tan & Zhiyong Huang

  2. School of Mechanical & Aerospace Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore , 639798, Singapore

    Benedict Tay & Taezoon Park

Authors
  1. Qianli Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jamie Ng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Odelia Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiyong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Benedict Tay
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Taezoon Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qianli Xu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, Q., Ng, J., Tan, O. et al. Methodological Issues in Scenario-Based Evaluation of Human–Robot Interaction. Int J of Soc Robotics 7, 279–291 (2015). https://doi.org/10.1007/s12369-014-0248-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-014-0248-9

Keywords

Search

Navigation