Skip to main content

Advertisement

SpringerLink
Log in

Experimental Studies of Human–Robot Interaction: Threats to Valid Interpretation from Methodological Constraints Associated with Experimental Manipulations

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

The rapid development of artificial intelligence brings with it the increasing likelihood of ubiquitous interaction between humans and robots. A significant contribution to studying human–robot interactions (HRI) comes from experimental studies, whereby humans and robots interact in controlled conditions and researchers observe and measure the reactions of humans (and robots). The use of experiments to understand human interactions has long been a central source of information in the field of experimental social psychology. These studies have yielded numerous major insights into the causes and outcomes of interaction. The methodology of experiments, however, including the demands made upon human participants to behave in predictable ways and the impact of experimenters’ expectancies upon results, has been a focus of much critical analysis. We examined a sample of 100 high impact HRI studies for evidence of potentially contaminating experimental artefacts and/or authors’ awareness of such factors. In our conclusions we highlight several methodological issues that appeared frequently in our sample, which may impede generalisations from laboratory experiments to real-world settings. Ultimately, we suggest that researchers may need to reformulate the methodologies used to study the unique features of HRI, and offer a number of recommendations for researchers designing HRI experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Barber TX, Silver MJ (1968) Fact, fiction and the experimenter-bias effect. Psychol Bull 70:1–29

    Article  Google Scholar 

  2. Bargh JA, Chen M, Burrows L (1996) Automaticity of social behaviour: direct effect of trait construct and stereotype activation in action. J Pers Soc Psychol 71:230–244

    Article  Google Scholar 

  3. Baumeister RF, Vohs KD, Funder DC (2007) Psychology as the science of self-reports and finger movements: What happened to actual behavior? Perspect Psychol Sci 2:396–403

    Article  Google Scholar 

  4. Berkowitz NH (1994) Evidence that subjects’ expectancies confound intergroup bias in Tajfel’s minimal group paradigm. Pers Soc Psychol Bull 20:184–185

    Article  Google Scholar 

  5. Bless H, Burger AM (2016) A closer look at social psychologists’ silver bullet: inevitable and evitable side effects of the experimental approach. Perspect Psychol Sci 11(2):296–308

    Article  Google Scholar 

  6. Brannigan A (2004) The rise and fall of social psychology: the use and misuse of the experimental method. de Gruyn, New York

    Google Scholar 

  7. Brewer WF (2012) The theory ladenness of the mental processes used in the scientific enterprise: evidence from cognitive psychology and the history of science. In: Proctor RW, Capaldi EJ (eds) Psychology of science: implicit and explicit processes. Oxford university Press, New York

    Google Scholar 

  8. Broad E (2018) Made by humans: the AI condition. Melbourne University Press, Melbourne

    Google Scholar 

  9. Campbell DT (1957) Factors relevant to the validity of experiments in social settings. Psychol Bull 54:297–312

    Article  Google Scholar 

  10. Campbell DT (1994) The social psychology of scientific validity: an epistemological perspective and a personalized history. In: Shadish WR, Fuller S (eds) The social psychology of science. Guilford, New York, pp 124–161

    Google Scholar 

  11. Chapanis NP, Chapanis A (1964) Cognitive dissonance: five years later. Psychol Bull 61:1–22

    Article  Google Scholar 

  12. Clegg B (2017) Big data: how the information revolution is transforming our lives. Icon, London

    Google Scholar 

  13. Collins H (2019) Forms of life: the method and meaning of sociology. MIT Press, Cambridge

    Google Scholar 

  14. Collins H (2018) Artifictional intelligence: against humanity’s surrender to computers. Polity, Cambridge

    Google Scholar 

  15. Collins H (2010) Tacit and explicit knowledge. University of Chicago Press, Chicago

    Book  Google Scholar 

  16. Collins H, Evans R (2007) Rethinking expertise. University of Chicago Press, Chicago

    Book  Google Scholar 

  17. Cook TD, Campbell DT (1978) Quasi-experimentation: design and analysis issues for field settings. Rand-McNally, Chicago

    Google Scholar 

  18. Cooper J (2007) Cognitive dissonance: 50 years of a classic theory. Sage, New York

    Google Scholar 

  19. Crawford JT, Jussim L (eds) (2018) The politics of social psychology. Routledge, New York

    Google Scholar 

  20. Deliginais C, Stanton C, McGarty C, Stevens CJ (2017) The impact of intergroup trust and approach behaviour towards a humanoid robot. J Hum Robot Interact 6:4–20

    Article  Google Scholar 

  21. Duarte J, Crawford J, Stern C, Haidt J, Jussim L, Tetlock P (2015) Political diversity will improve social psychological research. Behav Brain Sci 38:1–58

    Article  Google Scholar 

  22. Doyen S, Klein O, Pichon C-L, Cleermans A (2012) Behavioral priming: It’s all in the mind, but whose mind? PloS ONE 7(1):29081. https://doi.org/10.1371/journal.pone.0029081

    Article  Google Scholar 

  23. Festinger L (1957) A theory of cognitive dissonance. Stanford University Press, Stanford

    Google Scholar 

  24. Fleiss JL (1981) Statistical methods for rates and proportions, 1st edn. Wiley, London

    MATH  Google Scholar 

  25. Friedman N (1967) The social nature of psychological research. Basic Books, New York

    Google Scholar 

  26. Geiskkovitch DY, Cormier D, Seo SH, Young JE (2016) Please continue, we need more data: an exploration of obedience to robots. J Hum Robot Interact 5:82–99

    Article  Google Scholar 

  27. Gerard H (1999) A social psychologist examines his past and looks to the future. In: Rodrigues A, Levine R (eds) Reflections on 100 years of experimental social psychology. Basic Books, New York, pp 47–81

    Google Scholar 

  28. Grinnell F (2009) Everyday practice of science: where intuition and passion meet objectivity and logic. Oxford University Press, New York

    Google Scholar 

  29. Hart CW (1947) Some factors affecting the organization and prosecution of given research projects. Am Sociol Rev 12:514–519

    Article  Google Scholar 

  30. Harmon-Jones E, Mills J (1999) Cognitive dissonance: progress on a pivotal theory in social psychology. American Psychological Association, Washington, DC

    Book  Google Scholar 

  31. Haslam SA, Reicher SD, Millard K (2015) Shock treatment: using immersive digital realism to restage and re-examine Milgram’s ‘Obedience to Authority’ research. PloS ONE. https://doi.org/10.1371/journal.pone.0109015

    Article  Google Scholar 

  32. Hudson L (1972) The cult of the mind. Jonathan Cape, London

    Google Scholar 

  33. Inbar Y, Lammers J (2012) Political diversity in social and personality psychology. Perspect Psychol Sci 7(5):496–503

    Article  Google Scholar 

  34. Innes JM (2005) Decline of fact in artefact: loss of control in social psychological studies. Aust J Psychol Suppl 57:89

    Google Scholar 

  35. Innes JM, Fraser C (1971) Experimenter bias and other possible biases in psychological research. Eur J Soc Psychol 1:297–310

    Article  Google Scholar 

  36. Innes JM, Morrison BW (2017) Projecting the future impact of advanced technologies on the profession: Will a robot take my job? Aust Psychol Soc Psych 39(2):34–35

    Google Scholar 

  37. Kintz BL, Delprato DJ, Mettee DR, Persun CE, Schappe RH (1965) The experimenter effect. Psychol Bull 63:223–232

    Article  Google Scholar 

  38. Klein O, Doyen S, Leys C, daGama PA, Miller s, Questienne L, Cleeremans A (2012) Low hopes, high expectations: expectancy effects and the replicability of behavioural experiments. Perspect Psychol Sci 7(6):572–584

    Article  Google Scholar 

  39. Knapp RH (1963) Demographic, cultural and personality attributes of scientists. In: Taylor CW, Barron F (eds) Scientific creativity. Wiley, New York

    Google Scholar 

  40. Lana RE (1999) Pretest sensitization. In: Rosenthal R, Rosnow RL (eds) Artifact in behavioural research. Academic Press, New York, pp 121–146

    Google Scholar 

  41. Levin DT, Harriott C, Paul NA, Zhang T, Adams JA (2013) Cognitive dissonance as a measure of reactions to human–robot interaction. J Hum Robot Interact 2:1–17

    Article  Google Scholar 

  42. Li J, Ju W, Reeves B (2017) Touching a mechanical body: tactile contact with body parts of a humanoid robot is physiologically arousing. J Hum Robot Interact 6:118–130

    Article  Google Scholar 

  43. McAdams DP (2015) The art and science of personality development. Guilford, New York

    MATH  Google Scholar 

  44. MacCoun RJ (1998) Biases in the interpretation and use of research results. Annu Rev Psychol 49:259–287

    Article  Google Scholar 

  45. McGuigan FJ (1963) The experimenter: a neglected stimulus object. Psychol Bull 60:421–428

    Article  Google Scholar 

  46. McGuire WJ (1969) Suspiciousness of experimenter’s intent. In: Rosenthal R, Rosnow RL (eds) Artifact in behavioural research. Academic Press, New York, pp 13–60

    Google Scholar 

  47. Milgram S (1974) Obedience to authority. McGraw-Hill, New York

    Google Scholar 

  48. Miller AG (1972) The social psychology of psychological research. Free Press, New York

    Google Scholar 

  49. Miller N, Pollock VE (1994) Meta-analysis and some science-compromising problems of social psychology. In: Shadish WR, Fuller S (eds) The social psychology of science. Guilford, New York, pp 230–261

    Google Scholar 

  50. Muller JZ (2018) The tyranny of metrics. Princeton University Press, Princeton

    Book  Google Scholar 

  51. O’Gorman F (2017) Forgetfulness: making the modern culture of amnesia. Bloomsbury, London

    Book  Google Scholar 

  52. O’Neil C (2016) Weapons of math destruction. Penguin, London

    MATH  Google Scholar 

  53. Orne MT (1962) On the social psychology of the psychological experiment: with particular reference to demand characteristics. Am Psychol 17:776–783

    Article  Google Scholar 

  54. Orne MT (1969) Demand characteristics and the concept of quasi-controls. In: Rosenthal R, Rosnow RL (eds) Artifact in behavioural research. Academic Press, New York, pp 143–179

    Google Scholar 

  55. Passinin S, Morselli D (2010) The obedience–disobedience dynamic and the role of responsibility. J Commun Appl Soc Psychol 20:1–14

    Article  Google Scholar 

  56. Pastore N (1949) The nature–nurture controversy. King’s Crown Press, New York

    Google Scholar 

  57. Pepitone A (1999) Historical sketches and critical commentary about social psychology in the golden age. In: Rodrigues A, Levine R (eds) Reflections on 100 years of experimental social psychology. Basic Books, New York, pp 170–199

    Google Scholar 

  58. Polanyi M (1958) Personal knowledge. Routledge, London

    Google Scholar 

  59. Popper K (1959) The logic of scientific discovery. Basic Books, New York

    MATH  Google Scholar 

  60. Reicher SD, Haslam SA, Smith JR (2012) Working toward the experimenter: reconceptualizing obedience within the Milgram paradigm as identification-based followership. Perspect Psychol Sci 7:315–324

    Article  Google Scholar 

  61. Robinson KA, Goodman SN, S.N (2011) A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med 154:50

    Article  Google Scholar 

  62. Rosenthal R (1963) On the social psychology of the psychological experiment: the experimenter’s hypothesis as unintended determinant of experimental results. Am Sci 51:268–283

    Google Scholar 

  63. Rosenthal R (1966) Experimenter effects in behavioural research. Appleton-Century-Crofts, New York

    Google Scholar 

  64. Rosenthal R (1969) Interpersonal expectations: effects of the experimenter’s hypothesis. In: Rosenthal R, Rosnow RL (eds) Artifact in behavioural research. Academic Press, New York, pp 181–277

    Google Scholar 

  65. Rosenthal R (1976) Experimenter effects in behavioural research, enl. Irvington, New York

    Google Scholar 

  66. Rosenthal R (1994) On being one’s own case study: experimenter effects in behavioural research-30 years later. In: Shadish WR, Fuller S (eds) The social psychology of science. Guilford, New York, pp 214–229

    Google Scholar 

  67. Rosenthal R, Rosnow RL (1969) The volunteer subject. In: Rosenthal R, Rosnow RL (eds) Artifact in behavioural research. Academic Press, New York, pp 61–120

    Google Scholar 

  68. Rosenthal R, Rubin DB (1978) Interpersonal expectancy effects: the first 345 studies. Behav Brain Sci 3:377–386

    Article  Google Scholar 

  69. Rosenberg MJ (1965) When dissonance fails: on eliminating evaluation apprehension from attitude measurement. J Pers Soc Psychol 1:18–42

    Article  Google Scholar 

  70. Rosenberg MJ (1969) The conditions and consequences of evaluation apprehension. In: Rosenthal R, Rosnow RL (eds) Artifact in behavioural research. Academic Press, New York, pp 280–350

    Google Scholar 

  71. Rosenzweig R (1933) The experimental situation as a psychological problem. Psychol Rev 40:337–354

    Article  Google Scholar 

  72. Rosnow RL, Suls JM (1970) Reactive effects of pretesting in attitude research. J Pers Soc Psychol 15:338–343

    Article  Google Scholar 

  73. Rubin M (2016) The perceived awareness of the research hypothesis scale: assessing the influence of demand characteristics. Figshare. https://doi.org/10.6084/m9.figshare.4315778

    Article  Google Scholar 

  74. Sadler M, Regan N (2019) Game changer: AlphaZero’s ground breaking chess strategies and the promise of AI. Google Books, Alkmaar

    Google Scholar 

  75. Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental designs for generalised causal inference. Houghton-Mifflin, Boston

    Google Scholar 

  76. Sherwood JJ, Nataupsky M (1968) Predicting the conclusions of negro-white intelligence research from biographical characteristics of the investigator. J Pers Soc Psychol 8:53–58

    Article  Google Scholar 

  77. St. Claire L, Turner JC (1982) The role of demand characteristics in the social categorization paradigm. Eur J Soc Psychol 12:307–314

    Article  Google Scholar 

  78. Toomela A, Valsiner J (eds) (2010) Methodological thinking in psychology: 60 years gone astray? Information Age Publishing, Charlotte

    Google Scholar 

  79. Wagner AJ, Borenstein J, Howard A (2018) Overtrust in the robotic age: a contemporary ethical challenge. Commun ACM 61(9):22–24

    Article  Google Scholar 

  80. Walsh T (2018) 2062. LaTrobe University Press, Melbourne

    Google Scholar 

  81. Wilson TD, Aronson E, Carlsmith K (2010) The art of laboratory experimentation. In: Fiske ST, Gilbert DT, Lindzey G (eds) Handbook of social psychology. Wiley, Hoboken, pp 51–81

  82. Zajonc RB (1999) One hundred years of rationality assumptions in social psychology. In: Rodrigues A, Levine R (eds) Reflections on 100 years of experimental social psychology. Basic Books, New York, pp 200–214

    Google Scholar 

  83. Zimbardo PG (1999) Experimental social psychology: Behaviorism with minds and matters. In: Rodrigues A, Levine R (eds) Reflections on 100 years of experimental social psychology. Basic Books, New York, pp 135–157

    Google Scholar 

Download references

Funding

This was not a funded study.

Author information

Authors and Affiliations

  1. Australian College of Applied Psychology, 16 Coglin Street, Adelaide, SA, 5000, Australia

    J. Michael Innes

  2. University of South Australia, Adelaide, Australia

    J. Michael Innes

  3. Charles Sturt University, Bathurst, Australia

    Ben W. Morrison

  4. Macquarie University, Sydney, Australia

    Ben W. Morrison

Authors
  1. J. Michael Innes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ben W. Morrison
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. Michael Innes.

Ethics declarations

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Innes, J.M., W. Morrison, B. Experimental Studies of Human–Robot Interaction: Threats to Valid Interpretation from Methodological Constraints Associated with Experimental Manipulations. Int J of Soc Robotics 13, 765–773 (2021). https://doi.org/10.1007/s12369-020-00671-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-020-00671-8

Keywords

Search

Navigation