Towards Reproducible Language-Based HRI Experiments: Open-Sourcing a Generalized Experiment Project
Authors: 
Uthman Tijani, 
Hong Wang, 
Zhao HanAuthors Info & Claims
Uthman Tijani, Hong Wang, Zhao HanAuthors Info & ClaimsPages 949 - 953
Abstract
We are witnessing increasing calls for reproducibility and replicability in HRI studies to improve reliability and confidence in empirical findings. One solution to facilitate this is using a robot platform that researchers frequently use, making it easier to replicate studies to verify results. In this work, we focus on a popular, affordable, and rich-in-functionality robot platform, NAO/Pepper, and contribute a generalized experiment project specifically for conducting language-based HRI experiments where a robot instructs a human for a task, including objective data collection.
Specifically, we first describe a concrete workflow from an existing experiment and how it is generalized. We then evaluate the generalized project with a case study to show how adopters can quickly adapt to their specific experiment needs. This work provides inspiration for HRI researchers to not only provide their experiment code as supplementary material but also generalize them to benefit other researchers to advance empirical research in HRI. The generalized Choregraphe project with documentation, demo, and usage notes is available under MIT license on GitHub at https://github.com/TheRARELab/langex. We welcome questions by posting GitHub issues and pull requests to share adapted packages.
References
[1]
Aida Amirova, Nazerke Rakhymbayeva, Elmira Yadollahi, Anara Sandygulova, and Wafa Johal. 2021. 10 years of human-nao interaction research: A scoping review. Frontiers in Robotics and AI, Vol. 8 (2021), 744526.
[2]
Carl Auerbach and Louise B Silverstein. 2003. Qualitative data: An introduction to coding and analysis. Vol. 21. NYU press.
[3]
Shelly Bagchi, Patrick Holthaus, Gloria Beraldo, Emmanuel Senft, Daniel Hernandez Garcia, Zhao Han, Suresh Kumaar Jayaraman, Alessandra Rossi, Connor Esterwood, Antonio Andriella, et al. 2023. Towards Improved Replicability of Human Studies in Human-Robot Interaction: Recommendations for Formalized Reporting. In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 629--633.
[4]
Tony Belpaeme, James Kennedy, Aditi Ramachandran, Brian Scassellati, and Fumihide Tanaka. 2018. Social robots for education: A review. Science robotics, Vol. 3, 21 (2018), eaat5954.
[5]
Cynthia Breazeal, Paul L Harris, David DeSteno, Jacqueline M Kory Westlund, Leah Dickens, and Sooyeon Jeong. 2016. Young children treat robots as informants. Topics in cognitive science, Vol. 8, 2 (2016), 481--491.
[6]
Elizabeth Cha, Yunkyung Kim, Terrence Fong, Maja J Mataric, et al. 2018. A survey of nonverbal signaling methods for non-humanoid robots. Foundations and Trends® in Robotics, Vol. 6, 4 (2018), 211--323.
[7]
Julia R Cordero, Thomas R Groechel, and Maja J Matarić. 2022. A review and recommendations on reporting recruitment and compensation information in HRI research papers. In 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 1627--1633.
[8]
Maartje De Graaf, Somaya Ben Allouch, and Jan Van Dijk. 2017. Why do they refuse to use my robot? Reasons for non-use derived from a long-term home study. In Proceedings of the 2017 ACM/IEEE international conference on human-robot interaction. 224--233.
[9]
Jan de Wit, Arold Brandse, Emiel Krahmer, and Paul Vogt. 2020. Varied human-like gestures for social robots: Investigating the effects on children's engagement and language learning. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction. 359--367.
[10]
Juan Fasola and Maja J Matarić. 2015. Evaluation of a spatial language interpretation framework for natural human-robot interaction with older adults. In 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 301--308.
[11]
Kerstin Fischer. 2021. Effect confirmed, patient dead: A Commentary on Hoffman & Zhao's Primer for Conducting Experiments in HRI. ACM Transactions on Human-Robot Interaction (THRI), Vol. 10, 1 (2021), 1--4.
[12]
Marlena R Fraune, Iolanda Leite, Nihan Karatas, Aida Amirova, Amélie Legeleux, Anara Sandygulova, Anouk Neerincx, Gaurav Dilip Tikas, Hatice Gunes, Mayumi Mohan, et al. 2022. Lessons learned about designing and conducting studies from hri experts. Frontiers in Robotics and AI, Vol. 8 (2022), 772141.
[13]
Binnur Görer, Albert Ali Salah, and H Levent Akin. 2017. An autonomous robotic exercise tutor for elderly people. Autonomous Robots, Vol. 41 (2017), 657--678.
[14]
Erico Guizzo. 2014. How Aldebaran robotics built its friendly humanoid robot, Pepper. IEEE Spectrum (2014).
[15]
Hatice Gunes, Frank Broz, Chris S Crawford, Astrid Rosenthal-von der Pütten, Megan Strait, and Laurel Riek. 2022. Reproducibility in human-robot interaction: furthering the science of HRI. Current Robotics Reports, Vol. 3, 4 (2022), 281--292.
[16]
Zhao Han and Tom Williams. 2023 a. Evaluating Cognitive Status-Informed Referring Form Selection for Human-Robot Interactions. In 2023 Annual Meeting of the Cognitive Science Society (CogSci).
[17]
Zhao Han and Thomas Williams. 2023 b. Supplementary Materials for “Evaluating Cognitive Status-Informed Referring Form Selection for Human-Robot Interactions”'. https://doi.org/10.17605/OSF.IO/NZWVF
[18]
Guy Hoffman and Xuan Zhao. 2020. A primer for conducting experiments in human--robot interaction. ACM Transactions on Human-Robot Interaction (THRI), Vol. 10, 1 (2020), 1--31.
[19]
Malte Jung and Pamela Hinds. 2018. Robots in the wild: A time for more robust theories of human-robot interaction. ACM Transactions on Human-Robot Interaction (THRI), Vol. 7, 1 (2018), 1--5.
[20]
Rachel Kirby, Jodi Forlizzi, and Reid Simmons. 2010. Affective social robots. Robotics and Autonomous Systems, Vol. 58, 3 (2010), 322--332.
[21]
Hiroaki Kitano, Minoru Asada, Yasuo Kuniyoshi, Itsuki Noda, and Eiichi Osawa. 1997. Robocup: The robot world cup initiative. In Proceedings of the first international conference on Autonomous agents. 340--347.
[22]
Ross A Knepper, Todd Layton, John Romanishin, and Daniela Rus. 2013. Ikeabot: An autonomous multi-robot coordinated furniture assembly system. In 2013 IEEE International conference on robotics and automation. IEEE, 855--862.
[23]
Elly A Konijn, Brechtje Jansen, Victoria Mondaca Bustos, Veerle LNF Hobbelink, and Daniel Preciado Vanegas. 2022. Social robots for (second) language learning in (migrant) primary school children. International Journal of Social Robotics (2022), 1--17.
[24]
Namyeon Lee, Jeonghun Kim, Eunji Kim, and Ohbyung Kwon. 2017. The influence of politeness behavior on user compliance with social robots in a healthcare service setting. International Journal of Social Robotics, Vol. 9 (2017), 727--743.
[25]
Benedikt Leichtmann, Verena Nitsch, and Martina Mara. 2022. Crisis ahead? Why human-robot interaction user studies may have replicability problems and directions for improvement. Frontiers in Robotics and AI, Vol. 9 (2022), 838116.
[26]
Iolanda Leite, Marissa McCoy, Monika Lohani, Daniel Ullman, Nicole Salomons, Charlene Stokes, Susan Rivers, and Brian Scassellati. 2015. Emotional storytelling in the classroom: Individual versus group interaction between children and robots. In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction. 75--82.
[27]
Melissa & Doug. 2019. 100 Piece Wood Blocks Set. https://www.melissaanddoug.com/products/100-piece-wood-blocks-set. Accessed: 2023--11--17.
[28]
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, et al. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. Kobe, Japan, 5.
[29]
Aditi Ramachandran, Chien-Ming Huang, Edward Gartland, and Brian Scassellati. 2018. Thinking aloud with a tutoring robot to enhance learning. In Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction. 59--68.
[30]
Laurel D Riek. 2012. Wizard of oz studies in hri: a systematic review and new reporting guidelines. Journal of Human-Robot Interaction, Vol. 1, 1 (2012), 119--136.
[31]
Adam Robaczewski, Julie Bouchard, Kevin Bouchard, and Sébastien Gaboury. 2021. Socially assistive robots: The specific case of the NAO. International Journal of Social Robotics, Vol. 13 (2021), 795--831.
[32]
Aldebaran Robotics. 2018a. 2D Cameras - Aldebaran 2.5.11.14a documentation. http://doc.aldebaran.com/2--5/family/pepper_technical/video_2D_pep.html.
[33]
Aldebaran Robotics. 2018b. Choregraphe - Tutorials - Aldebaran 2.5.11.14a documentation. http://doc.aldebaran.com/2--5/software/choregraphe/tutos/index.html.
[34]
Aldebaran Robotics. 2018c. Flow diagram panel - Aldebaran 2.5.11.14a documentation. http://doc.aldebaran.com/2--5/software/choregraphe/panels/flow_diagram_panel.html.
[35]
Aldebaran Robotics. 2018d. Hello World 1 - using Choregraphe - Aldebaran 2.5.11.14a documentation. http://doc.aldebaran.com/2--5/getting_started/helloworld_choregraphe.html.
[36]
Aldebaran Robotics. 2018 e. Memory watcher panel - Aldebaran 2.5.11.14a documentation. http://doc.aldebaran.com/2--5/software/choregraphe/panels/memory_watcher_panel.html.
[37]
Aldebaran Robotics. 2018 f. Video camera - Aldebaran 2.5.11.14a documentation. http://doc.aldebaran.com/2--5/family/robots/video_robot.html.
[38]
Aldebaran Robotics. 2023 a. Aldebaran | Humanoid and programmable robots. https://www.aldebaran.com/en/.
[39]
Aldebaran Robotics. 2023 b. NAO the humanoid and programmable robot | Aldebaran. https://www.aldebaran.com/en/nao.
[40]
Sarah Sebo, Brett Stoll, Brian Scassellati, and Malte F Jung. 2020. Robots in groups and teams: a literature review. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW2 (2020), 1--36.
[41]
Kevin Spevak, Zhao Han, Tom Williams, and Neil T Dantam. 2022. Givenness hierarchy informed optimal document planning for situated human-robot interaction. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 6109--6115.
[42]
Laura Stegner, Emmanuel Senft, and Bilge Mutlu. 2023. Situated participatory design: A method for in situ design of robotic interaction with older adults. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1--15.
[43]
Wojciech 'Swika tkowski and Beno^it Dompnier. 2017. Replicability crisis in social psychology: Looking at the past to find new pathways for the future. International Review of Social Psychology, Vol. 30, 1 (2017), 111--124.
[44]
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. 2020. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, Vol. 3 (2020), 25--55.
[45]
Daniel Ullman, Salomi Aladia, and Bertram F Malle. 2021. Challenges and opportunities for replication science in hri: A case study in human-robot trust. In Proceedings of the 2021 ACM/IEEE international conference on human-robot interaction. 110--118.
[46]
Rianne Van den Berghe, Josje Verhagen, Ora Oudgenoeg-Paz, Sanne Van der Ven, and Paul Leseman. 2019. Social robots for language learning: A review. Review of Educational Research, Vol. 89, 2 (2019), 259--295.
[47]
Bradford J Wiggins and Cody D Christopherson. 2019. The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, Vol. 39, 4 (2019), 202.
[48]
Megan Zimmerman, Shelly Bagchi, Jeremy Marvel, and Vinh Nguyen. 2022. An analysis of metrics and methods in research from human-robot interaction conferences, 2015--2021. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 644--648. io
Index Terms
- Towards Reproducible Language-Based HRI Experiments: Open-Sourcing a Generalized Experiment Project
Recommendations
Open science – towards reproducible research
APE 2017, Berlin, Germany, January 16-18, 2017This paper presents an overview of several efforts towards reproducible research in the field of medical imaging and visualization. In the first section, the components of Open Science are presented: open access, open data and open source. In the second ...
ReHRI'17 - Towards Reproducible HRI Experiments: Scientific Endeavors, Benchmarking and Standardization
HRI '17: Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot InteractionThis workshop aims at advancing the topic of the relationships between robotics science and experiments in HRI. The full-day workshop follows on previous events in interdisciplinary venues focusing on efforts to establish reproducible, standardized ...
Reproducible Online Search Experiments
Advances in Information RetrievalAbstractIn the empirical sciences, the evidence is commonly manifested by experimental results. However, very often, these findings are not reproducible, hindering scientific progress. Innovations in the field of information retrieval (IR) are mainly ...
Comments
Information & Contributors
Information
Published In
March 2024
982 pages
ISBN:9798400703225
DOI:10.1145/3610977
- General Chairs:
- Dan Grollman,
- Elizabeth Broadbent,
- Program Chairs:
- Wendy Ju,
- Harold Soh,
- Tom Williams
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 March 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
HRI '24: ACM/IEEE International Conference on Human-Robot Interaction
March 11 - 15, 2024
CO, Boulder, USA
Acceptance Rates
Overall Acceptance Rate 268 of 1,124 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 106Total Downloads
- Downloads (Last 12 months)106
- Downloads (Last 6 weeks)25
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in