Abstract
This work investigates how the design of crowdsourced tasks can influence responses. As a formative line of inquiry, this study sought to understand how users would respond either through movement, response, or shift of focus to varying flight paths from a drone. When designing an experiment, running several proto-studies can help with generating a dataset that is actionable, but it has been unclear how differences in things such as phrasing or pre- and post-surveys can impact the results. Leveraging methods from psychology, computer-supported cooperative work, and the human-robot interaction communities this work explored the best practices and lessons learned for crowdsourcing to reduce time to actionable data for defining new communication paradigms. The lessons learned in this work will be applicable broadly within the human-robot interaction community, even outside those who are interested in defining flight paths, because they provide a scaffold on which to build future experiments seeking to communicate using non-anthropomorphic robots. Important results and recommendations include: increased negative affect with increased question quantity, completion time being relatively consistent based on total number of responses rather than number of videos, responses being more related to the video than the question, and necessity of varying question lengths to maintain engagement.
Supported by NSF-IIS-1925148, NSF-IIS-1750750, NSF-CNS-1757908, and NSF-IIS-1638099.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bartneck, C., Kanda, T., Mubin, O., Al Mahmud, A.: Does the design of a robot influence its animacy and perceived intelligence? Int. J. Social Robot. 1(2), 195–204 (2009)
Brosnan, K., Babakhani, N., Dolnicar, S.: “i know what you’re going to ask me” why respondents don’t read survey questions. Int. J. Market Res., 1470785318821025 (2019)
Cañamero, L., Fredslund, J.: I show you how i like you-can you read it in my face? [robotics]. IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 31(5), 454–459 (2001)
Casler, K.: Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. https://doi.org/10.1016/j.chb.2013.05.009
Chaminade, T., et al.: Brain response to a humanoid robot in areas implicated in the perception of human emotional gestures. PLoS ONE 5(7), e11577 (2010)
Christensen, L.B., Johnson, B., Turner, L.A., Christensen, L.B.: Research methods, design, and analysis (2011)
Duncan, B.A., Beachly, E., Bevins, A., Elbaum, S., Detweiler, C.: Investigation of communicative flight paths for small unmanned aerial systems* this work was supported by NSF NRI 1638099. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 602–609. IEEE (2018)
Firestone, J.W., Quiñones, R., Duncan, B.A.: Learning from users: an elicitation study and taxonomy for communicating small unmanned aerial system states through gestures. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 163–171. IEEE (2019)
Gadiraju, U., Kawase, R., Dietze, S., Demartini, G.: Understanding malicious behavior in crowdsourcing platforms: the case of online surveys. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1631–1640. ACM (2015)
Gadiraju, U., Yang, J., Bozzon, A.: Clarity is a worthwhile quality: on the role of task clarity in microtask crowdsourcing. In: Proceedings of the 28th ACM Conference on Hypertext and Social Media, pp. 5–14. ACM (2017)
Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics, 159–174 (1977)
Riek, L.D., Rabinowitch, T.C., Bremner, P., Pipe, A.G., Fraser, M., Robinson, P.: Cooperative gestures: effective signaling for humanoid robots. In: Proceedings of the 5th ACM/IEEE International Conference on Human-Robot Interaction, pp. 61–68. IEEE Press (2010)
Sorokin, A., Berenson, D., Srinivasa, S.S., Hebert, M.: People helping robots helping people: crowdsourcing for grasping novel objects. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2117–2122, October 2010. https://doi.org/10.1109/IROS.2010.5650464
Takayama, L., Dooley, D., Ju, W.: Expressing thought: improving robot readability with animation principles. In: 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 69–76. IEEE (2011)
Tellex, S., et al.: Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation, p. 8
Toris, R., Kent, D., Chernova, S.: The robot management system: a framework for conducting human-robot interaction studies through crowdsourcing. J, Hum.-Robot Interact. 3(2), 25 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Bevins, A., McPhaul, N., Duncan, B.A. (2020). Content Is King: Impact of Task Design for Eliciting Participant Agreement in Crowdsourcing for HRI. In: Wagner, A.R., et al. Social Robotics. ICSR 2020. Lecture Notes in Computer Science(), vol 12483. Springer, Cham. https://doi.org/10.1007/978-3-030-62056-1_53
Download citation
DOI: https://doi.org/10.1007/978-3-030-62056-1_53
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62055-4
Online ISBN: 978-3-030-62056-1
eBook Packages: Computer ScienceComputer Science (R0)