Zahra Ashktorab
Skip Abstract Section
Abstract
Human-AI interaction is pervasive across many areas of our day to day lives. In this paper, we investigate human-AI collaboration in the context of a collaborative AI-driven word association game with partially observable information. In our experiments, we test various dimensions of subjective social perceptions (rapport, intelligence, creativity and likeability) of participants towards their partners when participants believe they are playing with an AI or with a human. We also test subjective social perceptions of participants towards their partners when participants are presented with a variety of confidence levels. We ran a large scale study on Mechanical Turk (n=164) of this collaborative game. Our results show that when participants believe their partners were human, they found their partners to be more likeable, intelligent, creative and having more rapport and use more positive words to describe their partner's attributes than when they believed they were interacting with an AI partner. We also found no differences in game outcome including win rate and turns to completion. Drawing on both quantitative and qualitative findings, we discuss AI agent transparency, include design implications for tools incorporating or supporting human-AI collaboration, and lay out directions for future research. Our findings lead to implications for other forms of human-AI interaction and communication.
- Kemo Adrian, Aysenur Bilgin, Paul Van Eecke, et al. 2016. A Semantic Distance based Architecture for a Guesser Agent in ESSENCE's Location Taboo Challenge. DIVERSITY@ ECAI (2016), 33--39.Google Scholar
- Ines Arous, Jie Yang, Mourad Khayati, and Philippe Cudré-Mauroux. 2020. OpenCrowd: A Human-AI Collaborative Approach for Finding Social Influencers via Open-Ended Answers Aggregation. In Proceedings of The Web Conference 2020. 1851--1862.Google ScholarDigital Library
- Gloria Barczak, Felicia Lassk, and Jay Mulki. 2010. Antecedents of team creativity: An examination of team emotional intelligence, team trust and collaborative culture. Creativity and innovation management 19, 4 (2010), 332--345.Google Scholar
- Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. 2019. The Hanabi Challenge: A New Frontier for AI Research. arXiv preprint arXiv:1902.00506 (2019).Google Scholar
- Christoph Bartneck, Elizabeth Croft, and Susana Zoghbi. 2009. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International journal of social robotics 1, 1 (2009), 71--81.Google ScholarCross Ref
- Austin Beattie, Autumn P Edwards, and Chad Edwards. 2020. A Bot and a Smile: Interpersonal Impressions of Chatbots and Humans Using Emoji in Computer-mediated Communication. Communication Studies (2020), 1--19.Google Scholar
- Anol Bhattacherjee. 2001. Understanding information systems continuance: an expectation-confirmation model. MIS quarterly (2001), 351--370.Google Scholar
- TimothyWBickmore, LauraMPfeifer, and BrianWJack. 2009. Taking the time to care: empowering low health literacy hospital patients with virtual nurse agents. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 1265--1274. Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW2, Article 96. Publication date: October 2020. 96:18 Zahra Ashktorab et al.Google Scholar
- Noam Brown and Tuomas Sandholm. 2018. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science 359, 6374 (2018), 418--424.Google Scholar
- Carrie J Cai, Jonas Jongejan, and Jess Holbrook. 2019. The effects of example-based explanations in a machine learning interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 258--262.Google ScholarDigital Library
- Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019. " Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--24.Google ScholarDigital Library
- California Governor. 2018. California new Autobot Law. https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=201720180SB1001.Google Scholar
- Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. 2002. Deep blue. Artificial intelligence 134, 1--2 (2002), 57--83.Google Scholar
- Colleen M Carpinella, Alisa B Wyman, Michael A Perez, and Steven J Stroessner. 2017. The Robotic Social Attributes Scale (RoSAS) Development and Validation. In Proceedings of the 2017 ACM/IEEE International Conference on humanrobot interaction. 254--262.Google ScholarDigital Library
- Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O'Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu. 2019. Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 559.Google ScholarDigital Library
- James A Crowder, John Carbone, and Shelli Friess. 2020. Human--AI Collaboration. In Artificial Psychology. Springer, 35--50.Google Scholar
- Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision. 2951--2960.Google ScholarCross Ref
- Jonathan Dodge, Sean Penney, Claudia Hilderbrand, Andrew Anderson, and Margaret Burnett. 2018. How the experts do it: Assessing and explaining agent behaviors in real-time strategy games. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--12.Google ScholarDigital Library
- Nicole B Ellison and Danah M Boyd. 2013. Sociality through social network sites. In The Oxford handbook of internet studies.Google Scholar
- Stanley E Fawcett, Stephen L Jones, and Amydee M Fawcett. 2012. Supply chain trust: The catalyst for collaborative innovation. Business Horizons 55, 2 (2012), 163--178.Google ScholarCross Ref
- Susan R Fussell, Sara Kiesler, Leslie D Setlock, and Victoria Yew. 2008. How people anthropomorphize robots. In 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 145--152.Google ScholarDigital Library
- Katy Gero, Zahra Ashktorab, Casey Dugan, and Werner Geyer. [n. d.]. Mental Models of AI Agents in a Cooperative Game Setting. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM.Google Scholar
- Elizabeth Gibney. 2016. Google AI algorithm masters ancient game of Go. Nature News 529, 7587 (2016), 445.Google ScholarCross Ref
- Arthur C Graesser, Patrick Chipman, Brian C Haynes, and Andrew Olney. 2005. AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education 48, 4 (2005), 612--618.Google ScholarDigital Library
- Francisco J Gutierrez, Sergio F Ochoa, Raymundo Cornejo, and Julita Vassileva. 2019. Designing Computer-Supported Technology to Mediate Intergenerational Social Interaction: A Cultural Perspective. In Perspectives on Human-Computer Interaction Research with Older People. Springer, 199--214.Google Scholar
- Renate Häuslschmid, Max von Buelow, Bastian Pfleging, and Andreas Butz. 2017. Supportingtrust in autonomous driving. In Proceedings of the 22nd international conference on intelligent user interfaces. 319--329.Google ScholarDigital Library
- Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in neural information processing systems. 2149--2159.Google Scholar
- Fatimah Ishowo-Oloko, Jean-François Bonnefon, Zakariyah Soroye, Jacob Crandall, Iyad Rahwan, and Talal Rahwan. 2019. Behavioural evidence for a transparency--efficiency tradeoff in human--machine cooperation. Nature Machine Intelligence 1, 11 (2019), 517--521.Google ScholarCross Ref
- Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T Hancock, and Mor Naaman. 2019. AI-Mediated Communication: How the Perception that Profile Text was Written by AI Affects Trustworthiness. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 239.Google ScholarDigital Library
- Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency. 29--38.Google ScholarDigital Library
- Cliff AC Lampe, Nicole Ellison, and Charles Steinfield. 2007. A familiar face (book): profile elements as signals in an online social network. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 435--444.Google ScholarDigital Library
- Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems. 4190--4203. Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW2, Article 96. Publication date: October 2020. Human-AI Collaboration in a Game Setting: Measuring Social Perception and Outcomes 96:19Google Scholar
- Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. [n. d.]. Search in Cooperative Partially-Observable Games. ([n. d.]).Google Scholar
- Jamy Li, René Kizilcec, Jeremy Bailenson, and Wendy Ju. 2016. Social robots and virtual agents as lecturers for video instruction. Computers in Human Behavior 55 (2016), 1222--1230.Google ScholarDigital Library
- Jingyi Li, Michelle X Zhou, Huahai Yang, and Gloria Mark. 2017. Confiding in and listening to virtual agents: The effect of personality. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. 275--286.Google ScholarDigital Library
- Claire Liang, Julia Proft, Erik Andersen, and Ross A Knepper. 2019. Implicit communication of actionable information in human-ai teams. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarDigital Library
- Sissi Liu. 2019. Everybody's Song Making: Do-it-yourself with and against Artificial Intelligence. Performance Research 24, 1 (2019), 120--128.Google ScholarCross Ref
- Maximilian Mackeprang, Claudia Müller-Birn, and Maximilian Timo Stauss. 2019. Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extraction. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--30.Google ScholarDigital Library
- Michael Muller, Susan R Fussell, Ge Gao, Pamela J Hinds, Nigini Oliveira, Katharina Reinecke, Lionel Robert Jr, Kanya Siangliulue, Volker Wulf, and Chien-Wen Yuan. 2019. Learning from Team and Group Diversity: Nurturing and Benefiting from our Heterogeneity. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing. 498--505.Google ScholarDigital Library
- Robert Munro, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen, and Harry Tily. 2010. Crowdsourcing and language studies: the new generation of linguistic data. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's Mechanical Turk. Association for Computational Linguistics, 122--130.Google ScholarDigital Library
- KAYA NAz and Helena Epps. 2004. Relationship between color and emotion: A study of college students. College Student J 38, 3 (2004), 396.Google Scholar
- Douglas L Nelson, Cathy L McEvoy, and Thomas A Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, & Computers 36, 3 (2004), 402--407.Google ScholarCross Ref
- David Novick and Iván Gris. 2014. Building rapport between human and ECA: A pilot study. In International Conference on Human-Computer Interaction. Springer, 472--480.Google ScholarCross Ref
- Jay F Nunamaker, Douglas C Derrick, Aaron C Elkins, Judee K Burgoon, and Mark W Patton. 2011. Embodied conversational agent-based kiosk for automated interviewing. Journal of Management Information Systems 28, 1 (2011), 17--48.Google ScholarDigital Library
- Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarDigital Library
- Stephen Oliver. 2019. Communication and trust: rethinking the way construction industry professionals and software vendors utilise computer communication mediums. Visualization in Engineering 7, 1 (2019), 1.Google ScholarCross Ref
- Ted Pedersen, Siddharth Patwardhan, and Jason Michelizzi. 2004. WordNet:: Similarity: measuring the relatedness of concepts. In Demonstration papers at HLT-NAACL 2004. Association for Computational Linguistics, 38--41.Google ScholarDigital Library
- Michael Rovatsos, Dagmar Gromann, and Gábor Bella. 2018. The Taboo Challenge Competition. AI Magazine 39, 1 (2018), 84--87.Google ScholarDigital Library
- Saqib Saeed, Sardar Zafar Iqbal, Hina Gull, Yasser ABamarouf, Mohammed AAlqahtani, Madeeha Saqib, and AbdullahM Alghamdi. 2019. Collaboration at Workplace: Technology Design Challenges of Segregated Work Environments. In 2019 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT). IEEE, 1--5.Google ScholarCross Ref
- Ameneh Shamekhi, Q Vera Liao, Dakuo Wang, Rachel KE Bellamy, and Thomas Erickson. 2018. Face Value? Exploring the effects of embodiment for a group facilitation agent. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 391.Google ScholarDigital Library
- Weiyan Shi, Xuewei Wang, Yoo Jung Oh, Jingwen Zhang, Saurav Sahay, and Zhou Yu. 2020. "Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies". In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '20). 15.Google ScholarDigital Library
- Weiyan Shi, Xuewei Wang, Yoo Jung Oh, Jingwen Zhang, Saurav Sahay, and Zhou Yu. 2020. Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies. arXiv preprint arXiv:2001.04564 (2020).Google Scholar
- David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140--1144.Google Scholar
- Karolis Tijunaitis, Debora Jeske, and Kenneth S Shultz. 2019. Virtuality at work and social media use among dispersed workers: Promoting network ties, shared vision and trust. Employee Relations: The International Journal 41, 3 (2019), 358--373. Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW2, Article 96. Publication date: October 2020. 96:20 Zahra Ashktorab et al.Google ScholarCross Ref
- Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. 2017. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 (2017).Google Scholar
- Janet H Walker, Lee Sproull, and R Subramani. 1994. Using a human face in an interface. In Proceedings of the SIGCHI conference on human factors in computing systems. 85--91.Google ScholarDigital Library
- Joseph B Walther. 2007. Selective self-presentation in computer-mediated communication: Hyperpersonal dimensions of technology, language, and cognition. Computers in Human Behavior 23, 5 (2007), 2538--2557.Google ScholarDigital Library
- Dakuo Wang, Justin D Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human-AI Collaboration in Data Science: Exploring Data Scientists? Perceptions of Automated AI. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--24.Google ScholarDigital Library
- Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim. 2019. Designing Theory-Driven User-Centric Explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 601.Google ScholarDigital Library
- Gesa Wiegand, Matthias Schmidmaier, Thomas Weber, Yuanting Liu, and Heinrich Hussmann. 2019. I Drive-You Trust: Explaining Driving Behavior Of Autonomous Cars. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, LBW0163.Google ScholarDigital Library
- Piers R Williams, Diego Perez-Liebana, and Simon M Lucas. 2016. Cooperative games with partial observability. Technical Report. Technical Report. IGGI.Google Scholar
- Ludwig Wittgenstein. 2009. Philosophical investigations. John Wiley & Sons.Google Scholar
- Yang Xu and Charles Kemp. 2010. Inference and communication in the game of Password. In Advances in neural information processing systems. 2514--2522.Google Scholar
- Nick Yee, Jeremy N Bailenson, and Kathryn Rickertsen. 2007. A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 1--10.Google ScholarDigital Library
- Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM.Google ScholarDigital Library
Index Terms
- Human-AI Collaboration in a Cooperative Game Setting: Measuring Social Perception and Outcomes
Recommendations
Mental Models of AI Agents in a Cooperative Game Setting
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsAs more and more forms of AI become prevalent, it becomes increasingly important to understand how people develop mental models of these systems. In this work we study people's mental models of AI in a cooperative word guessing game. We run think-aloud ...
Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsIn Human-AI collaborative settings that are inherently interactive, direction of communication plays a role in how users perceive their AI partners. In an AI-driven cooperative game with partially observable information, players (be it the AI or the ...
Synthesizing Game Levels for Collaborative Gameplay in a Shared Virtual Environment
We developed a method to synthesize game levels that accounts for the degree of collaboration required by two players to finish a given game level. We first asked a game level designer to create playable game level chunks. Then, two artificial ...
Comments