Michael Bonfert
Nima Zargham
Florian Saade
ABSTRACT
Smart displays augment the concept of a smart home speaker with a touchscreen. Although the visual modality is added in this device variant, the virtual agent is still only represented through auditory output and remains invisible in most current products. We present an empirical study on the interaction of users with a smart display on which the agent is embodied with a humanoid representation. Three different conditions are compared in a between-group experiment: no agent embodiment, a digitally rendered character, and a photorealistic representation performed by a human actress. Our quantitative data do not indicate that agent visualization on a smart display affects the user experience significantly. On the other hand, our qualitative findings revealed differentiated perspectives by the users. We discuss potentials and challenges of embodying agents on smart displays, reflect on their continuous on-screen presence, present user considerations on their appearance, and how the visualization influenced the politeness of the users.
Supplemental Material
- Elisabeth André. 2011. Design and evaluation of embodied conversational agents for educational and advisory software. In Gaming and Simulations: Concepts, Methodologies, Tools and Applications. IGI Global, Hershey, Pennsylvania, USA, 668–686.Google Scholar
- Sean Andrist, Michael Gleicher, and Bilge Mutlu. 2017. Looking Coordinated: Bidirectional Gaze Mechanisms for Collaborative Interaction with Virtual Characters. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2571–2582. https://doi.org/10.1145/3025453.3026033Google ScholarDigital Library
- Wilma A Bainbridge, Justin Hart, Elizabeth S Kim, and Brian Scassellati. 2008. The effect of presence on human-robot interaction. In RO-MAN 2008-The 17th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, IEEE Press, NJ, USA, 701–706.Google ScholarCross Ref
- Erin Beneteau, Olivia K. Richards, Mingrui Zhang, Julie A. Kientz, Jason Yip, and Alexis Hiniker. 2019. Communication Breakdowns Between Families and Alexa. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, Article 243, 13 pages. https://doi.org/10.1145/3290605.3300473Google ScholarDigital Library
- Timothy W. Bickmore and Rosalind W. Picard. 2005. Establishing and Maintaining Long-term Human-computer Relationships. ACM Trans. Comput.-Hum. Interact. 12, 2 (June 2005), 293–327. https://doi.org/10.1145/1067860.1067867Google ScholarDigital Library
- Michael Bonfert, Maximilian Spliethöver, Roman Arzaroli, Marvin Lange, Martin Hanci, and Robert Porzel. 2018. If you ask nicely: A digital assistant rebuking impolite voice commands. In Proceedings of the 20th ACM International Conference on Multimodal Interaction : ICMI’18 : Boulder, CO, USA, October 16 - 20, 2018. ACM, New York, NY, 95–102. https://doi.org/10.1145/3242969.3242995Google ScholarDigital Library
- Sheryl Brahnam and Antonella De Angeli. 2012. Gender affordances of conversational agents. Interacting with Computers 24, 3 (2012), 139–153.Google ScholarDigital Library
- J. Cassell, T. Bickmore, M. Billinghurst, L. Campbell, K. Chang, H. Vilhjálmsson, and H. Yan. 1999. Embodiment in Conversational Interfaces: Rea. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 520–527. https://doi.org/10.1145/302979.303150Google Scholar
- Justine Cassell, Joseph Sullivan, Elizabeth Churchill, and Scott Prevost. 2000. Embodied conversational agents. MIT press, MA, USA.Google ScholarDigital Library
- Susana Castillo, Philipp Hahn, Katharina Legde, and Douglas W. Cunningham. 2018. Personality Analysis of Embodied Conversational Agents. In Proceedings of the 18th International Conference on Intelligent Virtual Agents (Sydney, NSW, Australia) (IVA ’18). Association for Computing Machinery, New York, NY, USA, 227–232. https://doi.org/10.1145/3267851.3267853Google ScholarDigital Library
- Benjamin R. Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. ”What Can I Help You with?”: Infrequent Users’ Experiences of Intelligent Personal Assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI ’17). ACM, New York, NY, USA, Article 43, 12 pages. https://doi.org/10.1145/3098279.3098539Google ScholarDigital Library
- Andreea Danielescu. 2020. Eschewing Gender Stereotypes in Voice Assistants to Promote Inclusion. In Proceedings of the 2nd Conference on Conversational User Interfaces (Bilbao, Spain) (CUI ’20). Association for Computing Machinery, New York, NY, USA, Article 46, 3 pages. https://doi.org/10.1145/3405755.3406151Google ScholarDigital Library
- Munjal Desai, Kristen Stubbs, Aaron Steinfeld, and Holly Yanco. 2009. Creating trustworthy robots: Lessons and inspirations from automated systems. In Proceedings of AISB Convention: New Frontiers in Human-Robot Interaction. Society for the Study of Artificial Intelligence and the Simulation of Behaviour, Bath, UK, 49–56.Google Scholar
- Patrick Ehrenbrink, Seif Osman, and Sebastian Möller. 2017. Google Now is for the Extraverted, Cortana for the Introverted: Investigating the Influence of Personality on IPA Preference. In Proceedings of the 29th Australian Conference on Computer-Human Interaction (Brisbane, Queensland, Australia) (OZCHI ’17). ACM, New York, NY, USA, 257–265. https://doi.org/10.1145/3152771.3152799Google ScholarDigital Library
- Jesse Fox, Sun Joo (Grace) Ahn, Joris H. Janssen, Leo Yeykelis, Kathryn Y. Segovia, and Jeremy N. Bailenson. 2015. Avatars Versus Agents: A Meta-Analysis Quantifying the Effect of Agency on Social Influence. Human–Computer Interaction 30, 5 (2015), 401–432. https://doi.org/10.1080/07370024.2014.921494Google ScholarDigital Library
- Ted Grover, Kael Rowan, Jina Suh, Daniel McDuff, and Mary Czerwinski. 2020. Design and Evaluation of Intelligent Agent Prototypes for Assistance with Focus and Productivity at Work. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 390–400. https://doi.org/10.1145/3377325.3377507Google ScholarDigital Library
- Peter A Hancock, Deborah R Billings, Kristin E Schaefer, Jessie YC Chen, Ewart J De Visser, and Raja Parasuraman. 2011. A meta-analysis of factors affecting trust in human-robot interaction. Human factors 53, 5 (2011), 517–527.Google Scholar
- David Hanson, Andrew Olney, Steve Prilliman, Eric Mathews, Marge Zielke, Derek Hammons, Raul Fernandez, and Harry Stephanou. 2005. Upending the uncanny valley. In AAAI, Vol. 5. ACM, New York, NY, USA, 1728–1729.Google Scholar
- Marc Hassenzahl, Michael Burmester, and Franz Koller. 2003. AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität [AttracDiff: A questionnaire to measure perceived hedonic and pragmatic quality]. In Mensch & Computer 2003, Gerd Szwillus and Jürgen Ziegler (Eds.). B. G. Teubner, Stuttgart, 187–196.Google Scholar
- Álvaro Hernández-Trapote, Beatriz López-Mencía, David Díaz, Rubén Fernández-Pozo, and Javier Caminero. 2008. Embodied Conversational Agents for Voice-Biometric Interfaces. In Proceedings of the 10th International Conference on Multimodal Interfaces (Chania, Crete, Greece) (ICMI ’08). Association for Computing Machinery, New York, NY, USA, 305–312. https://doi.org/10.1145/1452392.1452454Google ScholarDigital Library
- Guy Hoffman, Jodi Forlizzi, Shahar Ayal, Aaron Steinfeld, John Antanitis, Guy Hochman, Eric Hochendoner, and Justin Finkenaur. 2015. Robot Presence and Human Honesty: Experimental Evidence. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (Portland, Oregon, USA) (HRI ’15). Association for Computing Machinery, New York, NY, USA, 181–188. https://doi.org/10.1145/2696454.2696487Google ScholarDigital Library
- Gilhwan Hwang, Jeewon Lee, Cindy Yoonjung Oh, and Joonhwan Lee. 2019. It Sounds Like A Woman: Exploring Gender Stereotypes in South Korean Voice Assistants. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). ACM, New York, NY, USA, Article LBW2413, 6 pages. https://doi.org/10.1145/3290607.3312915Google ScholarDigital Library
- Gilhwan Hwang, Jeewon Lee, Cindy Yoonjung Oh, and Joonhwan Lee. 2019. It Sounds Like A Woman: Exploring Gender Stereotypes in South Korean Voice Assistants. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312915Google Scholar
- Katherine Isbister and Patrick Doyle. 2002. Design and evaluation of embodied conversational agents: A proposed taxonomy. In The first international joint conference on autonomous agents & multi-agent systems. ACM, NY, USA.Google Scholar
- Rabia Khan and Antonella De Angeli. 2009. The attractiveness stereotype in the evaluation of embodied conversational agents. In IFIP Conference on Human-Computer Interaction. Springer, Springer, Heidelberg, Germany, 85–97.Google ScholarDigital Library
- Sara Kiesler, Aaron Powers, Susan R Fussell, and Cristen Torrey. 2008. Anthropomorphic interactions with a robot and robot–like agent. Social Cognition 26, 2 (2008), 169–181.Google ScholarCross Ref
- K. Kim, L. Boelling, S. Haesler, J. Bailenson, G. Bruder, and G. F. Welch. 2018. Does a Digital Assistant Need a Body? The Influence of Visual Embodiment and Social Behavior on the Perception of Intelligent Virtual Agents in AR. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE Press, NJ, USA, 105–114. https://doi.org/10.1109/ISMAR.2018.00039Google Scholar
- Kangsoo Kim, Celso M de Melo, Nahal Norouzi, Gerd Bruder, and Gregory F Welch. 2020. Reducing Task Load with an Embodied Intelligent Virtual Assistant for Improved Performance in Collaborative Decision Making. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, IEEE Press, NJ, USA, 529–538.Google ScholarCross Ref
- Bret Kinsella and Ava Mutchler. 2019. U.S. Smart Speaker Consumer Adoption Report 2019. https://voicebot.ai/smart-speaker-consumer-adoption-report-2019/Google Scholar
- Tomoko Koda and Pattie Maes. 1996. Agents with faces: The effect of personification. In Proceedings 5th IEEE International Workshop on Robot and Human Communication. RO-MAN’96 TSUKUBA. IEEE, IEEE, NJ, USA, 189–194.Google ScholarCross Ref
- Michael Lankes, Regina Bernhaupt, and Manfred Tscheligi. 2007. An Experimental Setting to Measure Contextual Perception of Embodied Conversational Agents. In Proceedings of the International Conference on Advances in Computer Entertainment Technology (Salzburg, Austria) (ACE ’07). Association for Computing Machinery, New York, NY, USA, 56–59. https://doi.org/10.1145/1255047.1255058Google ScholarDigital Library
- Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and Usability Engineering Group. Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 63–76.Google ScholarDigital Library
- Ewa Luger and Abigail Sellen. 2016. ”Like Having a Really Bad PA”: The Gulf Between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA, 5286–5297. https://doi.org/10.1145/2858036.2858288Google ScholarDigital Library
- Michal Luria, Samantha Reig, Xiang Zhi Tan, Aaron Steinfeld, Jodi Forlizzi, and John Zimmerman. 2019. Re-Embodiment and Co-Embodiment: Exploration of Social Presence for Robots and Conversational Agents. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS ’19). Association for Computing Machinery, New York, NY, USA, 633–644. https://doi.org/10.1145/3322276.3322340Google ScholarDigital Library
- Karl F MacDorman, Robert D Green, Chin-Chang Ho, and Clinton T Koch. 2009. Too real for comfort? Uncanny responses to computer generated faces. Computers in human behavior 25, 3 (2009), 695–710.Google Scholar
- Angie Lorena Marin Mejia, Doori Jo, and Sukhan Lee. 2013. Designing Robotic Avatars: Are User’s Impression Affected by Avatar’s Age?. In Proceedings of the 8th ACM/IEEE International Conference on Human-Robot Interaction (Tokyo, Japan) (HRI ’13). IEEE Press, NJ, USA, 195–196.Google ScholarCross Ref
- Richard E Mayer and C Scott DaPra. 2012. An embodiment effect in computer-based learning with animated pedagogical agents.Journal of Experimental Psychology: Applied 18, 3 (2012), 239.Google Scholar
- Rachel McDonnell, Martin Breidt, and Heinrich H Bülthoff. 2012. Render me real? Investigating the effect of render style on the perception of animated virtual humans. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1–11.Google ScholarDigital Library
- Masahiro Mori 1970. The uncanny valley. Energy 7, 4 (1970), 33–35.Google Scholar
- Masahiro Mori, Karl F MacDorman, and Norri Kageki. 2012. The uncanny valley [from the field]. IEEE Robotics & Automation Magazine 19, 2 (2012), 98–100.Google ScholarCross Ref
- Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of social issues 56, 1 (2000), 81–103.Google ScholarCross Ref
- Clifford Nass, Youngme Moon, and Nancy Green. 1997. Are machines gender neutral? Gender-stereotypic responses to computers with voices. Journal of applied social psychology 27, 10 (1997), 864–876.Google ScholarCross Ref
- Clifford Nass, Jonathan Steuer, and Ellen R. Tauber. 1994. Computers Are Social Actors. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston Massachusetts USA) (CHI ’94). Association for Computing Machinery, New York, NY, USA, 72–78. https://doi.org/10.1145/191666.191703Google ScholarDigital Library
- Catherine S Oh, Jeremy N Bailenson, and Gregory F Welch. 2018. A systematic review of social presence: Definition, antecedents, and implications. Frontiers in Robotics and AI 5 (2018), 114.Google ScholarCross Ref
- Robert Porzel and Manja Baudis. 2004. The Tao of CHI: Towards Effective Human-Computer Interaction. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004. Association for Computational Linguistics, Boston, Massachusetts, USA, 209–216. https://www.aclweb.org/anthology/N04-1027Google Scholar
- Alisha Pradhan, Kanika Mehta, and Leah Findlater. 2018. ”Accessibility Came by Accident”: Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 459, 13 pages. https://doi.org/10.1145/3173574.3174033Google ScholarDigital Library
- Lingyun Qiu and Izak Benbasat. 2009. Evaluating anthropomorphic product recommendation agents: A social relationship perspective to designing information systems. Journal of management information systems 25, 4 (2009), 145–182.Google ScholarDigital Library
- Byron Reeves and Clifford Nass. 1996. The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, New York, NY, USA.Google ScholarDigital Library
- Kristin E. Schaefer, Jessie Y. C. Chen, James L. Szalma, and P. A. Hancock. 2016. A Meta-Analysis of Factors Influencing the Development of Trust in Automation: Implications for Understanding Autonomy in Future Systems. Human Factors 58, 3 (2016), 377–400. https://doi.org/10.1177/0018720816634228 arXiv:https://doi.org/10.1177/0018720816634228PMID: 27005902.Google ScholarCross Ref
- Susanne Schmidt, Gerd Bruder, and Frank Steinicke. 2018. Effects of Embodiment on Generic and Content-Specific Intelligent Virtual Agents as Exhibition Guides.. In ICAT-EGVE. The Eurographics Association, Geneve, Switzerland, 13–20.Google Scholar
- Susanne Schmidt, Gerd Bruder, and Frank Steinicke. 2019. Effects of virtual agent and object representation on experiencing exhibited artifacts. Computers & Graphics 83(2019), 1–10.Google ScholarDigital Library
- Valentin Schwind, Katrin Wolf, and Niels Henze. 2018. Avoiding the Uncanny Valley in Virtual Character Design. Interactions 25, 5 (Aug. 2018), 45–49. https://doi.org/10.1145/3236673Google ScholarDigital Library
- Jun’ichiro Seyama and Ruth S Nagayama. 2007. The uncanny valley: Effect of realism on the impression of artificial human faces. Presence: Teleoperators and virtual environments 16, 4(2007), 337–351.Google Scholar
- Selina Jeanne Sutton. 2020. Gender Ambiguous, Not Genderless: Designing Gender in Voice User Interfaces (VUIs) with Sensitivity. In Proceedings of the 2nd Conference on Conversational User Interfaces (Bilbao, Spain) (CUI ’20). Association for Computing Machinery, New York, NY, USA, Article 11, 8 pages. https://doi.org/10.1145/3405755.3406123Google ScholarDigital Library
- Akikazu Takeuchi and Taketo Naito. 1995. Situated Facial Displays: Towards Social Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’95). ACM Press/Addison-Wesley Publishing Co., USA, 450–455. https://doi.org/10.1145/223904.223965Google Scholar
- Isaac Wang, Jesse Smith, and Jaime Ruiz. 2019. Exploring Virtual Agents for Augmented Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, Article 281, 12 pages. https://doi.org/10.1145/3290605.3300511Google ScholarDigital Library
- Jun Xiao, John Stasko, and Richard Catrambone. 2004. An Empirical Study of the Effect of Agent Competence on User Performance and Perception. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems - Volume 1 (New York, New York) (AAMAS ’04). IEEE Computer Society, USA, 178–185.Google Scholar
- Beste F. Yuksel, Penny Collisson, and Mary Czerwinski. 2017. Brains or Beauty: How to Engender Trust in User-Agent Interactions. ACM Trans. Internet Technol. 17, 1, Article 2 (Jan. 2017), 20 pages. https://doi.org/10.1145/2998572Google ScholarDigital Library
- Sean Zdenek. 2007. “Just roll your mouse over me”: Designing virtual women for customer service on the web. Technical Communication Quarterly 16, 4 (2007), 397–430.Google ScholarCross Ref
Recommendations
Voice in Human–Agent Interaction: A Survey
Social robots, conversational agents, voice assistants, and other embodied AI are increasingly a feature of everyday life. What connects these various types of intelligent agents is their ability to interact with people through voice. Voice is becoming ...
Investigating Opportunities for Active Smart Assistants to Initiate Interactions With Users
MUM '23: Proceedings of the 22nd International Conference on Mobile and Ubiquitous MultimediaPassive voice assistants such as Alexa are widespread, responding to user requests. However, due to the rise of domestic robots, we envision active smart assistants initiating interactions seamlessly, weaving themselves into the user’s context, and ...
"Let's Face It": Investigating User Preferences for Virtual Humanoid Home Assistants
HAI '23: Proceedings of the 11th International Conference on Human-Agent InteractionWhile a growing number of households contain home assistants, they mainly remain voice-only devices where the virtual agent is not represented visually. The visual representation of the agent is limited to the device's housing and abstract light ...
Comments