Breck Baldwin
ABSTRACT
Evaluating large-scale customer-facing voice interfaces involves a variety of challenges, such as data privacy, fairness or unintended bias, and the cost of human labor. Comcast's Xfinity Voice Remote is one such voice interface aimed at users looking to discover content on their TVs. The artificial intelligence (AI) behind the voice remote currently powers multiple voice interfaces, serving tens of millions of requests every day, from users across the globe.In this talk, we introduce a novel Round-Trip system we have built to evaluate the AI serving these voice interfaces in a semi-automated manner, providing a robust and cheap alternative to traditional quality assurance methods. We discuss five specific challenges we have encountered in Round-Trip and describe our solutions in detail.
Supplemental Material
Index Terms
- Simulating Humans at Scale to Evaluate Voice Interfaces for TVs: the Round-Trip System at Comcast
Recommendations
Voice-based Conversational UX Studies and Design
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing SystemsVoice User Interfaces are becoming ubiquitously available, providing unprecedented opportunities to advance our understanding of voice interaction in a burgeoning array of practices and settings. We invite participants to contribute work-in-progress in ...
Design and Evaluation of Voice User Interfaces: What Should One Consider?
Design, Operation and Evaluation of Mobile CommunicationsAbstractVoice user interfaces (VUI) come in various forms of software or hardware, are controlled by voice, and can help the user in their daily life. Despite VUIs being readily available on smartphones, they have a low adoption rate. This can be ...
Towards a human-computer interaction model for voice user interfaces
CLIHC '19: Proceedings of the IX Latin American Conference on Human Computer InteractionThe user interaction with computer systems has evolved over the years, from Command Line Interfaces (CLIs), Graphics User Interfaces (GUIs), Natural User Interfaces (NUIs) and actually Voice User Interfaces (VUIs). The use of VUIs is increasingly common,...
Comments