Ting-Hao Kenneth Huang
Joseph Chee Chang
Saiganesh Swaminathan
ABSTRACT
Crowd-powered conversational assistants have found to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. One promising direction is to combined the two approaches for high quality and low cost solutions. However, traditional offline approaches of building automated systems with the crowd requires first collecting training data from the crowd, and then training a model before an online system can be launched. In this paper, we introduce Evorus, a crowd-powered conversational assistant with online-learning capability that automate itself over time. Evorus expands a previous crowd-powered conversation system by reducing its reliance on the crowd over time while maintaining the robustness and reliability of human intelligence, by (i) allowing new chatbots to be added to help contribute possible answers, (ii) learning to reuse past responses to similar queries over time, and (iii) learning to reduce the amount of crowd oversight necessary to retain quality. Our deployment study with 28 users show that automated responses were chosen 12.84% of the time, and voting cost was reduced by 6%. Evorus introduced a new framework for constructing crowd-powered conversation systems that can gradually automate themselves using machine learning, a concept that we believe can be generalize to other types of crowd-powered systems for future research.
Supplemental Material
Available for Download
- Azaria, A., and Hong, J. Recommender system with personality. In Proceedings of the 10th ACM conference on Recommender systems, ACM (2016). Google Scholar
Digital Library
- Carpenter, R. Cleverbot, 2006. {Online; accessed 08-March-2017}.Google Scholar
- Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., and Lin, C.-J. Liblinear - a library for large linear classification, 2008. The Weka classifier works with version 1.33 of LIBLINEAR.Google Scholar
- Huang, T.-H. K., Lasecki, W. S., Azaria, A., and Bigham, J. P. 'is there anything else i can help you with'?: Challenges in deploying an on-demand crowd-powered conversational agent. In Proceedings of AAAI Conference on Human Computation and Crowdsourcing 2016 (HCOMP 2016), AAAI (2016).Google ScholarCross Ref
- Lasecki, W. S., Wesley, R., Nichols, J., Kulkarni, A., Allen, J. F., and Bigham, J. P. Chorus: A crowd-powered conversational assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST '13, ACM (New York, NY, USA, 2013), 151--162. Google ScholarDigital Library
Index Terms
- Evorus: A Crowd-powered Conversational Assistant That Automates Itself Over Time
Recommendations
Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsCrowd-powered conversational assistants have been shown to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. A promising direction is to combine the two approaches for high quality, low latency, ...
Just the Right Mood for HIT!: Analyzing the Role of Worker Moods in Conversational Microtask Crowdsourcing
Web EngineeringAbstractConversational agents are playing an increasingly important role in providing users with natural communication environments, improving outcomes in a variety of domains in human-computer interaction. Crowdsourcing marketplaces are simultaneously ...
Changes in verbal and nonverbal conversational behavior in long-term interaction
ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interactionWe present an empirical investigation of conversational behavior in dyadic interaction spanning multiple conversations, in the context of a developing interpersonal relationship between a health counselor and her clients. Using a longitudinal video ...
Comments