skip to main content
10.1145/3340631.3394856acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
research-article
Best Student Paper

Predicting User Intents and Satisfaction with Dialogue-based Conversational Recommendations

Published:13 July 2020Publication History

ABSTRACT

To develop a multi-turn dialogue-based conversational recommender system (DCRS), it is important to predict users' intents behind their utterances and their satisfaction with the recommendation, so as to allow the system to incrementally refine user preference model and adjust its dialogue strategy. However, little work has investigated these issues so far. In this paper, we first contribute with two hierarchical taxonomies for classifying user intents and recommender actions respectively based on grounded theory. We then define various categories of feature considering content, discourse, sentiment, and context to predict users' intents and satisfaction by comparing different machine learning methods. The experimental results for user intent prediction task show that some models (such as XGBoost and SVM) can perform well in predicting user intents, and incorporating context features into the prediction model can significantly boost the performance. Our empirical study also demonstrates that leveraging dialogue behavior features (i.e., including both user intents and recommender actions) can achieve good results in predicting user satisfaction.

Skip Supplemental Material Section

Supplemental Material

3340631.3394856.mp4

Supplemental Video

mp4

46.7 MB

References

  1. Charu C. Aggarwal and ChengXiang Zhai. 2012. Mining Text Data .Springer Science & Business Media.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. A. Bhargava, A. Celikyilmaz, D. Hakkani-Tür, and R. Sarikaya. 2013. Easy Contextual Intent Prediction and Slot Detection. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '13). 8337--8341.Google ScholarGoogle Scholar
  3. Sumit Bhatia and Prasenjit Mitra. 2012. Classifying User Messages For Managing Web Forum Data. In WebDB. 13--18.Google ScholarGoogle Scholar
  4. Andrei Broder. 2002. A Taxonomy of Web Search. SIGIR Forum, Vol. 36, 2 (2002), 3--10. http://doi.acm.org/10.1145/792550.792552Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Wanling Cai and Li Chen. 2019. Towards a Taxonomy of User Feedback Intents for Conversational Recommendations. In Proceedings of ACM RecSys 2019 Late-breaking Results co-located with the 13th ACM Conference on Recommender Systems.Google ScholarGoogle Scholar
  6. Joyce Yue Chai, Malgorzata Budzikowska, Veronika Horvath, Nicolas Nicolov, Nanda Kambhatla, and Wlodek Zadrozny. 2001. Natural Language Sales Assistant - A Web-Based Dialog System for Online Sales. In Proceedings of the Thirteenth Conference on Innovative Applications of Artificial Intelligence Conference (IAAI '01). 19--26. http://dl.acm.org/citation.cfm?id=645453.653001Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A Survey on Dialogue Systems: Recent Advances and New Frontiers. ACM SIGKDD Explorations Newsletter, Vol. 19, 2 (2017), 25--35. http://doi.acm.org/10.1145/3166054.3166058Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Li Chen and Pearl Pu. 2006. Evaluating Critiquing-based Recommender Agents. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 1 (AAAI '06). 157--162. http://dl.acm.org/citation.cfm?id=1597538.1597564Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Li Chen and Pearl Pu. 2012. Critiquing-based Recommenders: Survey and Emerging Trends. User Modeling and User-Adapted Interaction, Vol. 22, 1--2 (2012), 125--150. http://dx.doi.org/10.1007/s11257-011--9108--6Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Konstantina Christakopoulou, Alex Beutel, Rui Li, Sagar Jain, and Ed H. Chi. 2018. Q&R: A Two-Stage Approach Toward Interactive Recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '18). 139--148. http://doi.acm.org/10.1145/3219819.3219894Google ScholarGoogle Scholar
  11. Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards Conversational Recommender Systems. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). 815--824. http://doi.acm.org/10.1145/2939672.2939746Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, Vol. 20, 1 (1960), 37--46.Google ScholarGoogle ScholarCross RefCross Ref
  13. Jane F Dye, Irene M Schatz, Brian A Rosenberg, and Susanne T Coleman. 2000. Constant Comparison Method: A Kaleidoscope of Data. The Qualitative Report, Vol. 4, 1 (2000), 1--10.Google ScholarGoogle Scholar
  14. Klaus-Peter Engelbrech, Florian Gödde, Felix Hartard, Hamed Ketabdar, and Sebastian Möller. 2009. Modeling User Satisfaction with Hidden Markov Model. In Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL '09). 170--177. http://dl.acm.org/citation.cfm?id=1708376.1708402Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Elena V. Epure, Dario Compagno, Camille Salinesi, Rebecca Deneckere, Marko Bajec, and Slavko vZ itnik. 2018. Process Models of Interrelated Speech Intentions from Online Health-Related Conversations. Artificial Intelligence in Medicine, Vol. 91 (2018), 23--38.Google ScholarGoogle ScholarCross RefCross Ref
  16. Barney G. Glaser. 1998. Doing Grounded Theory: Issues and Discussions .Sociology Press.Google ScholarGoogle Scholar
  17. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. The MIT Press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Jonathan Grudin and Richard Jacques. 2019. Chatbots, Humbots, and the Quest for Artificial General Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). 1--11. https://doi.org/10.1145/3290605.3300439Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Seyyed Hadi Hashemi, Kyle Williams, Ahmed El Kholy, Imed Zitouni, and Paul A. Crook. 2018. Measuring User Satisfaction on Smart Speaker Intelligent Assistants Using Intent Sensitive Query Embeddings. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18). 1183--1192. http://doi.acm.org/10.1145/3269206.3271802Google ScholarGoogle Scholar
  20. Ryuichiro Higashinaka, Yasuhiro Minami, Kohji Dohsaka, and Toyomi Meguro. 2010. Modeling User Satisfaction Transitions in Dialogues from Overall Ratings. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL '10). 18--27. http://dl.acm.org/citation.cfm?id=1944506.1944510Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. C.J. Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In International AAAI Conference on Web and Social Media (ICWSM '14). https://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/view/8109Google ScholarGoogle Scholar
  22. Jie Kang, Kyle Condiff, Shuo Chang, Joseph A. Konstan, Loren Terveen, and F. Maxwell Harper. 2017. Understanding How People Use Natural Language to Ask for Recommendations. In Proceedings of the Eleventh ACM Conference on Recommender Systems (RecSys '17). 229--237. http://doi.acm.org/10.1145/3109859.3109873Google ScholarGoogle Scholar
  23. Tsuneo Kato, Atsushi Nagai, Naoki Noda, Ryosuke Sumitomo, Jianming Wu, and Seiichi Yamamoto. 2017. Utterance Intent Classification of a Spoken Dialogue System with Efficiently Untied Recursive Autoencoders. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. 60--64. https://www.aclweb.org/anthology/W17--5508Google ScholarGoogle ScholarCross RefCross Ref
  24. Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Foundations and Trends in Information Retrieval, Vol. 3, 1--2 (2009), 1--224. https://doi.org/10.1561/1500000012Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP '14). 1746--1751. https://www.aclweb.org/anthology/D14--1181Google ScholarGoogle ScholarCross RefCross Ref
  26. Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards Deep Conversational Recommendations. In Advances in Neural Information Processing Systems 32. 9748--9758. http://papers.nips.cc/paper/8180-towards-deep-conversational-recommendations.pdfGoogle ScholarGoogle Scholar
  27. Bing Liu, Minqing Hu, and Junsheng Cheng. 2005. Opinion Observer: Analyzing and Comparing Opinions on the Web. In Proceedings of the 14th International Conference on World Wide Web (WWW '05). 342--351. https://doi.org/10.1145/1060745.1060797Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Chunxi Liu, Puyang Xu, and Ruhi Sarikaya. 2015. Deep Contextual Language Understanding in Spoken Dialogue Systems. In Sixteenth Annual Conference of the International Speech Communication Association.Google ScholarGoogle Scholar
  29. Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using Context Information for Dialog Act Classification in DNN Framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP '17). 2170--2178. https://www.aclweb.org/anthology/D17--1231Google ScholarGoogle ScholarCross RefCross Ref
  30. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press.Google ScholarGoogle Scholar
  31. Mary L McHugh. 2012. Interrater Reliability: the Kappa Statistic. Biochemia Medica, Vol. 22, 3 (2012), 276--282.Google ScholarGoogle ScholarCross RefCross Ref
  32. Rishabh Mehrotra, Imed Zitouni, Ahmed Hassan Awadallah, Ahmed El Kholy, and Madian Khabsa. 2017. User Interaction Sequences for Search Satisfaction Prediction. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). 165--174. http://doi.acm.org/10.1145/3077136.3080833Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Lian Meng and Minlie Huang. 2018. Dialogue Intent Classification with Long Short-Term Memory Networks. In National CCF Conference on Natural Language Processing and Chinese Computing (NLPCC '18). Springer, 42--50.Google ScholarGoogle Scholar
  34. Andrew Olney, Max Louwerse, Eric Matthews, Johanna Marineau, Heather Hite-Mitchell, and Arthur Graesser. 2003. Utterance Classification in AutoTutor. In Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications Using Natural Language Processing - Volume 2 (HLT-NAACL-EDUC '03). 1--8. https://doi.org/10.3115/1118894.1118895Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP '14). 1532--1543. https://www.aclweb.org/anthology/D14--1162Google ScholarGoogle ScholarCross RefCross Ref
  36. Bilih Priyogi. 2019. Preference Elicitation Strategy for Conversational Recommender System. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (WSDM '19). 824--825. http://doi.acm.org/10.1145/3289600.3291604Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Chen Qu, Liu Yang, W. Bruce Croft, Yongfeng Zhang, Johanne R. Trippas, and Minghui Qiu. 2019. User Intent Prediction in Information-seeking Conversations. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (CHIIR '19). 25--33. http://doi.acm.org/10.1145/3295750.3298924Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Dimitrios Rafailidis and Yannis Manolopoulos. 2019. The Technological Gap Between Virtual Assistants and Recommendation Systems. CoRR, Vol. abs/1901.00431 (2019). http://arxiv.org/abs/1901.00431Google ScholarGoogle Scholar
  39. Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2011. Classifier Chains for Multi-label Classification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. 254--269.Google ScholarGoogle Scholar
  40. Daniel E. Rose and Danny Levinson. 2004. Understanding User Goals in Web Search. In Proceedings of the 13th International Conference on World Wide Web (WWW '04). 13--19. http://doi.acm.org/10.1145/988672.988675Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Hideo Shimazu. 2001. ExpertClerk: Navigating Shoppers' Buying Process with the Combination of Asking and Proposing. In Proceedings of the 17th International Joint Conference on Artificial Intelligence - Volume 2 (IJCAI '01). 1443--1448. http://dl.acm.org/citation.cfm?id=1642194.1642287Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Mohammad S. Sorower. 2010. A Literature Survey on Algorithms for Multi-label Learning. Oregon State University, Corvallis, Vol. 18 (2010), 1--25.Google ScholarGoogle Scholar
  43. Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, Vol. 26, 3 (2000), 339--373. https://doi.org/10.1162/089120100561737Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Ming Sun, Yun-Nung Chen, and Alexander I. Rudnicky. 2015. Understanding User's Cross-domain Intentions in Spoken Dialog Systems. In NIPS workshop on Machine Learning for SLU and Interaction (NIPS-SLU).Google ScholarGoogle Scholar
  45. Yueming Sun and Yi Zhang. 2018. Conversational Recommender System. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18). 235--244. http://doi.acm.org/10.1145/3209978.3210002Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Dinoj Surendran and Gina-Anne Levow. [n.d.]. Dialog Act Tagging with Support Vector Machines and Hidden Markov Models. In Ninth International Conference on Spoken Language Processing (ICSLP-Interspeech '06).Google ScholarGoogle Scholar
  47. Cynthia A. Thompson, Mehmet H. Göker, and Pat Langley. 2004. A Personalized System for Conversational Recommendations. Journal of Artificial Intelligence Research, Vol. 21, 1 (2004), 393--428. http://dl.acm.org/citation.cfm?id=1622467.1622479Google ScholarGoogle ScholarCross RefCross Ref
  48. Kyle Williams and Imed Zitouni. 2017. Does That Mean You'Re Happy?: RNN-based Modeling of User Interaction Sequences to Detect Good Abandonment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM '17). 727--736. http://doi.acm.org/10.1145/3132847.3133035Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017. Building Task-oriented Dialogue Systems for Online Shopping. In Thirty-First AAAI Conference on Artificial Intelligence (AAAI '17). https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14261Google ScholarGoogle Scholar
  50. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level Convolutional Networks for Text Classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 (NIPS '15). 649--657. http://dl.acm.org/citation.cfm?id=2969239.2969312Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. 2018. Towards Conversational Search and Recommendation: System Ask, User Respond. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18). 177--186. http://doi.acm.org/10.1145/3269206.3271776Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Predicting User Intents and Satisfaction with Dialogue-based Conversational Recommendations

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in
        • Published in

          cover image ACM Conferences
          UMAP '20: Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization
          July 2020
          426 pages
          ISBN:9781450368612
          DOI:10.1145/3340631

          Copyright © 2020 ACM

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 13 July 2020

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article

          Acceptance Rates

          Overall Acceptance Rate162of633submissions,26%

          Upcoming Conference

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader