Skip to main content

Advertisement

Springer Nature Link
Log in

An empirical evaluation of active learning strategies for profile elicitation in a conversational recommender system

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Conversational Recommender Systems have received widespread attention in both research and practice. They assist people in finding relevant and interesting items through a multi-turn conversation. The use of natural language interaction also allows users to express their preferences with more flexibility. However, these systems often have to work in a cold-start situation, and most of the conversation is dedicated to the profile elicitation step. In order to ensure good recommendations, this profile should be as rich as possible, which requires great user effort. In this paper, we investigate the application of Active Learning techniques for improving the profile elicitation step in a Conversational Recommender System. We compared five different state-of-the-art techniques, and carried out a user study with 219 users in order to assess their effectiveness both in terms of recommendation accuracy and user effort. Results show that assisting users by providing personalized suggestions during the profile elicitation step improves the quality of the recommendations in terms of Hit Rate and nDCG, compared to a strategy that requires users to come up with preferences on their own.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. https://www.wikidata.org

  2. https://github.com/aiovine/conversational-recommender-jiis/tree/master

  3. dialogflow.com/

  4. https://stanfordnlp.github.io/CoreNLP/

  5. https://github.com/jberkel/imdb-movie-links/blob/master/top250.txt

  6. https://grouplens.org/datasets/movielens/

References

  • Adomavicius, G., & Kwon, Y. (2012). Improving aggregate recommendation diversity using ranking-based techniques. IEEE Transactions on Knowledge and Data Engineering, 24(5), 896–911. https://doi.org/10.1109/TKDE.2011.15.

    Article  Google Scholar 

  • Basile, P., Musto, C., de Gemmis, M., Lops, P., Narducci, F., & Semeraro, G. (2014). Content-based recommender systems+ DBpedia knowledge= semantics-aware recommender systems. In Semantic web evaluation challenge (pp. 163–169). Springer.

  • Bertomeu Castelló, N. (2012). Finding optimal presentation sequences for a conversational recommender system. In S. Greco, B. Bouchon-Meunier, G. Coletti, M. Fedrizzi, B. Matarazzo, & R.R. Yager (Eds.) Advances in computational intelligence. https://doi.org/10.1007/978-3-642-31724-8_34. http://link.springer.com/10.1007/978-3-642-31724-8_34. Series Title: Communications in Computer and Information Science, (Vol. 300 pp. 328–337). Berlin Heidelberg: Springer.

  • Brill, T.M., Munoz, L., & Miller, R.J. (2019). Siri, Alexa, and other digital assistants: a study of customer satisfaction with artificial intelligence applications. Journal of Marketing Management, 35(15–16), 1401–1436. https://doi.org/10.1080/0267257X.2019.1687571. https://www.tandfonline.com/doi/full/10.1080/0267257X.2019.1687571.

    Article  Google Scholar 

  • Burke, R. D., Hammond, K. J., & Yound, B. (1997). The findme approach to assisted browsing. IEEE Expert, 12(4), 32–40.

    Article  Google Scholar 

  • Carenini, G., Smith, J., & Poole, D. (2003). Towards more conversational and collaborative recommender systems. In Proceedings of the 8th international conference on intelligent user interfaces (pp. 12–18).

  • Catherine, R., & Cohen, W. (2017). Transnets: Learning to transform for recommendation. In Proceedings of the eleventh ACM conference on recommender systems (pp. 288–296).

  • Christakopoulou, K., Radlinski, F., & Hofmann, K. (2016). Towards conversational recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining - KDD ’16. https://doi.org/10.1145/2939672.2939746. http://dl.acm.org/citation.cfm?doid=2939672.2939746(pp. 815–824). San Francisco: ACM Press.

  • Chuan, C. H., & Morgan, S. (2020). Creating and evaluating chatbots as eligibility assistants for clinical trials: an active deep learning approach towards user-centered classification. ACM Transactions on Computing for Healthcare, 2(1), 1–19.

    Article  Google Scholar 

  • Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1–30. Publisher: JMLR. org.

    MathSciNet  MATH  Google Scholar 

  • Elahi, M., Ricci, F., & Rubens, N. (2014). Active learning strategies for rating elicitation in collaborative filtering: A system-wide perspective. ACM Transactions on Intelligent Systems and Technology (TIST), 5(1), 1–33. Publisher: ACM New York, NY, USA.

    Google Scholar 

  • Elahi, M., Ricci, F., & Rubens, N. (2016). A survey of active learning in collaborative filtering recommender systems. Computer Science Review, 20, 29–50. Publisher: Elsevier.

    Article  MathSciNet  Google Scholar 

  • Følstad, A., & Brandtzaeg, P.B. (2020). Users’ experiences with chatbots: findings from a questionnaire study. Quality and User Experience, 5(1), 1–14.

    Article  Google Scholar 

  • Ge, M., Delgado-Battenfeld, C., & Jannach, D. (2010). Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the fourth ACM conference on Recommender systems - RecSys ’10, p 257. https://doi.org/10.1145/1864708.1864761. http://portal.acm.org/citation.cfm?doid=1864708.1864761. Barcelona: ACM Press.

  • Gini, C. (1921). Meausrement of inequality and incomes. The Economic Journal, 31, 124–126.

    Article  Google Scholar 

  • Glass, J., Polifroni, J., Seneff, S., & Zue, V. (2000). Data collection and performance evaluation of spoken dialogue systems: The MIT experience. In Sixth international conference on spoken language processing.

  • Goker, M., & Thompson, C. (2000). The adaptive place advisor: a conversational recommendation system. In Proceedings of the 8th German workshop on case based reasoning (pp. 187–198). Citeseer.

  • Golbandi, N., Koren, Y., & Lempel, R. (2010). On bootstrapping recommender systems. In Proceedings of the 19th ACM international conference on Information and knowledge management (pp. 1805–1808).

  • Golbandi, N., Koren, Y., & Lempel, R. (2011). Adaptive bootstrapping of recommender systems using decision trees. In Proceedings of the fourth ACM international conference on Web search and data mining - WSDM ’11, p 595. https://doi.org/10.1145/1935826.1935910. http://portal.acm.org/citation.cfm?doid=1935826.1935910. Hong Kong: ACM Press.

  • Greco, C., Suglia, A., Basile, P., & Semeraro, G. (2017). Converse-et-impera: Exploiting deep learning and hierarchical reinforcement learning for conversational recommender systems. In Conference of the italian association for artificial intelligence (pp. 372–386). Springer.

  • Hammond, K. J., Burke, R. D., & Lytinen, S. L. (1995). A case-based approach to knowledge navigation. In IJCAI (pp. 2071–2072).

  • Haveliwala, T.H. (2003). Topic-sensitive pagerank: A context-sensitive ranking algorithm for web search. IEEE Transactions on Knowledge and Data Engineering, 15(4), 784–796. Publisher: IEEE.

    Article  Google Scholar 

  • Hernández-Rubio, M., Bellogín, A., & Cantador, I. (2020). Aspect-based active learning for user preference elicitation in recommender systems. In I. Cantador, M. Chevalier, M. Melucci, & J. Mothe (Eds.) Proceedings of the joint conference of the information retrieval communities in Europe (CIRCLE 2020), Samatan, Gers, France, July 6-9, 2020, CEUR Workshop Proceedings vol. 2621. CEUR-WS.org. http://ceur-ws.org/Vol-2621/CIRCLE20_16.pdf.

  • Iovine, A., Lops, P., Narducci, F., de Gemmis, M., & Semeraro, G. (2021). Improving preference elicitation in a conversational recommender system with active learning strategies. In Proceedings of the 36th Annual ACM symposium on applied computing (pp. 1375–1382).

  • Iovine, A., Narducci, F., & Semeraro, G. (2020). Conversational Recommender Systems and natural language: A study through the ConveRSE framework. Decision Support Systems p. 113250. https://doi.org/10.1016/j.dss.2020.113250. http://www.sciencedirect.com/science/article/pii/S0167923620300051.

  • Jannach, D., & Kreutler, G. (2007). Rapid development of knowledge-based conversational recommender applications with advisor suite. J. Web Eng., 6(2), 165–192.

    Google Scholar 

  • Jannach, D., Manzoor, A., Cai, W., & Chen, L. (2020). A survey on conversational recommender systems. arXiv:2004.00646.

  • Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422–446. Publisher: ACM New York, NY, USA.

    Article  Google Scholar 

  • Kaminskas, M., & Bridge, D. (2017). Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems, 7(1), 1–42. https://doi.org/10.1145/2926720. https://dl.acm.org/doi/10.1145/2926720.

    Article  Google Scholar 

  • Kang, J., Condiff, K., Chang, S., Konstan, J.A., Terveen, L., & Harper, F.M. (2017). Understanding how people use natural language to ask for recommendations. In Proceedings of the Eleventh ACM Conference on Recommender Systems - RecSys ’17. https://doi.org/10.1145/3109859.3109873. http://dl.acm.org/citation.cfm?doid=3109859.3109873 (pp. 229–237). Como: ACM Press.

  • Kiseleva, J., Williams, K., Jiang, J., Hassan Awadallah, A., Crook, A. C., Zitouni, I., & Anastasakos, T. (2016). Understanding user satisfaction with intelligent assistants. In Proceedings of the 2016 ACM on conference on human information interaction and retrieval (pp. 121–130).

  • Koren, Y. (2008). Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 426–434).

  • Lee, S.L. (2010). Commodity recommendations of retail business based on decisiontree induction. Expert Systems with Applications, 37(5), 3685–3694. Publisher: Elsevier.

    Article  Google Scholar 

  • Mahmood, T., & Ricci, F. (2009). Improving recommender systems with adaptive conversational strategies. In Proceedings of the 20th ACM conference on Hypertext and hypermedia - HT ’09. https://doi.org/10.1145/1557914.1557930. http://portal.acm.org/citation.cfm?doid=1557914.1557930 (p. 73). Torino: ACM Press.

  • Merialdo, A.K.B. (2001). Improving collaborative filtering for new-users by smart object selection.

  • Musto, C., Lops, P., de Gemmis, M., & Semeraro, G. (2021). Context-aware graph-based recommendations exploiting personalized pagerank. Knowledge-Based Systems, 106806, 216.

    Google Scholar 

  • Narducci, F., de Gemmis, M., Lops, P., & Semeraro, G. (2018). Improving the user experience with a conversational recommender system. In International conference of the italian association for artificial intelligence (pp. 528–538). Springer.

  • Narducci, F., de Gemmis, M., Lops, P., & Semeraro, G. (2018). Improving the user experience with a conversational recommender system. In C. Ghidini, B. Magnini, A. Passerini, & P. Traverso (Eds.) AI*IA 2018 - Advances in Artificial Intelligence - XVIIth International Conference of the Italian Association for Artificial Intelligence, Trento, Italy, November 20-23, 2018, Proceedings, Lecture Notes in Computer Science, (Vol. 11298 pp. 528–538). Springer.

  • Nickel, M., Rosasco, L., Poggio, T. A., & et al. (2016). Holographic embeddings of knowledge graphs. In The thirtieth AAAI conference on artificial intelligence (AAAI-16) (pp. 1955–1961).

  • Parapar, J., & Radlinski, F. (2021). Diverse user preference elicitation with multi-armed bandits. In Proceedings of the 14th ACM international conference on web search and data mining (pp. 130–138).

  • Priyogi, B. (2019). Preference elicitation strategy for conversational recommender system. In Proceedings of the twelfth ACM international conference on web search and data mining (pp. 824–825).

  • Pu, P., Chen, L., & Hu, R. (2011). A user-centric evaluation framework for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems - RecSys ’11. https://doi.org/10.1145/2043932.2043962. http://dl.acm.org/citation.cfm?doid=2043932.2043962 (p. 157). Chicago: ACM Press.

  • Rashid, A. M., Albert, I., Cosley, D., Lam, S. K., McNee, S. M., Konstan, J. A., & Riedl, J. (2002). Getting to know you: learning new user preferences in recommender systems. In Proceedings of the 7th international conference on intelligent user interfaces (pp. 127–134).

  • Rashid, A.M., Karypis, G., & Riedl, J. (2008). Learning preferences of new users in recommender systems: an information theoretic approach. Acm Sigkdd Explorations Newsletter, 10(2), 90–100. Publisher: ACM New York, NY, USA.

    Article  Google Scholar 

  • Real, R., & Vargas, J.M. (1996). The probabilistic basis of Jaccard’s index of similarity. Systematic Biology, 45(3), 380–385. Publisher: JSTOR.

    Article  Google Scholar 

  • Rubens, N., & Sugiyama, M. (2007). Influence-based collaborative active learning. In Proceedings of the 2007 ACM conference on Recommender systems (pp. 145–148).

  • Schröder, G., Thiele, M., & Lehner, W. (2011). Setting goals and choosing metrics for recommender system evaluations. In UCERSTI2 Workshop at the 5th ACM conference on recommender systems, chicago, USA, (Vol. 23 p. 53).

  • Sun, Y., & Zhang, Y. (2018). Conversational recommender system. arXiv:1806.03277[cs].

  • Tong, S. (2001). Active learning: theory and applications, vol. 1 Stanford University USA.

  • Wang, R., Ma, X., Jiang, C., Ye, Y., & Zhang, Y. (2020). Heterogeneous information network-based music recommendation system in mobile networks. Computer Communications, 150, 429–437.

    Article  Google Scholar 

  • Weld, H., Huang, X., Long, S., Poon, J., & Han, S. C. (2021). A survey of joint intent detection and slot-filling models in natural language understanding. arXiv:2101.08091.

  • Williams, J., Raux, A., & Henderson, M. (2016). The dialog state tracking challenge series: a review. Dialogue & Discourse, 7(3), 4–33. http://dad.uni-bielefeld.de/index.php/dad/article/view/3685.

    Article  Google Scholar 

  • Xu, K., Yang, J., Xu, J., Gao, S., Guo, J., & Wen, J.R. (2021). Adapting user preference to online feedback in multi-round conversational recommendation. In Proceedings of the 14th ACM international conference on web search and data mining (pp. 364–372).

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. University of Bari Aldo Moro, Bari, Italy

    Andrea Iovine, Pasquale Lops, Marco de Gemmis & Giovanni Semeraro

  2. Politecnico di Bari, Bari, Italy

    Fedelucio Narducci

Authors
  1. Andrea Iovine
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Pasquale Lops
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Fedelucio Narducci
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Marco de Gemmis
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Giovanni Semeraro
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Andrea Iovine.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iovine, A., Lops, P., Narducci, F. et al. An empirical evaluation of active learning strategies for profile elicitation in a conversational recommender system. J Intell Inf Syst 58, 337–362 (2022). https://doi.org/10.1007/s10844-021-00683-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-021-00683-4

Keywords