Paolo Cremonesi
Skip Abstract Section
Abstract
Recommender Systems (RSs) help users search large amounts of digital contents and services by allowing them to identify the items that are likely to be more attractive or useful. RSs play an important persuasion role, as they can potentially augment the users’ trust towards in an application and orient their decisions or actions towards specific directions. This article explores the persuasiveness of RSs, presenting two vast empirical studies that address a number of research questions.
First, we investigate if a design property of RSs, defined by the statistically measured quality of algorithms, is a reliable predictor of their potential for persuasion. This factor is measured in terms of perceived quality, defined by the overall satisfaction, as well as by how users judge the accuracy and novelty of recommendations. For our purposes, we designed an empirical study involving 210 subjects and implemented seven full-sized versions of a commercial RS, each one using the same interface and dataset (a subset of Netflix), but each with a different recommender algorithm. In each experimental configuration we computed the statistical quality (recall and F-measures) and collected data regarding the quality perceived by 30 users. The results show us that algorithmic attributes are less crucial than we might expect in determining the user’s perception of an RS’s quality, and suggest that the user’s judgment and attitude towards a recommender are likely to be more affected by factors related to the user experience.
Second, we explore the persuasiveness of RSs in the context of large interactive TV services. We report a study aimed at assessing whether measurable persuasion effects (e.g., changes of shopping behavior) can be achieved through the introduction of a recommender. Our data, collected for more than one year, allow us to conclude that, (1) the adoption of an RS can affect both the lift factor and the conversion rate, determining an increased volume of sales and influencing the user’s decision to actually buy one of the recommended products, (2) the introduction of an RS tends to diversify purchases and orient users towards less obvious choices (the long tail), and (3) the perceived novelty of recommendations is likely to be more influential than their perceived accuracy.
Overall, the results of these studies improve our understanding of the persuasion phenomena induced by RSs, and have implications that can be of interest to academic scholars, designers, and adopters of this class of systems.
- Adomavicius, G. and Tuzhilin, A. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 6, 734--749. Google Scholar
Digital Library
- An, C. and Fromm, H. 2005. Supply Chain Management On Demand: Strategies, Technologies, Applications. Springer. Google ScholarDigital Library
- Anderson, C. 2006. The Long Tail: Why the Future of Business Is Selling Less of More. Hyperion. Google ScholarDigital Library
- Baillie, R. T. 1980. Predictions from armax models. J. Econ. 12, 3, 365--374.Google ScholarCross Ref
- Balabanović, M. and Shoham, Y. 1997. Fab: Content-based, collaborative recommendation. Comm. ACM 40, 3, 66--72. Google ScholarDigital Library
- Bambini, R., Cremonesi, P., and Turrin, R. 2011. A Recommender System for an IPTV Service Provider: A Real Large-Scale Production Environment. Springer, 299--331.Google Scholar
- Bell, R. M. and Koren, Y. 2007. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proceedings of the 7th IEEE International Conference on Data Mining (ICDM). 43--52. Google ScholarDigital Library
- Bellogín, A., Castells, P., and Cantador, I. 2011. Precision-oriented evaluation of recommender systems: An algorithmic comparison. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys). 1--4 Google ScholarDigital Library
- Bennett, J. and Lanning, S. 2007. The Netflix Prize. In Proceedings of KDD Cup and Workshop. 3--6.Google Scholar
- Berry, M. W. 1992. Large-scale sparse singular value computations. Int. J. Supercomput. Appl. 6, 1, 13--49.Google ScholarDigital Library
- Bollen, D., Knijnenburg, B. P., Willemsen, M. C., and Graus, M. 2010. Understanding choice overload in recommender systems. In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys). 63--70. Google ScholarDigital Library
- Burke, R. 2002. Hybrid recommender systems: Survey and experiments. User Model. User-Adapted Interact. 12, 331--370. Google ScholarDigital Library
- Burke, R. 2007. The Adaptive Web. Springer-Verlag, Berlin, 377--408.Google Scholar
- Celma, Ò. 2010. Music Recommendation and Discovery - The Long Tail, Long Fail, and Long Play in the Digital Music Space. Springer. Google ScholarDigital Library
- Celma, O. and Cano, P. 2008. From hits to niches? Or how popular artists can bias music recommendation and discovery. In Proceedings of the 2nd KDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition. Google ScholarDigital Library
- Celma, O. and Herrera, P. 2008. A new approach to evaluating novel recommendations. In Proceedings of the ACM Conference on Recommender Systems (RecSys). 179--186. Google ScholarDigital Library
- Chen, L. and Pu, P. 2008. A cross-cultural user evaluation of product recommender interfaces. In Proceedings of the ACM Conference on Recommender Systems. (RecSys). 75--82. Google ScholarDigital Library
- Chen, L. and Pu, P. 2010. A user-centric evaluation framework of recommender systems. In Proceedings of the ACM Conference on Recommender Systems (RecSys), Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI). 14--21.Google Scholar
- Cialdini, R. 2001. Influence: Science and Practice. Allyn and Bacon.Google Scholar
- Cramer, H., Evers, V., Ramlal, S., Someren, M., Rutledge, L., Stash, N., Aroyo, L., and Wielinga, B. 2008. The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User-Adapted Interact 18, 455--496. Google ScholarDigital Library
- Cremonesi, P., Lentini, E., Matteucci, M., and Turrin, R. 2008. An evaluation methodology for recommender systems. In Proceedings of the 4th International Conference on Automated Solutions for Cross Media Content and Multi-Channel Distribution (AXMEDIS). 224--231. Google ScholarDigital Library
- Cremonesi, P., Koren, Y., and Turrin, R. 2010. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys). 39--46. Google ScholarDigital Library
- Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A., and Turrin, R. 2011a. Comparative evaluation of recommender system quality. In Proceedings of the Annual Conference Extended Abstracts on Human Factors In Computing Systems (CHI EA). 1927--1932. Google ScholarDigital Library
- Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A., and Turrin, R. 2011b. Looking for good recommendations: A comparative evaluation of recommender systems. In Proceedings of the 13th IFIP TC13 International Conference on Human Computer Interaction (INTERACT). Springer. Google ScholarDigital Library
- De Angeli, A., Sutcliffe, A., and Hartmann, J. 2006. Interaction, usability and aesthetics: What influences users’ preferences? In Proceedings of the 6th Conference on Designing Interactive Systems (DIS). 271--280. Google ScholarDigital Library
- Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., and Harshman, R. A. 1990. Indexing by latent semantic analysis. J. Amer. Soc. Inform. Sci. 41, 6, 391--407.Google ScholarCross Ref
- De Gemmis, M., Lops, P., and Semeraro, G. 2007. A content-collaborative recommender that exploits Wordnet-based user profiles for neighborhood formation. User Model. User-Adapt. Interact. 17, 217--255. Google ScholarDigital Library
- Deshpande, M. and Karypis, G. 2004. Item-based top-n recommendation algorithms. ACM Trans. Inform. Syst. 22, 1, 143--177. Google ScholarDigital Library
- Desrosiers, C. and Karypis, G. 2011. A comprehensive survey of neighborhood-based recommendation methods. In Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor Eds., Springer, 107--144.Google Scholar
- Felfernig, A., Friedrich, G., Gula, B., Hitz, M., Kruggel, T., Leitner, G., Melcher, R., Riepan, D., Strauss, S., Teppan, E., and Vitouch, O. 2007. Persuasive recommendation: Serial position effects in knowledge-based recommender systems. In Proceedings of the 2nd International Conference On Persuasive Technology (PERSUASIVE). Springer-Verlag, Berlin, 283--294. Google ScholarDigital Library
- Felfernig, A., Gula, B., Leitner, G., Maier, M., Melcher, R., and Teppan, E. 2008. Persuasion in knowledge-based recommendation. In Proceedings of the 3rd International Conference on Persuasive Technology (PERSUASIVE). Springer-Verlag, Berlin, 71--82. Google ScholarDigital Library
- Fleder, D. and Hosanagar, K. 2009. Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity. Manage. Sci. 55, 697--712. Google ScholarDigital Library
- Fogg, B. 2003. Persuasive Technology: Using Computers to Change What We Think and Do. Series in Interactive Technologies. Morgan Kaufmann Publishers. Google ScholarDigital Library
- Golbandi, N., Koren, Y., and Lempel, R. 2010. On bootstrapping recommender systems. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM). 1805--1808. Google ScholarDigital Library
- Golbandi, N., Koren, Y., and Lempel, R. 2011. Adaptive bootstrapping of recommender systems using decision trees. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM). 595--604. Google ScholarDigital Library
- Gretzel, U. and Fesenmaier, D. 2005. Persuasiveness of preference elicitation processes in destination recommendation systems. In Information and Communication Technologies in Tourism 2005, A. J. Frew Ed., Springer, 194--204.Google Scholar
- Gretzel, U. and Fesenmaier, D. 2006. Persuasion in recommender systems. Int. J. Electron. Commerce 11, 81--100. Google ScholarDigital Library
- Hand, S. and Varan, D. 2008. Interactive narratives: Exploring the links between empathy, interactivity and structure. In Proceedings of the 6th European Conference on Changing Television Environments (EUROITV). Google ScholarDigital Library
- Hartmann, J., De Angeli, A., and Sutcliffe, A. 2008. Framing the user experience: information biases on website quality judgement. In Proceedings of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems. (CHI). 855--864. Google ScholarDigital Library
- Herlocker, J., Konstan, J., Terveen, L., and Riedl, J. 2004. Evaluating collaborative filtering recommender systems. ACM Trans. Inform. Syst. 22, 1, 5--53. Google ScholarDigital Library
- Hu, R. and Pu, P. 2009. Acceptance issues of personality-based recommender systems. In Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys). 221--224. Google ScholarDigital Library
- Husbands, P., Simon, H., and Ding, C. 2000. On the Use of the Singular Value Decomposition for Text Retrieval. SIAM, Philadelphia, PA, 145--156.Google Scholar
- Jensen, J. 2008. Interactive television - a brief media history. In Proceedings of the 6th European Conference on Changing Television Environments (EUROITV). 1--10. Google ScholarDigital Library
- Jones, N. and Pu, P. 2007. User technology adoption issues in recommender systems. In Proceedings of the Networking and Electronic Conference (NAEC). 379--394.Google Scholar
- Kelly, D. and Teevan, J. 2003. Implicit feedback for inferring user preference: A bibliography. SIGIR Forum 37, 2, 18--28. Google ScholarDigital Library
- Koren, Y. 2008. Factorization meets the neighborhood: A multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 426--434. Google ScholarDigital Library
- Koren, Y. 2009. Collaborative filtering with temporal dynamics. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 447--456. Google ScholarDigital Library
- Koren, Y. 2010. Factor in the neighbors: Scalable and accurate collaborative filtering. ACM Trans. Knowl. Discov. Data 4, 1, 1--24. Google ScholarDigital Library
- Kramer, T. 2007. The effect of measurement task transparency on preference construction and evaluations of personalized recommendations. J. Market. Res. 44, 2, 224--233.Google ScholarCross Ref
- Lekakos, G. and Giaglis, G. M. 2007. A hybrid approach for improving predictive accuracy of collaborative filtering algorithms. User Model. User-Adapted Interact. 17, 5--40. Google ScholarDigital Library
- Lewis, D. D. and Gale, W. A. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 3--12. Google ScholarDigital Library
- Lops, P., De Gemmis, M., and Semeraro, G. 2011. Content-based recommender systems: State of the art and trends. In Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor Eds., Springer, 73--105.Google Scholar
- Magnini, B. and Strapparava, C. 2001. Improving user modelling with content-based techniques. In Proceedings of the 8th International Conference on User Modeling (UM). 74--83. Google ScholarDigital Library
- McNee, S., Lam, S., Konstan, J., and Riedl, J. 2003. Interfaces for eliciting new user preferences in recommender systems. In Proceedings of the 9th International Conference on User Modeling (UM). 178--188. Google ScholarDigital Library
- McNee, S. M., Riedl, J., and Konstan, J. A. 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In Proceedings of the Annual Conference Extended Abstracts On Human Factors In Computing Systems. (CHI EA). 1097--1101. Google ScholarDigital Library
- Middleton, S. E., Shadbolt, N. R., and De Roure, D. C. 2004. Ontological user profiling in recommender systems. ACM Trans. Inform. Syst. 22, 54--88. Google ScholarDigital Library
- Mihalcea, R. and Csomai, A. 2007. Wikify!: Linking documents to encyclopedic knowledge. In Proceedings of the 16th ACM Conference on Conference on Information and Knowledge Management (CIKM). 233--242. Google ScholarDigital Library
- Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., and Miller, K. 1990. Wordnet: An on-line lexical database. Int. J. Lexicog. 3, 235--244.Google ScholarCross Ref
- Murakami, T., Mori, K., and Orihara, R. 2008. Metrics for evaluating the serendipity of recommendation lists. In Proceedings of the Conference on New Frontiers in Artificial Intelligence (JSAI). 40--46. Google ScholarDigital Library
- Nanou, T., Lekakos, G., and Fouskas, K. G. 2010. The effects of recommendations’ presentation on persuasion and satisfaction in a movie recommender system. Multimedia Syst. 16, 4--5, 219--230.Google ScholarDigital Library
- Nisbett, R. E. and Wilson, T. D. 1977. The halo effect: Evidence for unconscious alteration of judgments. J. Personal. Social Psychol. 35, 250--256.Google ScholarCross Ref
- Paterek, A. 2007. Improving regularized singular value decomposition for collaborative filtering. In Proceedings of the KDD Cup and Workshop.Google Scholar
- Pazzani, M. and Billsus, D. 2006. Content-based recommendation systems. In The Adaptive Web: Methods and Strategies of Web Personalization. Lecture Notes in Computer Science, 325--341.Google Scholar
- Pu, P. and Chen, L. 2006. Trust building with explanation interfaces. In Proceedings of the 11th International Conference on Intelligent User Interfaces (IUI). 93--100. Google ScholarDigital Library
- Pu, P., Chen, L., and Kumar, P. 2008. Evaluating product search and recommender systems for e-commerce environments. J. Electron. Commerce Res. 8, 1--27. Google ScholarDigital Library
- Pu, P., Zhou, M., and Castagnos, S. 2009. Critiquing recommenders for public taste products. In Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys). 249--252. Google ScholarDigital Library
- Rafey, R. A., Gibbs, S., Hoch, M., Gong, H. L. V., and Wang, S. 2001. Enabling custom enhancements in digital sports broadcasts. In Proceedings of WEB3D’01, 101--107. Google ScholarDigital Library
- Raghavan, V., Bollmann, P., and Jung, G. S. 1989. A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inform. Syst. 7, 3, 205--229. Google ScholarDigital Library
- Rashid, A. M., Karypis, G., and Riedl, J. 2008. Learning preferences of new users in recommender systems: An information theoretic approach. SIGKDD Explorer Newslett. 10, 90--100. Google ScholarDigital Library
- Ricci, F., Rokach, L., Shapira, B., and Kantor, P. B., Eds. 2011. Recommender Systems Handbook. Springer. Google ScholarDigital Library
- Salton, G., Ed. 1988. Automatic Text Processing. Addison-Wesley Longman Publishing Co., Inc., Boston, MA. Google ScholarDigital Library
- Sarwar, B., Karypis, G., Konstan, J., and Riedl, J. 2000. Application of dimensionality reduction in recommender system-a case study. In Proceedings of ACM WebKDD Web Mining for ECommerce Workshop. 285--295.Google Scholar
- Sarwar, B., Karypis, G., Konstan, J., and Reidl, J. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conference on World Wide Web (WWW). 285--295. Google ScholarDigital Library
- Schafer, J. B., Frankowski, D., Herlocker, J., and Sen, S. 2007. The Adaptive Web. Springer-Verlag, Berlin, 291--324.Google Scholar
- Schein, A., Popescul, A., Ungar, L., and Pennock, D. 2002. Methods and metrics for cold-start recommendations. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002). 253--260. Google ScholarDigital Library
- Shani, G. and Gunawardana, A. 2011. Evaluating recommendation systems. In Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor Eds., Springer, 257--297.Google Scholar
- Shearer, A. W. 2001. User response to two algorithms as a test of collaborative filtering. In Extended Abstracts on Human Factors In Computing Systems (CHI EA). 451--452. Google ScholarDigital Library
- Sun, J. and Gao, S. 2007. IPTV based on IP network and streaming media service station. SPIE MIPPR 2007: Remote Sensing and GIS Data Processing and Applications and Innovative Multispectral Technology and Applications 6790, 1, 67904Q.Google Scholar
- Swearingen, K. and Sinha, R. 2001. Beyond algorithms: An HCI perspective on recommender systems. In Proceedings of ACM SIGIR Workshop on Recommender Systems.Google Scholar
- Swearingen, K. and Sinha, R. 2002. Interaction design for recommender systems. In Designing Interactive Systems. ACM.Google Scholar
- Takács, G., Pilászy, I., Németh, B., and Tikk, D. 2009. Scalable collaborative filtering approaches for large recommender systems. J. Mach. Learn. Res. 10, 623--656. Google ScholarDigital Library
- Tintarev, N. 2007. Explanations of recommendations. In Proceedings of the ACM Conference on Recommender Systems (RecSys). 203--206. Google ScholarDigital Library
- Tintarev, N. and Masthoff, J. 2011. Designing and evaluating explanations for recommender systems. In Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor Eds., Springer, 479--510.Google Scholar
- Van Rijsbergen, C. J. 1979. Information Retrieval 2nd Ed. Department of Computer Science, University of Glasgow. Google ScholarDigital Library
- Weng, L.-T., Xu, Y., Li, Y., and Nayak, R. 2007. Improving recommendation novelty based on topic taxonomy. In Proceedings of the IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology-Workshops (WI-IATW). 115--118. Google ScholarDigital Library
- Xiao, B. and Benbasat, I. 2007. E-commerce product recommendation agents: Use, characteristics, and impact. Manage. Inform. Syst. Quarterly 31, 137--209. Google ScholarDigital Library
- Yoo, K. H. and Gretzel, U. 2011. Creating more credible and persuasive recommender systems: The influence of source characteristics on recommender system evaluations. In Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor Eds., Springer, 455--477.Google Scholar
- Zhang, H., Zheng, S., and Yuan, J. 2005. A personalized TV guide system compliant with MHP. IEEE Trans. Consum. Electron. 51, 2, 731--737. Google ScholarDigital Library
- Zhang, M. 2009. Enhancing diversity in top-n recommendation. In Proceedings of the 3rd ACM Conference on Recommender Systems (RecSys). 397--400. Google ScholarDigital Library
- Zhang, M. and Hurly, N. 2009. Evaluating the diversity of top-n recommendations. In Proceedings of the 21st IEEE International Conference on Tools with Artificial Intelligence (ICTAI). 457--460. Google ScholarDigital Library
- Zhang, Y., Callan, J., and Minka, T. 2002. Novelty and redundancy detection in adaptive filtering. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 81--88. Google ScholarDigital Library
- Ziegler, C.-N., McNee, S. M., Konstan, J. A., and Lausen, G. 2005. Improving recommendation lists through topic diversification. In Proceedings of the 14th International Conference on World Wide Web (WWW). 22--32. Google ScholarDigital Library
Index Terms
- Investigating the Persuasion Potential of Recommender Systems from a Quality Perspective: An Empirical Study
Recommendations
Investigating serendipity in recommender systems based on real user feedback
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied ComputingOver the past several years, research in recommender systems has emphasized the importance of serendipity, but there is still no consensus on the definition of this concept and whether serendipitous items should be recommended is still not a well-...
A survey of serendipity in recommender systems
We summarize most efforts on serendipity in recommender systems.We compare definitions of serendipity in recommender systems.We classify the state-of-the-art serendipity-oriented recommendation algorithms.We review methods to assess serendipity in ...
A new approach to evaluating novel recommendations
RecSys '08: Proceedings of the 2008 ACM conference on Recommender systemsThis paper presents two methods, named Item- and User-centric, to evaluate the quality of novel recommendations. The former method focuses on analyzing the item-based recommendation network. The aim is to detect whether the network topology has any ...
Comments