skip to main content
10.1145/3308558.3313469acmotherconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article

How Serendipity Improves User Satisfaction with Recommendations? A Large-Scale User Evaluation

Published: 13 May 2019 Publication History

Abstract

Recommendation serendipity is being increasingly recognized as being equally important as the other beyond-accuracy objectives (such as novelty and diversity), in eliminating the “filter bubble” phenomenon of the traditional recommender systems. However, little work has empirically verified the effects of serendipity on increasing user satisfaction and behavioral intention. In this paper, we report the results of a large-scale user survey (involving over 3,000 users) conducted in an industrial mobile e-commerce setting. The study has identified the significant causal relationships from novelty, unexpectedness, relevance, and timeliness to serendipity, and from serendipity to user satisfaction and purchase intention. Moreover, our findings reveal that user curiosity plays a moderating role in strengthening the relationships from novelty to serendipity and from serendipity to satisfaction. Our third contribution lies in the comparison of several recommender algorithms, which demonstrates the significant improvements of the serendipity-oriented algorithm over the relevance- and novelty-oriented approaches in terms of user perceptions. We finally discuss the implications of this experiment, which include the feasibility of developing a more precise metric for measuring recommendation serendipity, and the potential benefit of a curiosity-based personalized serendipity strategy for recommender systems.

References

[1]
Panagiotis Adamopoulos and Alexander Tuzhilin. 2014. On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected. ACM Trans. Intell. Syst. Technol. 5, 4, Article 54 (Dec. 2014), 32 pages.
[2]
Daniel Ellis Berlyne. 1960. Conflict, Arousal and Curiosity. McGraw-Hill.
[3]
Toine Bogers and Lennart Björneborn. 2013. Micro-serendipity: Meaningful Coincidences in Everyday Life Shared on Twitter. In Proceedings of the iConference 2013. 196-208.
[4]
Barbara M. Byrne. 2016. Structural Equation Modeling With AMOS: Basic Concepts, Applications, and Programming (3rd ed.). Routledge.
[5]
Marco de Gemmis, Pasquale Lops, Giovanni Semeraro, and Cataldo Musto. 2015. An Investigation on the Serendipity Problem in Recommender Systems. Inf. Process. Manage. 51, 5 (Sept. 2015), 695-717.
[6]
Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User Perception of Differences in Recommender Algorithms. In Proceedings of the 8th ACM Conference on Recommender Systems(RecSys '14). ACM, New York, NY, USA, 161-168.
[7]
Donald E. Farrar and Robert R. Glauber. 1967. Multicollinearity in Regression Analysis: The Problem Revisited. The Review of Economics and Statistics 49, 1 (1967), 92-107.
[8]
Shengyi Gao, Patricia L. Mokhtarian, and Robert A. Johnston. 2008. Non-normality of Data in Structural Equation Models. Transportation Research Record 2082, 1 (2008), 116-124.
[9]
Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond Accuracy: Evaluating Recommender Systems by Coverage and Serendipity. In Proceedings of the 4th ACM Conference on Recommender Systems(RecSys '10). ACM, New York, NY, USA, 257-260.
[10]
Asela Gunawardana and Guy Shani. 2015. Evaluating Recommendation Systems. In Recommender Systems Handbook. Springer US, 265-308.
[11]
Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl. 2004. Evaluating Collaborative Filtering Recommender Systems. ACM Trans. Inf. Syst. 22, 1 (Jan. 2004), 5-53.
[12]
Daire Hooper, Joseph Coughlan, and Michael Mullen. 2008. Structural Equation Modelling: Guidelines for Determining Model Fit. Electronic Journal of Business Research Methods 6, 1 (2008), 53-60.
[13]
Dawn Iacobucci, Steven S. Posavac, Frank R. Kardes, Matthew J. Schneider, and Deidre L. Popovich. 2015. The Median Split: Robust, Refined, and Revived. Journal of Consumer Psychology 25, 4 (2015), 690-704.
[14]
Marius Kaminskas and Derek Bridge. 2016. Diversity, Serendipity, Novelty, and Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in Recommender Systems. ACM Trans. Interact. Intell. Syst. 7, 1, Article 2 (Dec. 2016), 42 pages.
[15]
Todd B. Kashdan, Matthew W. Gallagher, Paul J. Silvia, Beate P. Winterstein, William E. Breen, Daniel Terhar, and Michael F. Steger. 2009. The Curiosity and Exploration Inventory-II: Development, Factor Structure, and Psychometrics. Journal of Research in Personality 43, 6 (2009), 987-998.
[16]
Todd B. Kashdan, Paul Rose, and Frank D. Fincham. 2004. Curiosity and Exploration: Facilitating Positive Subjective Experiences and Personal Growth Opportunities. Journal of Personality Assessment 82, 3 (2004), 291-305.
[17]
Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the User Experience of Recommender Systems. User Modeling and User-Adapted Interaction 22, 4-5 (Oct. 2012), 441-504.
[18]
Denis Kotkov, Joseph A. Konstan, Qian Zhao, and Jari Veijalainen. 2018. Investigating Serendipity in Recommender Systems Based on Real User Feedback. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing(SAC '18). ACM, New York, NY, USA, 1341-1350.
[19]
Denis Kotkov, Shuaiqiang Wang, and Jari Veijalainen. 2016. A Survey of Serendipity in Recommender Systems. Knowledge-Based Systems 111 (Nov. 2016), 180-192.
[20]
Neal Lathia, Stephen Hailes, Licia Capra, and Xavier Amatriain. 2010. Temporal Diversity in Recommender Systems. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR '10). ACM, New York, NY, USA, 210-217.
[21]
Todd D. Little, Noel A. Card, James A. Bovaird, Kristopher J. Preacher, and Christian S. Crandall. 2012. Structural Equation Modeling of Mediation and Moderation with Contextual Factors. In Modeling Contextual Effects in Longitudinal Studies. Taylor and Francis, 207-230.
[22]
George Loewenstein. 1994. The Psychology of Curiosity: A Review and Reinterpretation. Psychological Bulletin(1994), 75-98.
[23]
Valentina Maccatrozzo, Manon Terstall, Lora Aroyo, and Guus Schreiber. 2017. SIRUP: Serendipity In Recommendations via User Perceptions. In Proceedings of the 22nd International Conference on Intelligent User Interfaces(IUI '17). ACM, New York, NY, USA, 35-44.
[24]
Frank J. Massey. 1951. The Kolmogorov-Smirnov Test for Goodness of Fit. J. Amer. Statist. Assoc. 46, 253 (1951), 68-78.
[25]
Christian Matt, Thomas Hess, Alexander Benlian, and Christian Weiβ. 2014. Escaping from the Filter Bubble? The Effects of Novelty and Serendipity on Users' Evaluations of Online Recommendations. (2014). https://EconPapers.repec.org/RePEc:dar:wpaper:66193
[26]
John H. McDonald. 2014. Handbook of Biological Statistics, Third Edition. Sparky House Publishing.
[27]
Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being Accurate is Not Enough: How Accuracy Metrics Have Hurt Recommender Systems. In CHI '06 Extended Abstracts on Human Factors in Computing Systems(CHI EA '06). ACM, New York, NY, USA, 1097-1101.
[28]
Alan Menk, Laura Sebastia, and Rebeca Ferreira. 2017. CURUMIM: A Serendipitous Recommender System for Tourism Based on Human Curiosity. In 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI). 788-795.
[29]
Tomoko Murakami, Koichiro Mori, and Ryohei Orihara. 2008. Metrics for Evaluating the Serendipity of Recommendation Lists. In New Frontiers in Artificial Intelligence. Springer Berlin Heidelberg, 40-46.
[30]
Bengt Muthe´n and David Kaplan. 1985. A Comparison of Methodologies for the Factor Analysis of Non-Normal Likert Variables. Brit. J. Math. Statist. Psych. 38, 1 (1985), 171-189.
[31]
Robert M. O'Brien. 2007. A Caution Regarding Rules of Thumb for Variance Inflation Factors. Quality & Quantity 41, 5 (01 Oct 2007), 673-690.
[32]
Kenta Oku and Fumio Hattori. 2012. User Evaluation of Fusion-Based Approach for Serendipity-Oriented Recommender System. In Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE 2012). 39-44.
[33]
Eli Pariser. 2011. The Filter Bubble: What the Internet Is Hiding from You. Penguin UK.
[34]
Robert A. Peterson. 1994. A Meta-Analysis of Cronbach's Coefficient Alpha. Journal of Consumer Research 21, 2 (1994), 381-391.
[35]
Pearl Pu, Li Chen, and Rong Hu. 2011. A User-centric Evaluation Framework for Recommender Systems. In Proceedings of the 5th ACM Conference on Recommender Systems(RecSys '11). ACM, New York, NY, USA, 157-164.
[36]
Alan Said, Ben Fields, Brijnesh J. Jain, and Sahin Albayrak. 2013. User-centric Evaluation of a K-furthest Neighbor Collaborative Filtering Recommender Algorithm. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work(CSCW '13). ACM, New York, NY, USA, 1399-1408.
[37]
Randall E. Schumacker and Richard G. Lomax. 2015. A Beginner's Guide to Structural Equation Modeling (4th ed.). Routledge.
[38]
Paul J. Silvia. 2008. Interest-The Curious Emotion. Current Directions in Psychological Science 17, 1 (02 2008), 57-60.
[39]
Kirsten Swearingen and Rashmi Sinha. 2001. Beyond Algorithms: An HCI Perspective on Recommender Systems. In ACM SIGIR. Workshop on Recommender Systems, Vol. 13. Citeseer, 1-11.
[40]
Paula Vicente, Elizabeth Reis, and Maria Santos. 2009. Using Mobile Phones for Survey Research: A Comparison with Fixed Phones. International Journal of Market Research 51, 5 (2009), 613-633.
[41]
Horace Walpole. 1960. To Mann, Monday 18 January 1754. In Horace Walpole's Correspondence. Yale University Press, 407-411.
[42]
Jen-Her Wu and Shu-Ching Wang. 2005. What Drives Mobile Commerce?Information and Management 42, 5 (July 2005), 719-729.
[43]
Wen Wu, Li Chen, and Yu Zhao. 2018. Personalizing Recommendation Diversity Based on User Personality. User Modeling and User-Adapted Interaction 28, 3 (01 Aug 2018), 237-276.
[44]
Shengquan Ye, Ting Kin Ng, Kin Hang Yim, and Jun Wang. 2015. Validation of the Curiosity and Exploration Inventory-II (CEI-II) Among Chinese University Students in Hong Kong. Journal of Personality Assessment 97, 4 (2015), 403-410. 25774779.
[45]
Yuan Cao Zhang, Diarmuid Ó Se´aghdha, Daniele Quercia, and Tamas Jambor. 2012. Auralist: Introducing Serendipity into Music Recommendation. In Proceedings of the 5th ACM International Conference on Web Search and Data Mining(WSDM '12). ACM, New York, NY, USA, 13-22.
[46]
Pengfei Zhao and Dik Lun Lee. 2016. How Much Novelty is Relevant?: It Depends on Your Curiosity. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR '16). ACM, New York, NY, USA, 315-324.
[47]
Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005. Improving Recommendation Lists Through Topic Diversification. In Proceedings of the 14th International Conference on World Wide Web(WWW '05). ACM, New York, NY, USA, 22-32.

Cited By

View all
  • (2025)“We do not always enjoy surprises”: investigating artificial serendipity in an online marketplace contextJournal of Documentation10.1108/JD-01-2024-0011Online publication date: 28-Jan-2025
  • (2025)The Role of Serendipity in Narratives: How Serendipitous Story Promotes Product InterestPsychology & Marketing10.1002/mar.22181Online publication date: 31-Jan-2025
  • (2024)Enhancing Recommendation Diversity and Novelty with Bi-LSTM and Mean Shift ClusteringElectronics10.3390/electronics1319384113:19(3841)Online publication date: 28-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
WWW '19: The World Wide Web Conference
May 2019
3620 pages
ISBN:9781450366748
DOI:10.1145/3308558
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • IW3C2: International World Wide Web Conference Committee

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Recommender systems
  2. curiosity
  3. large-scale user evaluation
  4. serendipity
  5. user satisfaction

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

WWW '19
WWW '19: The Web Conference
May 13 - 17, 2019
CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)375
  • Downloads (Last 6 weeks)21
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)“We do not always enjoy surprises”: investigating artificial serendipity in an online marketplace contextJournal of Documentation10.1108/JD-01-2024-0011Online publication date: 28-Jan-2025
  • (2025)The Role of Serendipity in Narratives: How Serendipitous Story Promotes Product InterestPsychology & Marketing10.1002/mar.22181Online publication date: 31-Jan-2025
  • (2024)Enhancing Recommendation Diversity and Novelty with Bi-LSTM and Mean Shift ClusteringElectronics10.3390/electronics1319384113:19(3841)Online publication date: 28-Sep-2024
  • (2024)The Art of Asking: Prompting Large Language Models for Serendipity RecommendationsProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672521(157-166)Online publication date: 2-Aug-2024
  • (2024)Group Validation in Recommender Systems: Framework for Multi-layer Performance EvaluationACM Transactions on Recommender Systems10.1145/36408202:1(1-25)Online publication date: 19-Jan-2024
  • (2024)Exploring the Landscape of Recommender Systems Evaluation: Practices and PerspectivesACM Transactions on Recommender Systems10.1145/36291702:1(1-31)Online publication date: 7-Mar-2024
  • (2024)Deep Recommender Models Inference: Automatic Asymmetric Data Flow Optimization2024 IEEE 42nd International Conference on Computer Design (ICCD)10.1109/ICCD63220.2024.00085(517-520)Online publication date: 18-Nov-2024
  • (2024)A Group Travel Recommender System Based on Group Approximate Constraint SatisfactionIEEE Access10.1109/ACCESS.2024.342712212(96113-96125)Online publication date: 2024
  • (2024)Individual Persistence Adaptation for User-Centric Evaluation of User Satisfaction in Recommender SystemsIEEE Access10.1109/ACCESS.2024.336069312(23626-23635)Online publication date: 2024
  • (2024)In-processing and post-processing strategies for balancing accuracy and sustainability in product recommendationsElectronic Commerce Research and Applications10.1016/j.elerap.2024.10143367(101433)Online publication date: Sep-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media