skip to main content
10.1145/1864708.1864722acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
research-article

On the stability of recommendation algorithms

Published: 26 September 2010 Publication History

Abstract

The paper introduces stability as a new measure of the recommender systems performance. In general, we define a recommendation algorithm to be "stable" if its predictions for the same items are consistent over a period of time, assuming that any new ratings that have been submitted to the recommender system over the same period of time are in complete agreement with system's prior predictions. In this paper, we advocate that stability should be a desired property of recommendation algorithms, because unstable recommendations can lead to user confusion and, therefore, reduce trust in recommender systems. Furthermore, we empirically evaluate stability of several popular recommendation algorithms. Our results suggest that model-based recommendation techniques demonstrate higher stability than memory-based collaborative filtering heuristics. We also find that the stability measure for recommendation techniques is influenced by many factors, including the sparsity of the initial rating data, the number of new incoming ratings (representing the length of the time period over which the stability is being measured), the distribution of the newly added rating values, and the rating normalization procedures employed by the recommendation algorithms.

Supplementary Material

JPG File (recsys2010-28092010-04-02.jpg)
MOV File (recsys2010-28092010-04-02.mov)

References

[1]
}}Adomavicius, G. and A. Tuzhilin. Toward the Next Generation of Recommendation System: A Survey of the State-of-the-Art and Possible Extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6): 734--749, 2005.
[2]
}}Bell, R. M. and Y. Koren. Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights. In Proceeding of the Seventh IEEE International Conference on Data Mining. Omaha, NE, USA, 2007.
[3]
}}Bell, R. M. and Y. Koren. Improved Neighborhood-based Collaborative Filtering. In Proceeding of the KDD Cup'07. San Jose, CA, USA, 2007.
[4]
}}Bell, R. M. and Y. Koren. Lessons from the Netflix prize challenge. ACM SIGKDD Explorations Newsletter, 9(2): 75--79, 2007.
[5]
}}Bennet, J. and S. Lanning. The Netflix Prize. In Proceeding of the KDD Cup and Workshop, 2007.
[6]
}}Funk, S. Netflix Update: Try This at Home. Netflix Update: Try This at Home, last update 2006, cited 2010. Available from: http://sifter.org/~simon/journal/20061211.html.
[7]
}}Grouplens. Movielens Data Sets. 2006.
[8]
}}Herlocker, J., J. Kostan, A. Borchers, and J. Riedl. An Algorithmic Framework for Performing Collaborative Filtering. In Proceeding of the 22nd ACM SIGIR Conference on Information Retrieval, 1999.
[9]
}}Herlocker, J., J. Konstan, K. Terveen, and J. Riedl. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1): 5--53, 2004.
[10]
}}Komiak, S. and I. Benbasat. The effects of personalization and familiarity on trust and adoption of recommendation agents. MIS Quarterly, 30(4): 941--960, 2006.
[11]
}}Koren, Y., R. Bell, and C. Volinsky. Matrix Factorization Techniques For Recommender Systems. IEEE Computer, 42: 30--37, 2009.
[12]
}}Kostan, J., B. Miller, D. Maltz, J. Herlocker, L. Gordon, and J. Riedl. GroupLens: Applying Collaborative Filtering to Usenet news. Communications of the ACM, 40: 77--87, 1997.
[13]
}}Lam, S. and J. Riedl. Shilling Recommender Systems for Fun and Profit. In Proceeding of the the 13th international conference on World Wide Web New York City, NY, 2004.
[14]
}}Massa, P. and B. Bhattacharjee. Using trust in recommender systems: An experimental analysis, in Trust Management, Proceeding, 221--235. Springer-Verlag Berlin: Berlin, 2004.
[15]
}}Mobasher, B., R. Burke, and J. J. Sandvig. Model-Based Collaborative Filtering as a Defense against Profile Injection Attacks. In Proceeding of the 21st Conference on Artificial Intelligence (AAAI'06). Boston, MA, 2006.
[16]
}}Mobasher, B., R. Burke, C. Williams, and R. Bhaumik. Analysis and detection of segment-focused attacks against collaborative recommendation, in Advances in Web Mining and Web Usage Analysis, 96--118. Springer-Verlag Berlin: Berlin, 2006.
[17]
}}Mobasher, B., R. Burke, R. Bhaumik, and C. Williams. Toward Trustworthy Recommender Systems: An Analysis of Attack Models and Algorithm Robustness. ACM Transactions on Internet Technology, 7(4): 23:1--23:38, 2007.
[18]
}}O'Donovan, J. and B. Smyth. Mining trust values from recommendation errors. International Journal on Artificial Intelligence Tools, 15(6): 945--962, 2006.
[19]
}}O'Donovan, J. and B. Smyth. Trust in recommender systems. In Proceeding of the 10th international conference on Intelligent user interfaces. San Diego, California, USA, 2005.
[20]
}}Turney, P. Technical Note: Bias and the Quantification of Stability. Machine Learning, 20: 23--33, 1995.
[21]
}}Wang, W. and I. Benbasat. Trust in and adoption of online recommendation agents. Journal of the Association for Information Systems, 6(3): 72--101, 2005.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RecSys '10: Proceedings of the fourth ACM conference on Recommender systems
September 2010
402 pages
ISBN:9781605589060
DOI:10.1145/1864708
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 September 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. collaborative filtering
  2. evaluation of recommender systems
  3. performance measures
  4. stability of recommendation algorithms

Qualifiers

  • Research-article

Conference

RecSys '10
Sponsor:
RecSys '10: Fourth ACM Conference on Recommender Systems
September 26 - 30, 2010
Barcelona, Spain

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 01 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Trust assessment in social networksInternational Journal of System Assurance Engineering and Management10.1007/s13198-023-02118-515:5(1650-1666)Online publication date: 3-Oct-2023
  • (2021)Dynamic Modeling of User Preferences for Stable RecommendationsProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3450613.3456830(262-266)Online publication date: 21-Jun-2021
  • (2017)SmaCHInternational Journal of Ad Hoc and Ubiquitous Computing10.1504/IJAHUC.2017.08702326:3(185-204)Online publication date: 1-Jan-2017
  • (2017)A location‐based IoT platform supporting the cultural heritage domainConcurrency and Computation: Practice and Experience10.1002/cpe.409129:11Online publication date: 17-Feb-2017
  • (2016)Classification, Ranking, and Top-K Stability of Recommendation AlgorithmsINFORMS Journal on Computing10.1287/ijoc.2015.066228:1(129-147)Online publication date: Feb-2016
  • (2016)Stabilized Nearest Neighbor Classifier and its Statistical PropertiesJournal of the American Statistical Association10.1080/01621459.2015.1089772111:515(1254-1265)Online publication date: 18-Oct-2016
  • (2016)Recommending multimedia visiting paths in cultural heritage applicationsMultimedia Tools and Applications10.1007/s11042-014-2062-775:7(3813-3842)Online publication date: 1-Apr-2016
  • (2015)A Survey on Trust ModelingACM Computing Surveys10.1145/281559548:2(1-40)Online publication date: 12-Oct-2015
  • (2015)Improving Stability of Recommender Systems: A Meta-Algorithmic ApproachIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2014.238450227:6(1573-1587)Online publication date: 1-Jun-2015
  • (2014)A Product-Customer Matching Framework for Web 2.0 ApplicationsWeb Information Systems Engineering – WISE 201410.1007/978-3-319-11746-1_36(489-504)Online publication date: 2014
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media