research-article

Stability of Recommendation Algorithms

Authors:

Gediminas Adomavicius,

Jingjing ZhangAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 30, Issue 4

Article No.: 23, Pages 1 - 31

https://doi.org/10.1145/2382438.2382442

Published: 01 November 2012 Publication History

Abstract

The article explores stability as a new measure of recommender systems performance. Stability is defined to measure the extent to which a recommendation algorithm provides predictions that are consistent with each other. Specifically, for a stable algorithm, adding some of the algorithm’s own predictions to the algorithm’s training data (for example, if these predictions were confirmed as accurate by users) would not invalidate or change the other predictions. While stability is an interesting theoretical property that can provide additional understanding about recommendation algorithms, we believe stability to be a desired practical property for recommender systems designers as well, because unstable recommendations can potentially decrease users’ trust in recommender systems and, as a result, reduce users’ acceptance of recommendations. In this article, we also provide an extensive empirical evaluation of stability for six popular recommendation algorithms on four real-world datasets. Our results suggest that stability performance of individual recommendation algorithms is consistent across a variety of datasets and settings. In particular, we find that model-based recommendation algorithms consistently demonstrate higher stability than neighborhood-based collaborative filtering techniques. In addition, we perform a comprehensive empirical analysis of many important factors (e.g., the sparsity of original rating data, normalization of input data, the number of new incoming ratings, the distribution of incoming ratings, the distribution of evaluation data, etc.) and report the impact they have on recommendation stability.

References

[1]

Adomavicius, G. and Tuzhilin, A. 2005. Toward the next generation of recommendation system: A survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Engin. 17, 734--749.

Digital Library

[2]

Adomavicius, G. and Zhang, J. 2010. On the stability of recommendation algorithms. In Proceedings of the ACM Conference on Recommender Systems. 47--54.

Digital Library

[3]

Balabanovic, M. and Shoham, Y. 1997. Fab: Content-based, collaborative recommendation. Comm. ACM 40, 3, 66--72.

Digital Library

[4]

Bell, R. M. and Koren, Y. 2007a. Improved neighborhood-based collaborative filtering. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 7--14.

[5]

Bell, R. M. and Koren, Y. 2007b. Lessons from the Netflix prize challenge. ACM SIGKDD Explor. Newslet. 9, 75--79.

Digital Library

[6]

Bell, R. M. and Koren, Y. 2007c. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proceedings of the 7th IEEE International Conference on Data Mining.

Digital Library

[7]

Bennett, J. and Lanning, S. 2007. The Netflix Prize. In Proceedings of the KDD-Cup and Workshop. www.netflixprize.com.

[8]

Billsus, D. and Pazzani, M. 1998. Learning collaborative information Filters. In Proceedings of the 15th International Conference on Machine Learning (ICML’98). 46--54.

Digital Library

[9]

Breese, J. S., Heckerman, D., and Kadie, C. 1998. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence.

Digital Library

[10]

Burden, R. L. and Faires, D. 2004. Numerical Analysis. Thomson Brooks/Cole.

[11]

D’stous, A. and Touil, N. 1999. Consumer evaluations of movies on the basis of critics’ judgments. Psych. Market. 16, 677--694.

[12]

Dias, M. B., Locher, D., Li, M., El-Deredy, W., and Lisboa, P. 2008. The value of personalised recommender systems to e-business: A case study. In Proceedings of the ACM Conference on Recommender Systems (RecSys’08). ACM, New York, NY, 291--294.

Digital Library

[13]

Duda, R. O., Hart, P. E., and Stork, D. G. 2000. Pattern Classification. Wiley.

Digital Library

[14]

Funk, S. 2006. Netflix update: Try this at home. http://sifter.org/~simon/journal/20061211.html.

[15]

Garfinkel, R., Gopal, R., Pathak, B., Venkatesan, R., and Yin, F. 2006. Empirical analysis of the business value of recommender systems. http://ssrn.com/abstract=958770.

[16]

Gershoff, A., Mukherjee, A., and Mukhopadhyay, A. 2003. Consumer acceptance of online agent advice: Extremity and positivity effects. J. Consumer Psych. 13, 1&2, 161--170.

[17]

Goldberg, K., Roeder, T., Gupta, D., and Perkins, C. 2001. Eigentaste: A constant time collaborative filtering algorithm. Inf. Retrieval 4, 133--151.

Digital Library

[18]

Grouplens. 2006. Movielens Data Sets. http://www.grouplens.org/.

[19]

Hastie, T., Tibshirani, R., and Friedman, J. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer.

[20]

Herlocker, J., Kostan, J., Borchers, A., and Riedl, J. 1999. An algorithmic framework for performing collaborative filtering. In Proceedings of the 22nd ACM SIGIR Conference on Information Retrieval. 230--237.

Digital Library

[21]

Herlocker, J., Kostan, J., Terveen, K., and Riedl, J. T. 2004. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 5--53.

Digital Library

[22]

Higham, N. J. 2002. Accuracy and Stability of Numerical Algorithms. SIAM, Philadelphia.

Digital Library

[23]

Komiak, S. and Benbasat, I. 2006. The effects of personalization and familiarity on trust and adoption of recommendation agents. MIS Quart. 30, 941--960.

Digital Library

[24]

Koren, Y. 2010. Factor in the neighbors: Scalable and accurate collaborative filtering. ACM Trans. Knowl. Discov. Data 4, 1--24.

Digital Library

[25]

Koren, Y., Bell, R., and Volinsky, C. 2009. Matrix factorization techniques for recommender systems. IEEE Comput. 42, 30--37.

Digital Library

[26]

Kostan, J., Miller, B., Maltz, D., Herlocker, J., Gordon, L., and Riedl, J. 1997. GroupLens: Applying collaborative filtering to usenet news. Comm. ACM 40, 77--87.

Digital Library

[27]

Lam, S. and Riedl, J. 2004. Shilling recommender systems for fun and profit. In Proceedings of the 13th International Conference on World Wide Web.

Digital Library

[28]

Legwinski, T. 2010. Recommendation explanations increase sales & customer confidence. http://blog.strands.com/2010/03/17/recommendations-increase-sales-customer-confidence/.

[29]

Lyapunov, A. M. 1992. The General Problem of the Stability of Motion. CRC Press.

[30]

Massa, P. and Avesani, P. 2006. Trust-aware Bootstrapping of recommender systems. In Proceedings of the ECAI Workshop on Recommender Systems.

Digital Library

[31]

Massa, P. and Bhattacharjee, B. 2004. Using trust in recommender systems: An experimental analysis. In Proceedings of the International Conference on Trust Management. Springer, 221--235.

[32]

Mobasher, B., Burke, R., and Sandvig, J. J. 2006a. Model-based collaborative filtering as a defense against profile injection attacks. In Proceedings of the 21st Conference on Artificial Intelligence (AAAI’06).

Digital Library

[33]

Mobasher, B., Burke, R., Williams, C., and Bhaumik, R. 2006b. Analysis and detection of segment-focused attacks against collaborative recommendation. In Advances in Web Mining and Web Usage Analysis. Springer, 96--118.

Digital Library

[34]

Mobasher, B., Burke, R., Bhaumik, R., and Williams, C. 2007. Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness. ACM Trans. Intern. Techn. 7, 23, 21--38.

Digital Library

[35]

O’Donovan, J. and Smyth, B. 2005. Trust in recommender systems. In Proceedings of the 10th International Conference on Intelligent User Interfaces.

Digital Library

[36]

O’Donovan, J. and Smyth, B. 2006. Mining trust values from recommendation errors. Int. J. Artif. Intell. Tools 15, 945--962.

[37]

O’Mahony, M. P., Hurley, N. J., and Silvestre, G. C. M. 2004. An evaluation of neighbourhood formation on the performance of collaborative filtering. Artif. Intell. Rev. 21, 215--228.

Digital Library

[38]

Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. 1994. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the Conference on Computer Supported Cooperative Work. 175--186.

Digital Library

[39]

Rigby, C. 2011. Internet Retailing webinars. Review: Boosting email marketing revenue with personalised recommendations by Silverpop, Baynote and MusicRoom.com. http://www.internetretailing.net/2011/10/internet-retailing-webinars-review-boosting-email-marketing-revenue-with-personalised-recommendations-by-silverpop-baynote-and-musicroom-com/.

[40]

Sarwar, B., Konstan, J., Borchers, A., Herlocker, J., Miller, B., and Riedl, J. 1998. Using filtering agents to improve prediction quality in the Grouplens research collaborative filtering system. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 345--354.

Digital Library

[41]

Sarwar, B., Karypis, G., Konstan, J. A., and Riedl, J. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conferenceon the World Wide Web. 285--295.

Digital Library

[42]

Shani, G. and Gunawardana, A. 2011. Evaluating recommender systems. In Recommender Systems Handbook: A Complete Guide for Research Scientists and Practitioners, P. Kantor, F. Ricci, L. Rokach, and B. Shapira Eds., Springer.

[43]

Simitses, G. and Hodges, D. 2005. Fundamentals of Structural Stability. Butterworth-Heinemann, Burlington, MA.

[44]

Turney, P. 1994. A theory of cross-validation error. J. Exp. Theor. Artif. Intell. 6, 361--391.

[45]

Turney, P. 1995. Technical note: Bias and the quantification of stability. Mach. Learn. 20, 23--33.

Digital Library

[46]

Van Swol, L. M. and Sniezek, J. A. 2005. Factors affecting the acceptance of expert advice. Brit. J. Social Psych. 44, 3, 443--461.

[47]

Wang, W. and Benbasat, I. 2005. Trust in and adoption of online recommendation agents. J. Assoc. Inf. Syst. 6, 72--101.

Cited By

Sun JShi ZSu LShen WWang ZLi YYu WLin WWu FHe BZhou J(2025)Helios: Efficient Distributed Dynamic Graph Sampling for Online GNN InferenceProceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming10.1145/3710848.3710854(2-15)Online publication date: 28-Feb-2025
https://dl.acm.org/doi/10.1145/3710848.3710854
Li DYang X(2024)Evaluation and improvement of personalized recommendation system in low-carbon network marketingInternational Journal of Low-Carbon Technologies10.1093/ijlct/ctae25219(2831-2838)Online publication date: 11-Dec-2024
https://doi.org/10.1093/ijlct/ctae252
Mao CChen JYi XWen L(2024)An empirical study on metamorphic testing for recommender systemsInformation and Software Technology10.1016/j.infsof.2024.107410169(107410)Online publication date: May-2024
https://doi.org/10.1016/j.infsof.2024.107410
Show More Cited By

Index Terms

Stability of Recommendation Algorithms
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction
  2. Information systems applications
    1. Data mining

Recommendations

On the stability of recommendation algorithms
RecSys '10: Proceedings of the fourth ACM conference on Recommender systems

The paper introduces stability as a new measure of the recommender systems performance. In general, we define a recommendation algorithm to be "stable" if its predictions for the same items are consistent over a period of time, assuming that any new ...
Classification, Ranking, and Top-K Stability of Recommendation Algorithms

Recommendation stability measures the extent to which a recommendation algorithm provides predictions that are consistent with each other. Several approaches have been proposed in prior work to defining, measuring, and improving the stability of ...
A recursive prediction algorithm for collaborative filtering recommender systems
RecSys '07: Proceedings of the 2007 ACM conference on Recommender systems

Collaborative filtering (CF) is a successful approach for building online recommender systems. The fundamental process of the CF approach is to predict how a user would like to rate a given item based on the ratings of some nearest-neighbor users (user-...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 30, Issue 4

November 2012

216 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/2382438

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2012

Accepted: 01 June 2012

Revised: 01 March 2012

Received: 01 October 2011

Published in TOIS Volume 30, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Division of Information and Intelligent Systems

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

56
Total Citations
View Citations
1,577
Total Downloads

Downloads (Last 12 months)89
Downloads (Last 6 weeks)18

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun JShi ZSu LShen WWang ZLi YYu WLin WWu FHe BZhou J(2025)Helios: Efficient Distributed Dynamic Graph Sampling for Online GNN InferenceProceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming10.1145/3710848.3710854(2-15)Online publication date: 28-Feb-2025
https://dl.acm.org/doi/10.1145/3710848.3710854
Li DYang X(2024)Evaluation and improvement of personalized recommendation system in low-carbon network marketingInternational Journal of Low-Carbon Technologies10.1093/ijlct/ctae25219(2831-2838)Online publication date: 11-Dec-2024
https://doi.org/10.1093/ijlct/ctae252
Mao CChen JYi XWen L(2024)An empirical study on metamorphic testing for recommender systemsInformation and Software Technology10.1016/j.infsof.2024.107410169(107410)Online publication date: May-2024
https://doi.org/10.1016/j.infsof.2024.107410
Zhao SLiu Y(2024)Confidence region for distributed stochastic optimization problem via stochastic gradient tracking methodAutomatica10.1016/j.automatica.2023.111352159(111352)Online publication date: Jan-2024
https://doi.org/10.1016/j.automatica.2023.111352
Benton P(2024)Democratic AI: Justification for a Broad View of Public ReasonArtificial Intelligence Research10.1007/978-3-031-78255-8_26(440-455)Online publication date: 26-Nov-2024
https://doi.org/10.1007/978-3-031-78255-8_26
Xu JLiu JMa ZWang YWang WNgai E(2024)KNN-Based Collaborative Filtering for Fine-Grained Intelligent Grad-School Recommendation SystemQuality, Reliability, Security and Robustness in Heterogeneous Systems10.1007/978-3-031-65126-7_41(494-508)Online publication date: 20-Aug-2024
https://doi.org/10.1007/978-3-031-65126-7_41
Ekstrand MCarterette BDiaz F(2023)Distributionally-Informed Recommender System EvaluationACM Transactions on Recommender Systems10.1145/36134552:1(1-27)Online publication date: 5-Aug-2023
https://dl.acm.org/doi/10.1145/3613455
Bhatia MAhanger TAlqahtani A(2023)Game-Theoretic Decision Making for Intelligent Power Consumption AnalysisIEEE Internet of Things Journal10.1109/JIOT.2022.321857610:9(7537-7544)Online publication date: 1-May-2023
https://doi.org/10.1109/JIOT.2022.3218576
Hong D(2023)How much is a “feedback” worth? User engagement and interaction for computer-supported adaptive quizzingInteractive Learning Environments10.1080/10494820.2023.2176521(1-16)Online publication date: 1-Mar-2023
https://doi.org/10.1080/10494820.2023.2176521
Han YJia XLi Y(2023)Application of speech recognition for collaborative filtering recommendation in intelligent financial sharingInternational Journal of System Assurance Engineering and Management10.1007/s13198-023-02024-wOnline publication date: 27-Jul-2023
https://doi.org/10.1007/s13198-023-02024-w
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents