Skip to main content
SpringerLink
Log in

CROC: A New Evaluation Criterion for Recommender Systems

  • Published:
Electronic Commerce Research Aims and scope Submit manuscript

Abstract

Evaluation of a recommender system algorithm is a challenging task due to the many possible scenarios in which such systems may be deployed. We have designed a new performance plot called the CROC curve with an associated statistic: the area under the curve. Our CROC curve supplements the widely used ROC curve in recommender system evaluation by discovering performance characteristics that standard ROC evaluation often ignores. Empirical studies on two domains and including several recommender system algorithms demonstrate that combining ROC and CROC curves in evaluation can lead to a more informed characterization of performance than using either curve alone.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

REFERENCES

  1. Basu, C., H. Hirsh, and W. Cohen. (1998). “Recommendation as Classification: Using Social and Content-Based Information in Recommendation.” In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pp. 714–720.

  2. Billsus, D. and M.J. Pazzani. (1998). “Learning Collaborative Information Filters.” In Proceedings of the 15th International Conference on Machine Learning, pp. 46–54.

  3. Breese, J.S., D. Heckerman, and C. Kadie. (1998). “Empirical Analysis of Predictive Algorithms for Collaborative Filtering.” In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pp. 43–52.

  4. Claypool, M., A. Gokhale, and T. Miranda. (1999). “Combining Content-Based and Collaborative Filters in an Online Newspaper.” In Proceedings of the ACM SIGIR Workshop on Recommender Systems — Implementation and Evaluation.

  5. Cohen, W.W., R.E. Schapire, and Y. Singer. (1999). “Learning to Order Things.” Journal of Artificial Intelligence Research10, 243–270.

    Google Scholar 

  6. Condliff, M.K., D.D. Lewis, D. Madigan, and C. Posse. (1999). “Bayesian Mixed-Effect Models for Recommender Systems.” In ACM SIGIR '99 Workshop on Recommender Systems: Algorithms and Evaluation.

  7. Feller, W. (1990). Introduction to Probability Theory and Its Application, Vol. 1. Wiley, 3rd edition.

  8. Freund, Y., R. Iyer, R.E. Schapire, and Y. Singer. (1998). “An Efficient Boosting Algorithm for Combining Preferences.” In Proceedings of the 15th International Conference on Machine Learning, pp. 170–178.

  9. González-Caro, C.N., M.L. Calderón-Benavides, J. de J. Pérez-Alcázar, J.C. García-Díaz, and J. Delgado. (2002). “Towards a More Comprehensive Comparison of Collaborative Filtering Algorithms.” In Proceedings of the 2002 String Processing and Information Retrieval Conference (SPIRE 2002), pp. 248–253.

  10. Good, N., J.B. Schafer, J.A. Konstan, A. Borchers, B.M. Sarwar, J.L. Herlocker, and J. Riedl. (1999). “Combining Collaborative Filtering with Personal Agents for Better Recommendations.” In Proceedings of the 16th National Conference on Artificial Intelligence, pp. 439–446.

  11. Guo, H. (1997). “SOAP: Live Recommendations through Social Agents.” In 5th DELOS Workshop on Filtering and Collaborative Filtering, Budapest.

  12. Heckerman, D., D.M. Chickering, C. Meek, R. Rounthwaite, and C. Kadie. (2000). “Dependency Networks for Collaborative Filtering and Data Visualization.” In Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, pp. 264–273.

  13. Herlocker, J.L., J.A. Konstan, A. Borchers, and J. Riedl. (1999). “An Algorithmic Framework for Performing Collaborative Filtering.” In Proceedings of the Conference on Research and Development in Information Retrieval.

  14. Hill, W., L. Stead, M. Rosenstein, and G. Furnas. (1995). “Recommending and Evaluating Choices in a Virtual Community of Use.” In Proceedings of the Conference on Human Factors in Computing Systems, pp. 194–201.

  15. Hofmann, T. (1999). “Probabilistic Latent Semantic Indexing.” In 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 50–57

  16. Hofmann, T. and J. Puzicha. (1999). “Latent Class Models for Collaborative Filtering.” In Proceedings of the 16th International Joint Conference on Artificial Intelligence, pp. 688–693.

  17. Konstan, J.A., B.N. Miller, D. Maltz, J.L. Herlocker, L.R. Gordon, and J. Riedl. (1997). “GroupLens: Applying Collaborative Filtering to Usenet News.” Communications of the ACM 40(3), 77–87.

    Google Scholar 

  18. Melville, P., R.J. Mooney, and R. Nagarajan. (2001). “Content-Boosted Collaborative Filtering.” In Pro-ceedings of the 2001 SIGIR Workshop on Recommender Systems.

  19. Mooney, R.J. and L. Roy. (2000). “Content-Based Book Recommending Using Learning for Text Catego-rization.” In Proceedings of the 5th ACM Conference on Digital Libraries, pp. 195–204.

  20. Nakamura, A. and N. Abe. (1998). “Collaborative Filtering Using Weighted Majority Prediction Algo-rithms.” In Proceedings of the 15th International Conference on Machine Learning, pp. 395–403.

  21. Pennock, D.M., E. Horvitz, and C.L. Giles. (2000). “Social Choice Theory and Recommender Systems: Analysis of the Axiomatic Foundations of Collaborative Filtering.” In Proceedings of the 17th National Conference on Artificial Intelligence, pp. 729–734.

  22. Pennock, D.M., E. Horvitz, S. Lawrence, and C.L. Giles. (2000). “Collaborative Filtering by Personality Diagnosis: A Hybrid Memory-and Model-Based Approach.” In Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, pp. 473–480.

  23. Popescul, A., L.H. Ungar, D.M. Pennock, and S. Lawrence. (2001). “Probabilistic Models for Unified Collaborative and Content-Based Recommendation in Sparse-Data Environments.” In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence.

  24. Resnick, P., N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl. (1994). “GroupLens: An Open Architecture for Collaborative Filtering of Netnews.” In Proceedings of the ACM Conference on Computer Supported Cooperative Work, pp. 175–186.

  25. Resnick, P. and H.R. Varian. (1997). “Recommender Systems.” Communications of the ACM40(3), 56–58.

    Google Scholar 

  26. Salton, G. and M. McGill. (1983). Introduction to Modern Information Retrieval. McGraw-Hill.

  27. Sarwar, B.M., G. Karypis, J.A. Konstan, and J.T. Riedl. (2000). “Analysis of Recommender Algorithms for E-Commerce.” In Proceedings of the 2nd ACM Conference on Electronic Commerce, pp. 158–167.

  28. Sarwar, B.M., G. Karypis, J.A. Konstan, and J.T. Riedl. (2000). “Application of Dimensionality Reduction in Recommender System—A Case Study.” In ACM WebKDD Web Mining for E-Commerce Workshop.

  29. Sarwar, B.M., G. Karypis, J.A. Konstan, and J.T. Riedl. (2001). “Item-Based Collaborative Filtering Rec-ommendation Algorithms.” In Proceedings of the 10th International World Wide Web Conference, pp. 285–295.

  30. Schein, A.I., A. Popescul, and L.H. Ungar. (2001). “PennAspect: A Two-Way Aspect Model Implementa-tion.” Technical Report MS-CIS-01-25, Department of Computer and Information Science, The University of Pennsylvania.

  31. Schein, A.I., A. Popescul, L.H. Ungar, and D.M. Pennock. (2001). “Generative Models for Cold-Start Recommendations.” In Proceedings of the 2001 SIGIR Workshop on Recommender Systems.

  32. Schein, A.I., A. Popescul, L.H. Ungar, and D.M. Pennock. (2002). “Methods and Metrics for Cold-Start Recommendations.” In Proceedings of the 25th International ACM Conference on Research and Development in Information Retrieval.

  33. Schein, A.I., L.K. Saul, and L.H. Ungar. (2003). “A Generalized Linear Model for Principal Component Analysis of Binary Data.” In Proceedings of the 9th International Workshop on Artificial Intelligence and Statistics.

  34. Shardanand, U. and P. Maes. (1995.) “Social Information Filtering: Algorithms for Automating ‘Word of Mouth’.” In Proceedings of the Conference on Human Factors in Computing Systems, pp. 210–217.

  35. Swets, J.A. (1988). “Measuring the Accuracy of Diagnostic Systems.” Science240, 1285–1293.

    Google Scholar 

  36. Ungar, L.H. and D.P. Foster. (1998). “Clustering Methods for Collaborative Filtering.” In Workshop on Recommendation Systems at the 15th National Conference on Artificial Intelligence.

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, University of Pennsylvania, Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104-6389, USA

    Andrew I. Schein, Alexandrin Popescul & Lyle H. Ungar

  2. Overture Services, Inc., 74 N. Pasadena Ave, 3rd floor, Pasadena, CA 91103, USA

    David M. Pennock

Authors
  1. Andrew I. Schein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexandrin Popescul
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lyle H. Ungar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David M. Pennock
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schein, A.I., Popescul, A., Ungar, L.H. et al. CROC: A New Evaluation Criterion for Recommender Systems. Electronic Commerce Research 5, 51–74 (2005). https://doi.org/10.1023/B:ELEC.0000045973.51289.8c

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/B:ELEC.0000045973.51289.8c

Search

Navigation