skip to main content
10.1145/3240323.3240351acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
research-article

Judging similarity: a user-centric study of related item recommendations

Published: 27 September 2018 Publication History

Abstract

Related item recommenders operate in the context of a particular item. For instance, a music system's page about the artist Radio-head might recommend other similar artists such as The Flaming Lips. Often central to these recommendations is the computation of similarity between pairs of items. Prior work has explored many algorithms and features that allow for the computation of similarity scores, but little work has evaluated these approaches from a user-centric perspective. In this work, we build and evaluate six similarity scoring algorithms that span a range of activity- and content-based approaches. We evaluate the performance of these algorithms using both offline metrics and a new set of more than 22,000 user-contributed evaluations. We integrate these results with a survey of more than 700 participants concerning their expectations about item similarity and related item recommendations. We find that content-based algorithms outperform ratings- and clickstream-based algorithms in terms of how well they match user expectations for similarity and recommendation quality. Our results yield a number of implications to guide the construction of related item recommendation algorithms.

Supplementary Material

ZIP File (p288-yao-aux.zip)
Supplemental files.
MP4 File (p288-yao.mp4)

References

[1]
Deepak Agarwal, Bee-Chung Chen, Pradheep Elango, and Raghu Ramakrishnan. 2013. Content Recommendation on Web Portals. Commun. ACM 56, 6 (June 2013), 92--101.
[2]
Rakesh Agrawal, Tomasz Imieliński, and Arun Swami. 1993. Mining Association Rules Between Sets of Items in Large Databases. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data (SIGMOD '93). ACM, New York, NY, USA, 207--216.
[3]
Hyung Jun Ahn. 2008. A new similarity measure for collaborative filtering to alleviate the new user cold-starting problem. Information Sciences 178, 1 (Jan. 2008), 37--51.
[4]
Kamal Ali and Wijnand van Stam. 2004. TiVo: Making Show Recommendations Using a Distributed Collaborative Filtering Architecture. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '04). ACM, New York, NY, USA, 394--401.
[5]
Paulo J. Azevedo and Alípio M. Jorge. 2007. Comparing Rule Measures for Predictive Association Rules. In Machine Learning: ECML 2007 (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg, 510--517.
[6]
O. Barkan and N. Koenigstein. 2016. ITEM2VEC: Neural item embedding for collaborative filtering. In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP). 1--6.
[7]
Bert Boyce. 1982. Beyond topicality: A two stage view of relevance and the retrieval process. Information Processing & Management 18, 3 (Jan. 1982), 105--109.
[8]
Sergey Brin, Rajeev Motwani, Jeffrey D. Ullman, and Shalom Tsur. 1997. Dynamic Itemset Counting and Implication Rules for Market Basket Data. In Proceedings of the 1997 ACM SIGMOD International Conference on Management of Data (SIGMOD '97). ACM, New York, NY, USA, 255--264.
[9]
Robin Burke and Maryam Ramezani. 2011. Matching Recommendation Technologies and Domains. In Recommender Systems Handbook. Springer, Boston, MA, 367--386.
[10]
R. H. B. Christensen. 2018. ordinal---Regression Models for Ordinal Data. https://CRAN.R-project.org/package=ordinal R package version 2018.4-19.
[11]
Lucas Colucci, Prachi Doshi, Kun-Lin Lee, Jiajie Liang, Yin Lin, Ishan Vashishtha, Jia Zhang, and Alvin Jude. 2016. Evaluating Item-Item Similarity Algorithms for Movies. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 2141--2147.
[12]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16). ACM, New York, NY, USA, 191--198.
[13]
James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, and Dasarathi Sampath. 2010. The YouTube Video Recommendation System. In Proceedings of the Fourth ACM Conference on Recommender Systems (RecSys '10). ACM, New York, NY, USA, 293--296.
[14]
Christian Desrosiers and George Karypis. 2011. A Comprehensive Survey of Neighborhood-based Recommendation Methods. In Recommender Systems Handbook. Springer, Boston, MA, 107--144.
[15]
J. Stephen Downie, Andreas F. Ehmann, Mert Bay, and M. Cameron Jones. 2010. The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. In Advances in Music Information Retrieval. Springer, Berlin, Heidelberg, 93--115.
[16]
Y. Hu, Y. Koren, and C. Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In 2008 Eighth IEEE International Conference on Data Mining. 263--272.
[17]
J. Katukuri, T. Könik, R. Mukherjee, and S. Kolay. 2014. Recommending similar items in large-scale online marketplaces. In 2014 IEEE International Conference on Big Data (Big Data). 868--876.
[18]
Peter Knees and Markus Schedl. 2013. A Survey of Music Similarity and Recommendation from Music Context Data. ACM Trans. Multimedia Comput. Commun. Appl. 10, 1 (Dec. 2013), 2:1--2:21.
[19]
Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the User Experience of Recommender Systems. User Modeling and User-Adapted Interaction 22, 4-5 (Oct. 2012), 441--504.
[20]
Y. Koren, R. Bell, and C. Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (Aug. 2009), 30--37.
[21]
Quoc V. Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. arXiv:1405.4053 {cs} (May 2014). arXiv: 1405.4053.
[22]
Jin Ha Lee. 2010. Crowdsourcing music similarity judgments using mechanical turk. In Proceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR 2010.
[23]
Greg Linden, Brent Smith, and Jeremy York. 2003. Amazon.Com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing 7, 1 (Jan. 2003), 76--80.
[24]
Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. 2011. Content-based Recommender Systems: State of the Art and Trends. In Recommender Systems Handbook. Springer, Boston, MA, 73--105.
[25]
Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being Accurate is Not Enough: How Accuracy Metrics Have Hurt Recommender Systems. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). ACM, New York, NY, USA, 1097--1101.
[26]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2 (NIPS'13). Curran Associates Inc., USA, 3111--3119.
[27]
Peter Organisciak and J. Stephen Downie. 2015. Improving Consistency of Crowdsourced Multimedia Similarity for Evaluation. In Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '15). ACM, New York, NY, USA, 115--118.
[28]
Parivash Pirasteh, Jason J. Jung, and Dosam Hwang. 2014. Item-Based Collaborative Filtering with Attribute Correlation: A Case Study on Movie Recommendation. In Intelligent Information and Database Systems (Lecture Notes in Computer Science). Springer, Cham, 245--252.
[29]
Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K. Lam, Sean M. McNee, Joseph A. Konstan, and John Riedl. 2002. Getting to Know You: Learning New User Preferences in Recommender Systems. In Proceedings of the 7th International Conference on Intelligent User Interfaces (IUI '02). ACM, New York, NY, USA, 127--134.
[30]
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2000. Application of Dimensionality Reduction in Recommender System - A Case Study. Technical Report TR-00-043. University of Minnesota.
[31]
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based Collaborative Filtering Recommendation Algorithms. In Proceedings of the Tenth International World Wide Web Conference. Hong Kong.
[32]
J. Ben Schafer, Joseph Konstan, and John Riedl. 1999. Recommender Systems in e-Commerce. In Proceedings of the 1st ACM Conference on Electronic Commerce (EC '99). ACM, New York, NY, USA, 158--166.
[33]
Markus Schedl, Peter Knees, and Fabien Gouyon. 2017. New Paths in Music Recommender Systems Research. In Proceedings of the Eleventh ACM Conference on Recommender Systems (RecSys '17). ACM, New York, NY, USA, 392--393.
[34]
Shilad Sen, Jesse Vig, and John Riedl. 2009. Learning to Recognize Valuable Tags. In Proceedings of the 14th International Conference on Intelligent User Interfaces (IUI '09). ACM, New York, NY, USA, 87--96.
[35]
Shilad Sen, Jesse Vig, and John Riedl. 2009. Tagommenders: Connecting Users to Items Through Tags. In Proceedings of the 18th International Conference on World Wide Web (WWW '09). ACM, New York, NY, USA, 671--680.
[36]
B. Smith and G. Linden. 2017. Two Decades of Recommender Systems at Amazon.com. IEEE Internet Computing 21, 3 (June 2017), 12--18.
[37]
R. Typke, R. C. Veltkamp, and F. Wiering. 2006. A Measure for Evaluating Retrieval Techniques based on Partially Ordered Ground Truth Lists. In 2006 IEEE International Conference on Multimedia and Expo. 1793--1796.
[38]
Jesse Vig, Shilad Sen, and John Riedl. 2012. The Tag Genome: Encoding Community Knowledge to Support Novel Interaction. ACM Trans. Interact. Intell. Syst. 2, 3 (Sept. 2012), 13:1--13:44.
[39]
C. Wang, A. Agrawal, X. Li, T. Makkad, E. Veljee, O. Mengshoel, and A. Jude. 2017. Content-based top-N recommendations with perceived similarity. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 1052--1057.
[40]
Chun Zeng, Chun-Xiao Xing, and Li-Zhu Zhou. 2003. Similarity Measure and Instance Selection for Collaborative Filtering. In Proceedings of the 12th International Conference on World Wide Web (WWW '03). ACM, New York, NY, USA, 652--658.
[41]
Renjie Zhou, Samamon Khemmarat, and Lixin Gao. 2010. The Impact of YouTube Recommendation System on Video Views. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement (IMC '10). ACM, New York, NY, USA, 404--410.
[42]
Radim Rehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45--50.

Cited By

View all
  • (2024)Improving Recommendations for Non-Mainstream Users by Addressing Subjective Item ViewsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664916(35-39)Online publication date: 27-Jun-2024
  • (2024)Incorporating Editorial Feedback in the Evaluation of News Recommender SystemsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664866(148-153)Online publication date: 27-Jun-2024
  • (2024)Shaping the Future of Content-based News Recommenders: Insights from Evaluating Feature-Specific Similarity MetricsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659560(201-211)Online publication date: 22-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems
September 2018
600 pages
ISBN:9781450359016
DOI:10.1145/3240323
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. collaborative filtering
  2. recommender systems
  3. related item recommendations
  4. rule mining
  5. similarity metrics
  6. user survey
  7. word2vec

Qualifiers

  • Research-article

Funding Sources

Conference

RecSys '18
Sponsor:
RecSys '18: Twelfth ACM Conference on Recommender Systems
October 2, 2018
British Columbia, Vancouver, Canada

Acceptance Rates

RecSys '18 Paper Acceptance Rate 32 of 181 submissions, 18%;
Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)46
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Improving Recommendations for Non-Mainstream Users by Addressing Subjective Item ViewsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664916(35-39)Online publication date: 27-Jun-2024
  • (2024)Incorporating Editorial Feedback in the Evaluation of News Recommender SystemsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664866(148-153)Online publication date: 27-Jun-2024
  • (2024)Shaping the Future of Content-based News Recommenders: Insights from Evaluating Feature-Specific Similarity MetricsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659560(201-211)Online publication date: 22-Jun-2024
  • (2024)User Perceptions of Diversity in Recommender SystemsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659555(212-222)Online publication date: 22-Jun-2024
  • (2024)Examining the merits of feature-specific similarity functions in the news domain using human judgmentsUser Modeling and User-Adapted Interaction10.1007/s11257-024-09412-234:4(995-1042)Online publication date: 7-Aug-2024
  • (2024)Non-binary evaluation of next-basket food recommendationUser Modeling and User-Adapted Interaction10.1007/s11257-023-09369-834:1(183-227)Online publication date: 1-Mar-2024
  • (2023)Semi-supervised Adversarial Learning for Complementary Item RecommendationProceedings of the ACM Web Conference 202310.1145/3543507.3583462(1804-1812)Online publication date: 30-Apr-2023
  • (2023)When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591785(942-952)Online publication date: 19-Jul-2023
  • (2023)FaiRIR: Mitigating Exposure Bias From Related Item Recommendations in Two-Sided PlatformsIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.316465510:3(1301-1313)Online publication date: Jun-2023
  • (2022)A Comparative Study of Data-Driven Models for Travel Destination CharacterizationFrontiers in Big Data10.3389/fdata.2022.8299395Online publication date: 7-Apr-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media