skip to main content
10.1145/3301275.3302287acmconferencesArticle/Chapter ViewAbstractPublication PagesiuiConference Proceedingsconference-collections
research-article

Evaluating narrative-driven movie recommendations on Reddit

Published: 17 March 2019 Publication History

Abstract

Recommender systems have become omni-present tools that are used by a wide variety of users in everyday life tasks, such as finding products in Web stores or online movie streaming portals. However, in situations where users already have an idea of what they are looking for (e.g., 'The Lord of the Rings', but in space with a dark vibe), most traditional recommender algorithms struggle to adequately address such a priori defined requirements. Therefore, users have built dedicated discussion boards to ask peers for suggestions, which ideally fulfill the stated requirements. In this paper, we set out to determine the utility of well-established recommender algorithms for calculating recommendations when provided with such a narrative. To that end, we first crowdsource a reference evaluation dataset from human movie suggestions. We use this dataset to evaluate the potential of five recommendation algorithms for incorporating such a narrative into their recommendations. Further, we make the dataset available for other researchers to advance the state of research in the field of narrative-driven recommendations. Finally, we use our evaluation dataset to improve not only our algorithmic recommendations, but also existing empirical recommendations of IMDb. Our findings suggest that the implemented recommender algorithms yield vastly different suggestions than humans when presented with the same a priori requirements. However, with carefully configured post-filtering techniques, we can outperform the baseline by up to 100%. This represents an important first step towards more refined algorithmic narrative-driven recommendations.

Supplementary Material

MP4 File (p1-eberhard.mp4)

References

[1]
Gediminas Adomavicius, Ramesh Sankaranarayanan, Shahana Sen, and Alexander Tuzhilin. 2005. Incorporating contextual information in recommender systems using a multidimensional approach. ACM Transactions on Information Systems (TOIS) 23, 1 (2005), 103--145.
[2]
Gediminas Adomavicius and Alexander Tuzhilin. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering 17, 6 (2005), 734--749.
[3]
Gediminas Adomavicius and Alexander Tuzhilin. 2011. Context-Aware Recommender Systems. Springer US, Boston, MA, 217--253.
[4]
Gediminas Adomavicius, Alexander Tuzhilin, and Rong Zheng. 2011. REQUEST: A query language for customizing recommendations. Information Systems Research 22, 1 (2011), 99--117.
[5]
Chumki Basu, Haym Hirsh, William Cohen, et al. 1998. Recommendation as classification: Using social and content-based information in recommendation. In Aaai/iaai. 714--720.
[6]
Jason Michael Baumgartner. 2015. Reddit comment dataset. Website. (July 2015). https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment.
[7]
Toine Bogers. 2015. Searching for Movies: An Exploratory Analysis of Movie-related Information Needs. iConference 2015 Proceedings (2015).
[8]
Toine Bogers and Marijn Koolen. 2017. Defining and Supporting Narrative-driven Recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 238--242.
[9]
Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based N-gram Models of Natural Language. Comput. Linguist. 18, 4 (Dec. 1992), 467--479.
[10]
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards Conversational Recommender Systems. In KDD. 815--824.
[11]
Christina Christakou, Spyros Vrettos, and Andreas Stafylopatis. 2007. A hybrid movie recommender system based on neural networks. International Journal on Artificial Intelligence Tools 16, 05 (2007), 771--792.
[12]
Christian Desrosiers and George Karypis. 2011. A comprehensive survey of neighborhood-based recommendation methods. Recommender systems handbook (2011), 107--144.
[13]
Simon Funk. 2006. Netflix update: Try this at home. (2006).
[14]
Sumit Ghosh, Manisha Mundhe, Karina Hernandez, and Sandip Sen. 1999. Voting for Movies: The Anatomy of a Recommender System. In Proceedings of the Third Annual Conference on Autonomous Agents (AGENTS '99). ACM, New York, NY, USA, 434--435.
[15]
Negar Hariri, Bamshad Mobasher, and Robin Burke. 2013. Query-driven Context Aware Recommendation. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys '13). ACM, New York, NY, USA, 9--16.
[16]
Zan Huang, Xin Li, and Hsinchun Chen. 2005. Link Prediction Approach to Collaborative Filtering. In Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05). ACM, New York, NY, USA, 141--142.
[17]
Yehuda Koren. 2008. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 426--434.
[18]
Daniel Lamprecht, Florian Geigl, Tomas Karas, Simon Walk, Denis Helic, and Markus Strohmaier. 2015. Improving Recommender System Navigability Through Diversification: A Case Study of IMDb. In Proceedings of the 15th International Conference on Knowledge Technologies and Data-driven Business (i-KNOW '15). ACM, New York, NY, USA, Article 21, 8 pages.
[19]
Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning. 1188--1196.
[20]
Tariq Mahmood and Francesco Ricci. 2009. Improving Recommender Systems with Adaptive Conversational Strategies. In Proceedings of the 20th ACM Conference on Hypertext and Hypermedia (HT '09). ACM, New York, NY, USA, 73--82.
[21]
Harry Mak, Irena Koprinska, and Josiah Poon. 2003. Intimate: A web-based movie recommender using text categorization. In Web Intelligence, 2003. WI 2003. Proceedings. IEEE/WIC International Conference on. IEEE, 602--605.
[22]
Lorraine McGinty and James Reilly. 2011. On the evolution of critiquing recommenders. In Recommender Systems Handbook. Springer, 419--453.
[23]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
[24]
Kenta Oku, Shinsuke Nakajima, Jun Miyazaki, and Shunsuke Uemura. 2006. Context-aware SVM for context-dependent information recommendation. In Proceedings of the 7th international Conference on Mobile Data Management. IEEE Computer Society, 109.
[25]
Umberto Panniello, Alexander Tuzhilin, Michele Gorgoglione, Cosimo Palmisano, and Anto Pedone. 2009. Experimental comparison of pre-vs. post-filtering approaches in context-aware recommender systems. In Proceedings of the third ACM conference on Recommender systems. ACM, 265--268.
[26]
Arkadiusz Paterek. 2007. Improving regularized singular value decomposition for collaborative filtering. In Proceedings of KDD cup and workshop, Vol. 2007. 5--8.
[27]
Patrice Perny and Jean-Daniel Zucker. 2001. Preference-based search and machine learning for collaborative filtering: the "film-conseil" movie recommender system. Information, Interaction, Intelligence 1, 1 (2001), 9--48.
[28]
David Martin Powers. 2011. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. (2011).
[29]
Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994. GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW '94). ACM, New York, NY, USA, 175--186.
[30]
Ruslan Salakhutdinov and Andriy Mnih. 2007. Probabilistic Matrix Factorization. In Nips, Vol. 1. 2--1.
[31]
Ruslan Salakhutdinov and Andriy Mnih. 2008. Bayesian probabilistic matrix factorization using Markov chain Monte Carlo. In Proceedings of the 25th international conference on Machine learning. ACM, 880--887.
[32]
Gerard Salton and Michael J McGill. 1986. Introduction to modern information retrieval. (1986).
[33]
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based Collaborative Filtering Recommendation Algorithms. In Proceedings of the 10th International Conference on World Wide Web (WWW '01). ACM, New York, NY, USA, 285--295.
[34]
Paul Seitlinger, Dominik Kowald, Simone Kopeinik, Ilire Hasani-Mavriqi, Tobias Ley, and Elisabeth Lex. 2015. Attention Please! A Hybrid Resource Recommender Mimicking Attention-Interpretation Dynamics. In Proceedings of the 24th International Conference on World Wide Web (WWW '15 Companion). ACM, New York, NY, USA, 339--345.
[35]
Emine Yilmaz, Evangelos Kanoulas, and Javed A. Aslam. 2008. A Simple and Efficient Sampling Method for Estimating AP and NDCG. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '08). ACM, New York, NY, USA, 603--610.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces
March 2019
713 pages
ISBN:9781450362726
DOI:10.1145/3301275
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. crowdsourcing
  2. dataset
  3. narrative-driven recommendations

Qualifiers

  • Research-article

Conference

IUI '19
Sponsor:

Acceptance Rates

IUI '19 Paper Acceptance Rate 71 of 282 submissions, 25%;
Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)84
  • Downloads (Last 6 weeks)13
Reflects downloads up to 01 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)“If I Like BLANK, What Else Will I Like?”: Analyzing a Human Recommendation Community on RedditWisdom, Well-Being, Win-Win10.1007/978-3-031-57850-2_6(70-83)Online publication date: 15-Apr-2024
  • (2023)Large Language Model Augmented Narrative Driven RecommendationsProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608829(777-783)Online publication date: 14-Sep-2023
  • (2023)Movie Account Recommendation on InstagramACM Transactions on Internet Technology10.1145/357985223:1(1-21)Online publication date: 13-Jan-2023
  • (2022)A Virtual Assistant for the Movie Domain Exploiting Natural Language Preference Elicitation StrategiesAdjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3511047.3536407(7-12)Online publication date: 4-Jul-2022
  • (2021)Multi-Keyword Classification: A Case Study in Finnish Social Sciences Data ArchiveInformation10.3390/info1212049112:12(491)Online publication date: 25-Nov-2021
  • (2021)Personalised context-aware re-ranking in recommender systemConnection Science10.1080/09540091.2021.199791534:1(319-338)Online publication date: 3-Nov-2021
  • (2020)Tell Me What You WantProceedings of the 31st ACM Conference on Hypertext and Social Media10.1145/3372923.3404818(301-306)Online publication date: 13-Jul-2020
  • (2020)Mind the Gap: Exploring Shopping Preferences Across Fashion Retail ChannelsProceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3340631.3394866(257-265)Online publication date: 7-Jul-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media