skip to main content
10.1145/3450614.3461687acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
extended-abstract

Recommending Videos in Cold Start With Automatic Visual Tags

Published: 22 June 2021 Publication History

Abstract

This paper addresses the so-called New Item problem in video Recommender Systems, as part of Cold Start. New item problem occurs when a new item is added to the system catalog, and the recommender system has no or little data describing that item. This could cause the system to fail to meaningfully recommend the new item to the users. We propose a novel technique that can generate cold start recommendation by utilizing automatic visual tags, i.e., tags that are automatically annotated by deeply analyzing the content of the videos and detecting faces, objects, and even celebrities within the videos. The automatic visual tags do not need any human involvement and have been shown to be very effective in representing the video content. In order to evaluate our proposed technique, we have performed a set of experiments using a large dataset of videos. The results have shown that the automatically extracted visual tags can be incorporated into the cold start recommendation process and achieve superior results compared to the recommendation based on human-annotated tags.

Supplementary Material

MP4 File (UMAP-ADJ21-umap09lb.mp4)
This paper addresses the so-called New Item problem in video Recommender Systems, as part of Cold Start. New item problem occurs when a new item is added to the system catalog, and the recommender system has no or little data describing that item. This could cause the system to fail to meaningfully recommend the new item to the users. We propose a novel technique that can generate cold-start recommendations by utilizing automatic visual tags, i.e., tags that are automatically annotated by deeply analyzing the content of the videos and detecting faces, objects, and even celebrities within the videos. The automatic visual tags do not need any human involvement and have been shown to be very effective in representing the video content. The results of our experiments have shown that the automatically extracted visual tags can be incorporated into the cold start recommendation process and achieve superior results compared to the recommendation based on human-annotated tags.

References

[1]
Benjamin Adrian, Leo Sauermann, and Thomas Roth-Berghofer. 2007. Contag: A semantic tag recommendation system. Proceedings of I-Semantics 7 (2007), 297–304.
[2]
Syed M Ali, Gopal K Nayak, Rakesh K Lenka, and Rabindra K Barik. 2018. Movie recommendation system using genome tags and content-based filtering. In Advances in Data and Information Sciences. Springer, 85–94.
[3]
Fahad Anwar, Naima Iltaf, Hammad Afzal, and Haider Abbas. 2019. A Deep Learning Framework to Predict Rating for Cold Start Item Using Item Metadata. In 2019 IEEE 28th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE). IEEE, 313–319.
[4]
Edoardo Ardizzone, Marco La Cascia, and Davide Molinelli. 1996. Motion and color-based video indexing and retrieval. In Proceedings of 13th International Conference on Pattern Recognition, Vol. 3. IEEE, 135–139.
[5]
Aparna Bharati, Richa Singh, Mayank Vatsa, and Kevin W Bowyer. 2016. Detecting facial retouching using supervised deep learning. IEEE Transactions on Information Forensics and Security 11, 9(2016), 1903–1913.
[6]
Toine Bogers. 2018. Tag-based recommendation. In Social Information Access. Springer, 441–479.
[7]
D. Brezeale and D. J. Cook. 2008. Automatic Video Classification: A Survey of the Literature. Trans. Sys. Man Cyber Part C 38, 3 (May 2008), 416–430. https://doi.org/10.1109/TSMCC.2008.919173
[8]
Qiang Chen, Junshi Huang, Rogerio Feris, Lisa M Brown, Jian Dong, and Shuicheng Yan. 2015. Deep domain adaptation for describing people based on fine-grained clothing attributes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5315–5324.
[9]
Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems. 39–46.
[10]
James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, 2010. The YouTube video recommendation system. In Proceedings of the fourth ACM conference on Recommender systems. 293–296.
[11]
Yashar Deldjoo, Mehdi Elahi, Paolo Cremonesi, Franca Garzotto, Pietro Piazzolla, and Massimo Quadrana. 2016. Content-Based Video Recommendation System Based on Stylistic Visual Features. Journal on Data Semantics(2016), 1–15. https://doi.org/10.1007/s13740-016-0060-9
[12]
Yashar Deldjoo, Mehdi Elahi, Paolo Cremonesi, Farshad Bakhshandegan Moghaddam, and Andrea Luigi Edoardo Caielli. 2016. How to combine visual features with tags to improve movie recommendation accuracy?. In International conference on electronic commerce and web technologies. Springer, 34–45.
[13]
Yashar Deldjoo, Mehdi Elahi, Massimo Quadrana, and Paolo Cremonesi. 2018. Using visual features based on MPEG-7 and deep learning for movie recommendation. International journal of multimedia information retrieval 7, 4(2018), 207–219. https://doi.org/10.1007/s13735-018-0155-1
[14]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
[15]
Mehdi Elahi, Matthias Braunhofer, Tural Gurbanov, and Francesco Ricci. 2018. User Preference Elicitation, Rating Sparsity and Cold Start.
[16]
Mehdi Elahi, Yashar Deldjoo, Farshad Bakhshandegan Moghaddam, Leonardo Cella, Stefano Cereda, and Paolo Cremonesi. 2017. Exploring the semantic gap for movie recommendations. In Proceedings of the Eleventh ACM Conference on Recommender Systems. 326–330.
[17]
Manuel Enrich, Matthias Braunhofer, and Francesco Ricci. 2013. Cold-Start Management with Cross-Domain Collaborative Filtering and Tags. In Proceedings of the 13th International Conference on E-Commerce and Web Technologies. Springer, 101–112. https://doi.org/10.1007/978-3-642-39878-0_10
[18]
Ignacio Fernández-Tobías and Iván Cantador. 2014. Exploiting Social Tags in Matrix Factorization Models for Cross-domain Collaborative Filtering. In Proceedings of the 1st Workshop on New Trends in Content-based Recommender Systems, Foster City, California, USA. 34–41.
[19]
Zeno Gantner, Lucas Drumond, Christoph Freudenthaler, Steffen Rendle, and Lars Schmidt-Thieme. 2010. Learning attribute-to-feature mappings for cold-start recommendations. In 2010 IEEE International Conference on Data Mining. IEEE, 176–185.
[20]
Mouzhi Ge, Mehdi Elahi, Ignacio Fernaández-Tobías, Francesco Ricci, and David Massimo. 2015. Using tags and latent factors in a food recommender system. In Proceedings of the 5th International Conference on Digital Health 2015. 105–112.
[21]
Fatih Gedikli and Dietmar Jannach. 2013. Improving recommendation accuracy based on item-specific tag preferences. ACM Transactions on Intelligent Systems and Technology (TIST) 4, 1(2013), 1–19.
[22]
F Maxwell Harper and Joseph A Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article Article 19 (Dec. 2015), 19 pages. https://doi.org/10.1145/2827872
[23]
Naieme Hazrati and Mehdi Elahi. 2020. Addressing the New Item problem in video recommender systems by incorporation of visual features with restricted Boltzmann machines. Expert Systems (2020), e12645.
[24]
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web. 507–517.
[25]
Ruining He and Julian McAuley. 2016. VBPR: visual bayesian personalized ranking from implicit feedback. In Thirtieth AAAI Conference on Artificial Intelligence.
[26]
Weiming Hu, Nianhua Xie, Li, Xianglin Zeng, and Stephen Maybank. 2011. A Survey on Visual Content-Based Video Indexing and Retrieval. Trans. Sys. Man Cyber Part C 41, 6 (Nov. 2011), 797–819. https://doi.org/10.1109/TSMCC.2011.2109710
[27]
Shatha Jaradat. 2017. Deep cross-domain fashion recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems. 407–410.
[28]
Yehuda Koren and Robert Bell. 2011. Advances in Collaborative Filtering. Springer US, Boston, MA, 145–186. https://doi.org/10.1007/978-0-387-85820-3_5
[29]
Huizhi Liang, Yue Xu, Yuefeng Li, and Richi Nayak. 2009. Tag Based Collaborative Filtering for Recommender Systems. In Rough Sets and Knowledge Technology, 4th International Conference, RSKT 2009, Gold Coast, Australia, July 14-16, 2009. Proceedings. 666–673. https://doi.org/10.1007/978-3-642-02962-2_84
[30]
Blerina Lika, Kostas Kolomvatsos, and Stathes Hadjiefthymiades. 2014. Facing the cold start problem in recommender systems. Expert Systems with Applications 41, 4 (2014), 2065–2073.
[31]
Si Liu, Zheng Song, Guangcan Liu, Changsheng Xu, Hanqing Lu, and Shuicheng Yan. 2012. Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 3330–3337.
[32]
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1096–1104.
[33]
Pasquale Lops, Dietmar Jannach, Cataldo Musto, Toine Bogers, and Marijn Koolen. 2019. Trends in content-based recommendation. User Modeling and User-Adapted Interaction 29, 2 (2019), 239–249.
[34]
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, Nov (2008), 2579–2605.
[35]
Marcelo Garcia Manzato. 2013. GSVD++: Supporting Implicit Feedback on Recommender Systems with Metadata Awareness(SAC ’13). Association for Computing Machinery, New York, NY, USA, 908–913. https://doi.org/10.1145/2480362.2480536
[36]
David Massimo, Mehdi Elahi, Mouzhi Ge, and Francesco Ricci. 2017. Item contents good, user tags better: Empirical evaluation of a food recommender system. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. 373–374.
[37]
Pablo Messina, Vicente Dominguez, Denis Parra, Christoph Trattner, and Alvaro Soto. 2019. Content-based artwork recommendation: integrating painting metadata with neural and manually-engineered visual features. User Modeling and User-Adapted Interaction 29, 2 (2019), 251–290.
[38]
Pablo Messina, Vicente Dominquez, Denis Parra, Christoph Trattner, and Alvaro Soto. 2018. Exploring Content-based Artwork Recommendation with Metadata and Visual Features. User Modeling and User-Adapted Interaction (UMUAI) 29, 2 (July 2018), 251–290. https://doi.org/10.1007/s11257-018-9206-9
[39]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1301.3781
[40]
Farshad Bakhshandegan Moghaddam and Mehdi Elahi. 2019. Cold start solutions for recommendation systems. Big Data Recommender Systems, Recent Trends and Advances. IET (2019).
[41]
H. R. Naphide and Thomas Huang. 2001. A probabilistic framework for semantic video indexing, filtering, and retrieval. IEEE Transactions on Multimedia 3, 1 (March 2001), 141–151. https://doi.org/10.1109/6046.909601
[42]
Abhishek A Patwardhan, Santanu Das, Sakshi Varshney, Maunendra Sankar Desarkar, and Debi Prosad Dogra. 2019. ViTag: Automatic video tagging using segmentation and conceptual inference. In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM). IEEE, 271–276.
[43]
Michael J Pazzani and Daniel Billsus. 2007. Content-based recommendation systems. In The adaptive web. Springer, 325–341.
[44]
Zeeshan Rasheed, Yaser Sheikh, and Mubarak Shah. 2005. On the Use of Computable Features for Film Classification. IEEE Trans. Cir. and Sys. for Video Technol. 15, 1 (Jan. 2005), 52–64. https://doi.org/10.1109/TCSVT.2004.839993
[45]
Mohammad Hossein Rimaz, Mehdi Elahi, Farshad Bakhshandegan Moghadam, Christoph Trattner, Reza Hosseini, and Marko Tkalčič. 2019. Exploring the Power of Visual Features for the Recommendation of Movies(UMAP ’19). Association for Computing Machinery, New York, NY, USA, 303–308. https://doi.org/10.1145/3320435.3320470
[46]
Mohammad H Rimaz, Reza Hosseini, Mehdi Elahi, and Farshad Bakhshandegan Moghaddam. [n.d.]. AudioLens: Audio-Aware Video Recommendation for Mitigating New Item Problem. ([n. d.]).
[47]
Jose San Pedro and Stefan Siersdorfer. 2009. Ranking and Classifying Attractiveness of Photos in Folksonomies(WWW ’09). Association for Computing Machinery, New York, NY, USA, 771–780. https://doi.org/10.1145/1526709.1526813
[48]
Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Elahi. 2018. Current challenges and visions in music recommender systems research. International Journal of Multimedia Information Retrieval 7, 2(2018), 95–116.
[49]
Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. 2019. Annotating objects and relations in user-generated videos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval. 279–287.
[50]
Jiangbo Shu, Xiaoxuan Shen, Hai Liu, Baolin Yi, and Zhaoli Zhang. 2018. A content-based recommendation algorithm for learning resources. Multimedia Systems 24, 2 (2018), 163–173.
[51]
Cees G.M. Snoek and Marcel Worring. 2005. Multimodal Video Indexing: A Review of the State-of-the-art. Multimedia Tools and Applications 25, 1 (01 Jan 2005), 5–35. https://doi.org/10.1023/B:MTAP.0000046380.27575.a5
[52]
Hridya Sobhanam and AK Mariappan. 2013. Addressing cold start problem in recommender systems using association rules and clustering technique. In 2013 International Conference on Computer Communication and Informatics. IEEE, 1–5.
[53]
Michele Svanera, Mattia Savardi, Alberto Signoroni, András Bálint Kovács, and Sergio Benini. 2018. Who is the director of this movie? Automatic style recognition based on shot features. CoRR abs/1807.09560(2018). arxiv:1807.09560http://arxiv.org/abs/1807.09560
[54]
Nava Tintarev and Judith Masthoff. 2011. Designing and evaluating explanations for recommender systems. In Recommender systems handbook. Springer, 479–510.
[55]
Marko Tkalčič, Nima Maleki, Matevž Pesek, Mehdi Elahi, Francesco Ricci, and Matija Marolt. 2017. A Research Tool for User Preferences Elicitation with Facial Expressions(RecSys ’17). ACM, New York, NY, USA, 353–354. https://doi.org/10.1145/3109859.3109978
[56]
Marko Tkalčič, Nima Maleki, Matevž Pesek, Mehdi Elahi, Francesco Ricci, and Matija Marolt. 2019. Prediction of Music Pairwise Preferences from Facial Expressions(IUI ’19). ACM, New York, NY, USA, 150–159. https://doi.org/10.1145/3301275.3302266
[57]
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3733–3742.
[58]
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR) 52, 1 (2019), 1–38.

Cited By

View all
  • (2024)Predicting movies’ eudaimonic and hedonic scoresInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10361061:2Online publication date: 12-Apr-2024
  • (2023)An overview of video recommender systems: state-of-the-art and research issuesFrontiers in Big Data10.3389/fdata.2023.12816146Online publication date: 30-Oct-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UMAP '21: Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization
June 2021
431 pages
ISBN:9781450383677
DOI:10.1145/3450614
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2021

Check for updates

Author Tags

  1. Cold Start
  2. Recommender Systems
  3. Visual Features
  4. Visual Tags

Qualifiers

  • Extended-abstract
  • Research
  • Refereed limited

Funding Sources

  • Industry partners and Research council of Norway

Conference

UMAP '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 162 of 633 submissions, 26%

Upcoming Conference

UMAP '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)0
Reflects downloads up to 30 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Predicting movies’ eudaimonic and hedonic scoresInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10361061:2Online publication date: 12-Apr-2024
  • (2023)An overview of video recommender systems: state-of-the-art and research issuesFrontiers in Big Data10.3389/fdata.2023.12816146Online publication date: 30-Oct-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media