Samuel E. L. Oliveira
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Recommender Systems are tools designed to help users find relevant information from the myriad of content available online. They work by actively suggesting items that are relevant to users according to their historical preferences or observed actions. Among recommender systems, top-N recommenders work by suggesting a ranking of N items that can be of interest to a user. Although a significant number of top-N recommenders have been proposed in the literature, they often disagree in their returned rankings, offering an opportunity for improving the final recommendation ranking by aggregating the outputs of different algorithms.
Rank aggregation was successfully used in a significant number of areas, but only a few rank aggregation methods have been proposed in the recommender systems literature. Furthermore, there is a lack of studies regarding rankings’ characteristics and their possible impacts on the improvements achieved through rank aggregation. This work presents an extensive two-phase experimental analysis of rank aggregation in recommender systems. In the first phase, we investigate the characteristics of rankings recommended by 15 different top-N recommender algorithms regarding agreement and diversity. In the second phase, we look at the results of 19 rank aggregation methods and identify different scenarios where they perform best or worst according to the input rankings’ characteristics.
Our results show that supervised rank aggregation methods provide improvements in the results of the recommended rankings in six out of seven datasets. These methods provide robustness even in the presence of a big set of weak recommendation rankings. However, in cases where there was a set of non-diverse high-quality input rankings, supervised and unsupervised algorithms produced similar results. In these cases, we can avoid the cost of the former in favor of the latter.
Supplemental Material
Available for Download
zip
oliveira.zip (1 MB)
Supplemental movie, appendix, image and software files for, Is Rank Aggregation Effective in Recommender Systems? An Experimental Analysis
- Javed A. Aslam and Mark Montague. 2001. Models for metasearch. In Proceedings of the 24th International ACM SIGIR Conference on Research and Development in Information Retrieval. 276--284.Google Scholar
- Avradeep Bhowmik and Joydeep Ghosh. 2017. LETOR methods for unsupervised rank aggregation. In Proceedings of the 26th International Conference on World Wide Web (WWW’17). International World Wide Web Conference Steering Committee, Republic and Canton of Geneva, Switzerland, 1331--1340.Google ScholarDigital Library
- Pavel Brazdil, Christophe Giraud-Carrier, Carlos Soares, and Ricardo Vilalta. 2008. Metalearning: Applications to Data Mining (1st ed.). Springer Publishing Company, Incorporated.Google Scholar
- Leo Breiman. 2001. Random forests. Mach. Learn. 45, 1 (2001), 5--32.Google ScholarDigital Library
- Christopher J. Burges, Robert Ragno, and Quoc V Le. 2007. Learning to rank with nonsmooth cost functions. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 193--200.Google Scholar
- Allison Chaney, David M. Blei, and Tina Eliassi-Rad. 2015. A probabilistic model for using social networks in personalized item recommendation. In Proceedings of the 9th ACM Conference on Recommender Systems. 43--50.Google ScholarDigital Library
- Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 758--759.Google ScholarDigital Library
- Tiago Cunha, Carlos Soares, and André C. P. L. F. de Carvalho. 2018. Metalearning and recommender systems: A literature review and empirical study on the algorithm selection problem for collaborative filtering. Inf. Sci. 423 (2018), 128--144. DOI:https://doi.org/10.1016/j.ins.2017.09.050Google ScholarDigital Library
- Edjalma Queiroz da Silva, Celso G. Camilo-Junior, Luiz Mario L. Pascoal, and Thierson C. Rosa. 2014. An evolutionary approach for combining results of recommender systems techniques based on collaborative filtering. Expert Syst. Appl. 53, C (July 2014), 204--218.Google Scholar
- Jean C. de Borda. 1781. Mémoire Sur les Élections au Scrutin. Histoire de l’Academie Royale des Sciences.Google Scholar
- Robert P. DeConde, Sarah Hawley, Seth Falcon, Nigel Clegg, Beatrice Knudsen, and Ruth Etzioni. 2006. Combining results of microarray experiments: A rank aggregation approach. Stat. Appl. Genet. Molec. Biol. 5, 1 (2006).Google Scholar
- Maunendra Sankar Desarkar, Sudeshna Sarkar, and Pabitra Mitra. 2016. Preference relations based unsupervised rank aggregation for metasearch. Expert Syst. Appl. 49, C (May 2016), 86--98.Google Scholar
- J. Ding, D. Han, J. Dezert, and Y. Yang. 2016. A new hierarchical ranking aggregation method. In Proceedings of the 19th International Conference on Information Fusion (FUSION’16). 1562--1569.Google Scholar
- Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. 2001. Rank aggregation methods for the web. In Proceedings of the 10th International Conference on World Wide Web (WWW’01). ACM, New York, NY, 613--622.Google Scholar
- Ronald Fagin, Ravi Kumar, Mohammad Mahdian, D. Sivakumar, and Erik Vee. 2006. Comparing partial rankings. SIAM J. Discret. Math. 20, 3 (Mar. 2006), 628--648.Google ScholarDigital Library
- Ronald Fagin, Ravi Kumar, and D. Sivakumar. 2003. Efficient similarity search and classification via rank aggregation. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 301--312.Google Scholar
- Mohamed Farah and Daniel Vanderpooten. 2007. An outranking approach for rank aggregation in information retrieval. In Proceedings of the 30th International ACM Conference on Research and Development in Information Retrieval. 591--598.Google ScholarDigital Library
- Edward A. Fox and Joseph A. Shaw. 1994. Combination of multiple searches. In Proceedings of the 2nd Text Retrieval Conference (TREC’94) 500-215 (1994), 243--252.Google Scholar
- Zeno Gantner, Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2011. MyMediaLite: A free recommender system library. In Proceedings of the 5th ACM Conference on Recommender Systems. 305--308.Google ScholarDigital Library
- David F. Gleich and Lek-heng Lim. 2011. Rank aggregation via nuclear norm minimization. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 60--68.Google Scholar
- Ken Goldberg, Theresa Roeder, Dhruv Gupta, and Chris Perkins. 2001. Eigentaste: A constant time collaborative filtering algorithm. Inf. Retr. 4, 2 (01 July 2001), 133--151.Google Scholar
- Prem Gopalan, Jake M. Hofman, and David M. Blei. 2015. Scalable recommendation with hierarchical Poisson factorization. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence. 326--335.Google Scholar
- Tom Griffiths. 2002. Gibbs Sampling in the Generative Model of Latent Dirichlet Allocation. Technical Report. Stanford University.Google Scholar
- Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. 2015. LibRec: A Java library for recommender systems. In Proceedings of the 23rd Conference on User Modeling, Adaptation, and Personalization (UMAP’15), Posters, Demos, Late-breaking Results, and Workshop.Google Scholar
- Guibing Guo, Jie Zhang, and Neil Yorke-Smith. 2013. A novel Bayesian similarity measure for recommender systems. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence. AAAI Press, 2619--2625.Google Scholar
- F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19 pages.Google Scholar
- Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In Proceedings of the 8th IEEE International Conference on Data Mining. 263--272.Google ScholarDigital Library
- Santosh Kabbur, Xia Ning, and George Karypis. 2013. Factored item similarity models for top-N recommender systems. In Proceedings of the 19th ACM International Conference on Knowledge Discovery and Data Mining. 659--667.Google ScholarDigital Library
- Raivo Kolde, Sven Laur, Priit Adler, and Jaak Vilo. 2012. Robust rank aggregation for gene list integration and meta-analysis. Bioinformatics 28, 4 (2012), 573--580.Google ScholarDigital Library
- Anísio Lacerda, Marco Cristo, Marcos André Gonçalves, Weiguo Fan, Nivio Ziviani, and Berthier Ribeiro-Neto. 2006. Learning to advertise. In Proceedings of the 29th International ACM Conference on Research and Development in Information Retrieval. 549--556.Google ScholarDigital Library
- Xue Li, Xinlei Wang, and Guanghua Xiao. 2017. A comparative study of rank aggregation methods for partial and top ranked lists in genomic applications. Briefings in Bioinformatics (22 Aug. 2017), bbx101. Retrieved from arXiv:/oup/backfile/content_public/journal/bib/pap/10.1093_bib_bbx101/1/bbx101.pdf.Google Scholar
- Dawen Liang, Jaan Altosaar, Laurent Charlin, and David M. Blei. 2016. Factorization meets the item embedding: Regularizing matrix factorization with item co-occurrence. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys’16). ACM, New York, NY, 59--66.Google Scholar
- Shangsong Liang, Ilya Markov, Zhaochun Ren, and Maarten de Rijke. 2018. Manifold learning for rank aggregation. In Proceedings of the World Wide Web Conference (WWW’18). International World Wide Web Conference Steering Committee, 1735--1744.Google ScholarDigital Library
- Shili Lin. 2010. Rank aggregation methods. Wiley Interdisc. Rev.: Comput. Stat. 2, 5 (2010), 555--570.Google ScholarDigital Library
- Shili Lin. 2010. Space oriented rank-based data integration. Stat. Appl. Gen. Molec. Biol. 9, 1 (2010).Google Scholar
- Tie-Yan Liu. 2009. Learning to rank for information retrieval. Found. Trends Inf. Retr. 3, 3 (2009), 225--331.Google ScholarDigital Library
- Ramon Lopes, Renato Assunção, and Rodrygo L. T. Santos. 2016. Efficient Bayesian methods for graph-based recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 333--340.Google Scholar
- Donald Metzler and W. Bruce Croft. 2007. Linear feature-based models for information retrieval. Inf. Retr. 10, 3 (2007), 257--274.Google ScholarCross Ref
- Xia Ning and George Karypis. 2011. SLIM: Sparse linear methods for top-N recommender systems. In Proceedings of the IEEE 11th International Conference on Data Mining. 497--506.Google ScholarDigital Library
- Shuzi Niu, Yanyan Lan, Jiafeng Guo, Xueqi Cheng, Lei Yu, and Guoping Long. 2015. Listwise approach for rank aggregation in crowdsourcing. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining. 253--262.Google ScholarDigital Library
- Samuel Oliveira, Victor Diniz, Anisio Lacerda, and Gisele Lobo Pappa. 2016. Evolutionary rank aggregation for recommender systems. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC’16). 255--262.Google ScholarDigital Library
- Samuel Oliveira, Pedro Valadares, Anisio Lacerda, and Gisele L. Pappa. 2018. Exploiting multiple recommenders to improve group recommendation. In Proceedings of the Brazilian Conference on Intelligent Systems.Google Scholar
- Aäron van den Oord, Sander Dieleman, and Benjamin Schrauwen. 2013. Deep content-based music recommendation. In Proceedings of the 26th International Conference on Neural Information Processing Systems. 2643--2651.Google Scholar
- Rong Pan, Yunhong Zhou, Bin Cao, Nathan N. Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. 2008. One-class collaborative filtering. In Proceedings of the 8th IEEE International Conference on Data Mining. 502--511.Google ScholarDigital Library
- Tao Qin, Xiubo Geng, and Tie-Yan Liu. 2010. A new probabilistic model for rank aggregation. In Proceedings of the 23rd International Conference on Neural Information Processing Systems. 1948--1956.Google Scholar
- Steffen Rendle. 2011. Context-aware Ranking with Factorization Models. Springer.Google Scholar
- Steffen Rendle. 2012. Factorization machines with libFM. ACM Trans. Intell. Syst. Technol. 3, 3, Article 57 (May 2012), 22 pages.Google ScholarDigital Library
- Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence. 452--461.Google ScholarDigital Library
- Marco Tulio Ribeiro, Nivio Ziviani, Edleno Silva De Moura, Itamar Hata, Anisio Lacerda, and Adriano Veloso. 2014. Multiobjective Pareto-efficient approaches for recommender systems. ACM Trans. Intell. Syst. Technol. 5, 4, Article 53 (Dec. 2014), 20 pages.Google ScholarDigital Library
- Francesco Ricci, Lior Rokach, and Bracha Shapira. 2011. Introduction to Recommender Systems Handbook. Springer.Google ScholarDigital Library
- Stephen Robertson. 2008. A new interpretation of average precision. In Proceedings of the 31st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 689--690.Google ScholarDigital Library
- Hinrich Schütze, Christopher D. Manning, and Prabhakar Raghavan. 2008. Introduction to Information Retrieval. Vol. 39. Cambridge University Press.Google Scholar
- B. Smyth and P. Cotter. 2000. A personalised TV listings service for the digital TV age. Knowl.-based Syst. 13, 2--3 (Apr. 2000), 53--59.Google Scholar
- Joshua M. Stuart, Eran Segal, Daphne Koller, and Stuart K. Kim. 2003. A gene-coexpression network for global discovery of conserved genetic modules. Science 302, 5643 (2003), 249--255.Google Scholar
- Gábor Takács and Domonkos Tikk. 2012. Alternating least squares for personalized ranking. In Proceedings of the 6th ACM Conference on Recommender Systems (RecSys’12). ACM, New York, NY, 83--90.Google ScholarDigital Library
- Roberto Torres, Sean M. McNee, Mara Abel, Joseph A. Konstan, and John Riedl. 2004. Enhancing digital libraries with TechLens. In Proceedings of the Joint ACM/IEEE Conference on Digital Libraries. 228--236.Google ScholarDigital Library
- Daniel Valcarce, Javier Parapar, and Álvaro Barreiro. 2017. Combining top-N recommenders with metasearch algorithms. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 805--808.Google ScholarDigital Library
- Lucas Pascotti Valem and Daniel Carlos Guimarães Pedronette. 2019. An unsupervised genetic algorithm framework for rank selection and fusion on image retrieval. In Proceedings of the International Conference on Multimedia Retrieval (ICMR’19). ACM, New York, NY, 58--62.Google ScholarDigital Library
- Javier Alvaro Vargas Muñoz, Ricardo da Silva Torres, and Marcos André Gonçalves. 2015. A soft computing approach for learning to aggregate rankings. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. 83--92.Google ScholarDigital Library
- Maksims N. Volkovs, Hugo Larochelle, and Richard S. Zemel. 2012. Learning to rank by aggregating expert preferences. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. 843--851.Google Scholar
- Maksims N. Volkovs and Richard S. Zemel. 2012. A flexible generative model for preference aggregation. In Proceedings of the 21st International Conference on World Wide Web (WWW’12). 479--488.Google Scholar
- Maksims N. Volkovs and Richard S. Zemel. 2013. CRF framework for supervised preference aggregation. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management. 89--98.Google Scholar
- Markus Weimer, Alexandros Karatzoglou, Quoc Viet Le, and Alex Smola. 2007. COFIRANK maximum margin matrix factorization for collaborative ranking. In Proceedings of the 20th International Conference on Neural Information Processing Systems. 1593--1600.Google Scholar
- Markus Weimer, Alexandros Karatzoglou, and Alex Smola. 2008. Improving maximum margin matrix factorization. Mach. Learn. 72, 3 (01 Sep. 2008), 263--276.Google Scholar
- Qiang Wu, Christopher J. C. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Inf. Retr. 13, 3 (01 June 2010), 254--270.Google Scholar
- Jun Xu and Hang Li. 2007. AdaRank: A boosting algorithm for information retrieval. In Proceedings of the 30th International ACM SIGIR Conference on Research and Development in Information Retrieval. 391--398.Google ScholarDigital Library
- Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúš Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. 2010. Solving the apparent diversity-accuracy dilemma of recommender systems. Proc. Nat. Acad. Sci. 107, 10 (2010), 4511--4515.Google ScholarCross Ref
- Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005. Improving recommendation lists through topic diversification. In Proceedings of the 14th International Conference on World Wide Web. 22--32.Google ScholarDigital Library
Index Terms
- Is Rank Aggregation Effective in Recommender Systems? An Experimental Analysis
Recommendations
Group recommendations with rank aggregation and collaborative filtering
RecSys '10: Proceedings of the fourth ACM conference on Recommender systemsThe majority of recommender systems are designed to make recommendations for individual users. However, in some circumstances the items to be selected are not intended for personal usage but for a group; e.g., a DVD could be watched by a group of ...
Effective rank aggregation for metasearching
Nowadays, mashup services and especially metasearch engines play an increasingly important role on the Web. Most of users use them directly or indirectly to access and aggregate information from more than one data sources. Similarly to the rest of the ...
Effective methods for increasing aggregate diversity in recommender systems
In order to make a recommendation, a recommender system typically first predicts a user's ratings for items and then recommends a list of items to the user which have high predicted ratings. Quality of predictions is measured by accuracy, that is, how ...
Comments