research-article

An Empirical Study on Recommendation with Multiple Types of Feedback

Authors:

Bee-Chung Chen,

Deepak AgarwalAuthors Info & Claims

KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Pages 283 - 292

https://doi.org/10.1145/2939672.2939690

Published: 13 August 2016 Publication History

Abstract

User feedback like clicks and ratings on recommended items provides important information for recommender systems to predict users' interests in unseen items. Most systems rely on models trained using a single type of feedback, e.g., ratings for movie recommendation and clicks for online news recommendation. However, in addition to the primary feedback, many systems also allow users to provide other types of feedback, e.g., liking or sharing an article, or hiding all articles from a source. These additional feedback potentially provides extra information for the recommendation models. To optimize user experience and business objectives, it is important for a recommender system to use both the primary feedback and additional feedback. This paper presents an empirical study on various training methods for incorporating multiple user feedback types based on LinkedIn recommendation products. We study three important problems that we face at LinkedIn: (1) Whether to send an email based on clicks and complaints, (2) how to rank updates in LinkedIn feeds based on clicks and hides and (3) how jointly optimize for viral actions and clicks in LinkedIn feeds. Extensive offline experiments on historical data show the effectiveness of these methods in different situations. Online A/B testing results further demonstrate the impact of these methods on LinkedIn production systems.

Supplementary Material

MP4 File (kdd2016_tang_empirical_study_01-acm.mp4)

Download
254.08 MB

References

[1]

G. Adomavicius and Y. Kwon. New Recommendation Techniques for Multicriteria Rating Systems. IEEE Expert / IEEE Intelligent Systems, 22:48--55, 2007.

Digital Library

[2]

R. Ando and T. Zhang. A high-performance semi-supervised learning method for text chunking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 1--9. Association for Computational Linguistics Morristown, NJ, USA, 2005.

Digital Library

[3]

A. Argyriou, T. Evgeniou, and M. Pontil. Multi-task feature learning. In Advances in Neural Information Processing Systems: Proceedings of the 2006 Conference, page 41. MIT Press, 2007.

Digital Library

[4]

A. Argyriou, C. Micchelli, M. Pontil, and Y. Ying. A spectral regularization framework for multi-task structure learning. Advances in Neural Information Processing Systems, 20, 2008.

Digital Library

[5]

A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183--202, 2009.

Digital Library

[6]

S. Bickel, M. Brückner, and T. Scheffer. Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference on Machine learning, pages 81--88. ACM New York, NY, USA, 2007.

Digital Library

[7]

J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman. Learning bounds for domain adaptation. Advances in Neural Information Processing Systems, 20, 2008.

Digital Library

[8]

J. Blitzer, R. McDonald, and F. Pereira. Domain adaptation with structural correspondence learning. In Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), 2006.

Digital Library

[9]

E. Bonilla, K. Chai, and C. Williams. Multi-task gaussian process prediction. Advances in Neural Information Processing Systems, 20:153--160.

Digital Library

[10]

W. Dai, G. Xue, Q. Yang, and Y. Yu. Co-clustering based classification for out-of-domain documents. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 210--219. ACM New York, NY, USA, 2007.

Digital Library

[11]

W. Dai, Q. Yang, G. Xue, and Y. Yu. Boosting for transfer learning. In Proceedings of the 24th international conference on Machine learning, pages 193--200. ACM New York, NY, USA, 2007.

Digital Library

[12]

H. Daume. Frustratingly easy domain adaptation. In Annual meeting-association for computational linguistics, volume 45, page 256, 2007.

[13]

H. Daume III and D. Marcu. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research, 26:101--126, 2006.

Digital Library

[14]

T. Evgeniou and M. Pontil. Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 109--117. ACM New York, NY, USA, 2004.

Digital Library

[15]

F. Fouss and M. Saerens. Evaluating performance of recommender systems: An experimental comparison. In Web Intelligence and Intelligent Agent Technology, 2008. WI-IAT'08. IEEE/WIC/ACM International Conference on, volume 1, pages 735--738. IEEE, 2008.

Digital Library

[16]

J. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33(1):1, 2010.

[17]

M. Harel and S. Mannor. Learning from multiple outlooks. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML '11, pages 401--408, New York, NY, USA, June 2011. ACM.

[18]

J. He and R. Lawrence. A graph-based framework for multi-task multi-view learning. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML '11, pages 25--32, New York, NY, USA, June 2011. ACM.

[19]

J. Huang, A. Smola, A. Gretton, K. Borgwardt, and B. Scholkopf. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19:601, 2007.

Digital Library

[20]

N. Hurley and M. Zhang. Novelty and diversity in top-n recommendation--analysis and evaluation. ACM Transactions on Internet Technology (TOIT), 10(4):14, 2011.

Digital Library

[21]

J. Jiang and C. Zhai. Instance weighting for domain adaptation in NLP. In Annual meeting-assosciation for computational linguistics, volume 45, page 264, 2007.

[22]

T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23--26, 2002, Edmonton, Alberta, Canada, pages 133--142, 2002.

Digital Library

[23]

N. Kawamae. Serendipitous recommendations via innovators. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 218--225. ACM, 2010.

Digital Library

[24]

K. Lakiotaki, N. F. Matsatsinis, and A. Tsoukiàs. Multicriteria user modeling in recommender systems. IEEE Intelligent Systems, 26(2):64--76, 2011.

Digital Library

[25]

N. Lathia, S. Hailes, L. Capra, and X. Amatriain. Temporal diversity in recommender systems. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 210--217. ACM, 2010.

Digital Library

[26]

N. Lawrence and J. Platt. Learning to learn with the informative vector machine. In Proceedings of the twenty-first international conference on Machine learning. ACM New York, NY, USA, 2004.

Digital Library

[27]

S. Lee, V. Chatalbashev, D. Vickrey, and D. Koller. Learning a meta-level prior for feature relevance from multiple related tasks. In Proceedings of the 24th international conference on Machine learning, pages 489--496. ACM New York, NY, USA, 2007.

Digital Library

[28]

X. Liao, Y. Xue, and L. Carin. Logistic regression with an auxiliary data source. In MACHINE LEARNING-INTERNATIONAL WORKSHOP THEN CONFERENCE-, volume 22, page 505, 2005.

Digital Library

[29]

P. Luo, F. Zhuang, H. Xiong, Y. Xiong, and Q. He. Transfer learning from multiple source domains via consensus regularization. In CIKM '08: Proceeding of the 17th ACM conference on Information and knowledge management, pages 103--112, New York, NY, USA, 2008. ACM.

Digital Library

[30]

K. Miettinen. Nonlinear Multiobjective Optimization. International Series in Operations Research & Management Science. Springer US, 1999.

[31]

J. Nocedal and S. Wright. Numerical Optimization. Springer series in operations research and financial engineering. Springer, 1999.

[32]

R. Raina, A. Battle, H. Lee, B. Packer, and A. Ng. Self-taught learning: Transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, pages 759--766. ACM New York, NY, USA, 2007.

Digital Library

[33]

M. T. Ribeiro, A. Lacerda, A. Veloso, and N. Ziviani. Pareto-efficient hybridization for multi-objective recommender systems. In Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys '12, pages 19--26, New York, NY, USA, 2012. ACM.

Digital Library

[34]

A. Schwaighofer, V. Tresp, and K. Yu. Learning Gaussian process kernels via hierarchical Bayes. Advances in Neural Information Processing Systems, 17:1209--1216, 2005.

Digital Library

[35]

G. Shani and A. Gunawardana. Evaluating recommendation systems. In Recommender systems handbook, pages 257--297. Springer, 2011.

[36]

M. Sugiyama, S. Nakajima, H. Kashima, P. von Bunau, and M. Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in Neural Information Processing Systems, 20, 2008.

Digital Library

[37]

P. Tseng and S. Yun. A coordinate gradient descent method for nonsmooth separable minimization. Mathematical Programming, 117(1--2):387--423, 2009.

[38]

C. Wang and S. Mahadevan. Heterogeneous domain adaptation using manifold alignment. In IJCAI, pages 1541--1546, 2011.

Digital Library

[39]

Q. Yang, Y. Chen, G.-R. Xue, W. Dai, and Y. Yu. Heterogeneous transfer learning for image clustering via the social web. ACL '09, pages 1--9, 2009.

Digital Library

[40]

C.-N. Ziegler, S. M. McNee, J. A. Konstan, and G. Lausen. Improving recommendation lists through topic diversification. In Proceedings of the 14th international conference on World Wide Web, pages 22--32. ACM, 2005.

Digital Library

Cited By

Zhang SChu HLi JZhou YWang SSun QNejdl WAuer SKarras OCha MMoens MNajork M(2025)DeMBR: Denoising Model with Memory Pruning and Semantic Guidance for Multi-Behavior RecommendationProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703532(521-529)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703532
Lee SKo GSong HJung JSerra ESpezzano F(2024)MuLe: Multi-Grained Graph Learning for Multi-Behavior RecommendationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679709(1163-1173)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679709
Liu DXian SWu YYang CTang XHe XMing ZHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)AutoDCS: Automated Decision Chain Selection in Deep Recommender SystemsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657818(956-965)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657818
Show More Cited By

Index Terms

An Empirical Study on Recommendation with Multiple Types of Feedback

Recommendations

Personalized Recommendation Algorithm Using User Demography Information
WKDD '09: Proceedings of the 2009 Second International Workshop on Knowledge Discovery and Data Mining

Personalized recommendation systems are web-based systems that aim at predicting a user’s interest on available products and services by relying on previously rated items and dealing with the problem of information and product overload. User demography ...
Personalized rough-set-based recommendation by integrating multiple contents and collaborative information

In recent years, explosively-growing information makes the users confused in making decisions among various kinds of products such as music, movies, books, etc. As a result, it is a challenging issue to help the user identify what she/he prefers. To ...
Userrank for item-based collaborative filtering recommendation

With the recent explosive growth of the Web, recommendation systems have been widely accepted by users. Item-based Collaborative Filtering (CF) is one of the most popular approaches for determining recommendations. A common problem of current item-based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 2016

2176 pages

ISBN:9781450342322

DOI:10.1145/2939672

General Chairs:
Balaji Krishnapuram
IBM
,
Mohak Shah
Bosch
,
Program Chairs:
Alex Smola
Amazon
,
Charu Aggarwal
IBM
,
Dou Shen
Baidu
,
Rajeev Rastogi
Amazon

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '16

Sponsor:

KDD '16: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 13 - 17, 2016

California, San Francisco, USA

Acceptance Rates

KDD '16 Paper Acceptance Rate 66 of 1,115 submissions, 6%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
930
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)6

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang SChu HLi JZhou YWang SSun QNejdl WAuer SKarras OCha MMoens MNajork M(2025)DeMBR: Denoising Model with Memory Pruning and Semantic Guidance for Multi-Behavior RecommendationProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703532(521-529)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703532
Lee SKo GSong HJung JSerra ESpezzano F(2024)MuLe: Multi-Grained Graph Learning for Multi-Behavior RecommendationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679709(1163-1173)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679709
Liu DXian SWu YYang CTang XHe XMing ZHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)AutoDCS: Automated Decision Chain Selection in Deep Recommender SystemsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657818(956-965)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657818
Xia LHuang CXu YDai PBo L(2024)Multi-Behavior Graph Neural Networks for Recommender SystemIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.3204775(1-15)Online publication date: 2024
https://doi.org/10.1109/TNNLS.2022.3204775
Pavankumar PKumar AShekar AAppaji V(2024)Optimizing Feedback Recommendation in Smart Training Framework using NLP2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP)10.1109/IDAP64064.2024.10710987(1-7)Online publication date: 21-Sep-2024
https://doi.org/10.1109/IDAP64064.2024.10710987
Zhang ZJia YHou YYu X(2024)Explicit Behavior Interaction with Heterogeneous Graph for Multi-behavior RecommendationData Science and Engineering10.1007/s41019-023-00238-39:2(133-151)Online publication date: 19-Jan-2024
https://doi.org/10.1007/s41019-023-00238-3
Bandreddy SEzeife CGidado A(2024)Mining Historical Multi-behavior Sequential Patterns for e-Commerce RecommendationInformation Integration and Web Intelligence10.1007/978-3-031-78093-6_5(55-74)Online publication date: 4-Dec-2024
https://doi.org/10.1007/978-3-031-78093-6_5
Xian DCui SWang BCui L(2023)H&M Personalized Fashion Product Recommendation Using LightgbmRankerProceedings of the 2nd International Academic Conference on Blockchain, Information Technology and Smart Finance (ICBIS 2023)10.2991/978-94-6463-198-2_23(201-208)Online publication date: 26-Jul-2023
https://doi.org/10.2991/978-94-6463-198-2_23
Meng CZhai CYang YZhang HLi XFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Parallel Knowledge Enhancement based Framework for Multi-behavior RecommendationProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615004(1797-1806)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615004
Meng CZhang HGuo WGuo HLiu HZhang YZheng HTang RLi XZhang RSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Hierarchical Projection Enhanced Multi-behavior RecommendationProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599838(4649-4660)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599838
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten