skip to main content
10.1145/3357384.3357817acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Learning to be Relevant: Evolution of a Course Recommendation System

Published: 03 November 2019 Publication History

Abstract

We present the evolution of a large-scale content recommendation platform for LinkedIn Learning, serving 645M+ LinkedIn users across several different channels (e.g., desktop, mobile). We address challenges and complexities from both algorithms and infrastructure perspectives. We describe the progression from unsupervised models that exploit member similarity with course content, to supervised learning models leveraging member interactions with courses, and finally to hyper-personalized mixed-effects models with several million coefficients. For all the experiments, we include metric lifts achieved via online A/B tests and illustrate the trade-offs between computation and storage requirements.

References

[1]
Mathieu Bastian, Matthew Hayes, William Vaughan, Sam Shah, Peter Skomoroch, Hyungjin Kim, Sal Uryasev, and Christopher Lloyd. 2014. LinkedIn Skills: Large-scale Topic Extraction and Inference. In Proceedings of the 8th ACM Conference on Recommender systems. ACM, 1--8.
[2]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 191--198.
[3]
Zhe Cui and Shivani Rao. 2017. Video to Skill Tagging using Transcripts under Weak Supervision. In Workshop on Learning with Limited Labeled Data, Neural Information Processing System .
[4]
Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000. Explaining Collaborative Fltering Recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. ACM, 241--250.
[5]
Jay Kreps. 2009. Project Voldemort. LinkedIn Blog. https://blog.linkedin.com/2009/04/01/project-voldemort-part-ii-how-it-works.
[6]
Pei Lee, Laks VS Lakshmanan, Mitul Tiwari, and Sam Shah. 2014. Modeling Impression Discounting in Large-scale Recommender Systems. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . ACM, 1837--1846.
[7]
David C Liu, Stephanie Rogers, Raymond Shiau, Dmitry Kislyuk, Kevin C Ma, Zhigang Zhong, Jenny Liu, and Yushi Jing. 2017. Related pins at pinterest: The evolution of a real-world recommender system. In Proceedings of the 26th International Conference on World Wide Web Companion . International World Wide Web Conferences Steering Committee, 583--592.
[8]
Michael Mandel. 2014. Connections as a Tool for Growth: Evidence from the LinkedIn Economic Graph. November. http://www. slideshare. net/linkedin/mandel-linked-in-connections-reportnov-2014 (2014).
[9]
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding-based news recommendation for millions of users. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1933--1942.
[10]
Andrew I Schein, Alexandrin Popescul, Lyle H Ungar, and David M Pennock. 2002. Methods and Metrics for Cold-start Recommendations. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 253--260.
[11]
Trinh Xuan Tuan and Tu Minh Phuong. 2017. 3D Convolutional Networks for Session-based Recommendation with Content Features. In Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 138--146.
[12]
Xianxing Zhang, Deepak Agarwal, Bee-Chung Chen, and Paul Ogilvie. 2016a. Building Recommender Systems using Photon ML. https://github.com/linkedin/photon-ml.
[13]
XianXing Zhang, Yitong Zhou, Yiming Ma, Bee-Chung Chen, Liang Zhang, and Deepak Agarwal. 2016b. Glmix: Generalized linear mixed models for large-scale response prediction. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 363--372.

Cited By

View all
  • (2024) Recommending the Rules Related to Course Grades in Education Sector using SK C A AR Algorithm 2024 Second International Conference on Emerging Trends in Information Technology and Engineering (ICETITE)10.1109/ic-ETITE58242.2024.10493294(1-10)Online publication date: 22-Feb-2024
  • (2023)Development Of Undergraduate Students Course Recommender System2023 3rd International Conference on Emerging Smart Technologies and Applications (eSmarTA)10.1109/eSmarTA59349.2023.10293642(1-8)Online publication date: 10-Oct-2023
  • (2022)Using AI-Based LinkedIn Video Platform for Personalised Learning: The Case at Infineon TechnologiesArtificial Intelligence Education in the Context of Work10.1007/978-3-031-14489-9_14(227-247)Online publication date: 29-Oct-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management
November 2019
3373 pages
ISBN:9781450369763
DOI:10.1145/3357384
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. glmix
  2. logistic regression
  3. machine learning
  4. recommender systems

Qualifiers

  • Research-article

Conference

CIKM '19
Sponsor:

Acceptance Rates

CIKM '19 Paper Acceptance Rate 202 of 1,031 submissions, 20%;
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)43
  • Downloads (Last 6 weeks)4
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024) Recommending the Rules Related to Course Grades in Education Sector using SK C A AR Algorithm 2024 Second International Conference on Emerging Trends in Information Technology and Engineering (ICETITE)10.1109/ic-ETITE58242.2024.10493294(1-10)Online publication date: 22-Feb-2024
  • (2023)Development Of Undergraduate Students Course Recommender System2023 3rd International Conference on Emerging Smart Technologies and Applications (eSmarTA)10.1109/eSmarTA59349.2023.10293642(1-8)Online publication date: 10-Oct-2023
  • (2022)Using AI-Based LinkedIn Video Platform for Personalised Learning: The Case at Infineon TechnologiesArtificial Intelligence Education in the Context of Work10.1007/978-3-031-14489-9_14(227-247)Online publication date: 29-Oct-2022
  • (2021)The State of the Art in Methodologies of Course Recommender Systems—A Review of Recent ResearchData10.3390/data60200186:2(18)Online publication date: 11-Feb-2021
  • (2021)New Performance Metrics for Offline Content-Based TV Recommender SystemAdvances in Bias and Fairness in Information Retrieval10.1007/978-3-030-78818-6_14(156-169)Online publication date: 25-Jun-2021
  • (2020)Course Recommendation System for Post-graduate (Masters in Science) AspirantsProceedings of International Conference on Computational Intelligence and Data Engineering10.1007/978-981-15-8767-2_31(375-386)Online publication date: 21-Dec-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media