Practical Design of Performant Recommender Systems using Large-scale Linear Programming-based Global Inference

Several key problems in web-scale recommender systems, such as optimal matching and allocation, can be formulated as large-scale linear programs (LPs) [4, 1]. These LPs take predictions from ML models such as probabilities of click, like, etc. as inputs and optimize recommendations made to users. In recent years, there has been an explosion in the research and development of large-scale recommender systems, but effective optimization of business objectives using the output of those systems remains a challenge. Although LPs can help optimize such business objectives, and algorithms for solving LPs have existed since the 1950s [5, 8], generic LP solvers cannot handle the scale of these problems. At LinkedIn, we have developed algorithms that can solve LPs of various forms with trillions of variables in a Spark-based library called "DuaLip" [7], a novel distributed solver that solves a perturbation of the LP problem at scale via gradient-based algorithms on the smooth dual of the perturbed LP. DuaLip has been deployed in production at LinkedIn and powers several very large-scale recommender systems. DuaLip is open-sourced and extensible in terms of features and algorithms.

In this first-of-its-kind tutorial, we will motivate the application of LPs to improve recommender systems, cover the theory of key LP algorithms [8, 6], and introduce DuaLip (https://github.com/linkedin/DuaLip), a highly performant Spark-based library that solves extreme-scale LPs for a large variety of recommender system problems. We will describe practical successes of large-scale LP in the industry [3, 2, 9] followed by a hands-on exercise to run DuaLip.