Multi-Armed Bandits in Recommendation Systems: A survey of the state-of-the-art and future directions

Highlights

• A literature review of studies about MAB in recommender systems from 2000 to 2020.

• A discussion about MAB algorithms, datasets, and evaluation metrics.

• An updated panorama about the current practices and models applied in MAB researches.

• Discussion on the applicability of MAB models in the main recommendation challenges.

• Future directions to be explored by using MAB in the recommendation field.

Abstract

Recommender Systems (RSs) have assumed a crucial role in several digital companies by directly affecting their key performance indicators. Nowadays, in this era of big data, the information available about users and items has been continually updated and the application of traditional batch learning paradigms has become more restricted. In this sense, the current efforts in the recommendation field have concerned about this online environment and modeled their systems as a Multi-Armed Bandit (MAB) problem. Nevertheless, there is not a consensus about the best practices to design, perform, and evaluate the MAB implementations in the recommendation field. Thus, this work performs a systematic literature review (SLR) to shed light on this new topic. By inspecting 1327 articles published from the last twenty years (2000–2020), this work: (1) consolidates an updated picture of the main research conducted in this area so far; (2) highlights the most used concepts and methods, their core characteristics, and main limitations; and (3) evaluates the applicability of MAB-based recommendation approaches in some traditional RSs’ challenges, such as data sparsity, scalability, cold-start, and explainability. These discussions and analyzes also allow us to identify several gaps in the current literature, providing a strong guideline for future research.

Introduction

In the last three decades, the exponential growth of digital information on the Web has induced users to a stressful situation in which they do not know what to buy, listen to, or to watch. This problem is known in the literature as information overload and it has influenced several researchers to work on Recommendation Systems (RSs) to provide suggestions of items (e.g., movies, books, songs, etc.) and mitigate this problem (Shapira, Ricci, Kantor, & Rokach, 2011). Formally, RSs aim to estimate the user’s preference or even a specific rating for the available items in order to provide recommendations that increase both the user’s satisfaction and the system’s profit (Pathak, Garfinkel, Gopal, Venkatesan, & Yin, 2010). Distinct algorithms have been proposed so far based on the main recommendation strategies, such as Collaborative Filtering (CF), Demographic Filtering (DF), Content-based (CB), and Knowledge-based (KB) (Bobadilla et al., 2013, Jannach et al., 2010, Park et al., 2012).

Current efforts have proposed to handle the online recommendation task with concepts from the Reinforcement Learning (RL) field by modeling it as a Multi-Armed Bandit (MAB) problem (Wang et al., 2016, Wang, Wu et al., 2017, Wu et al., 2016, Zhao et al., 2013). Traditionally, MAB is defined as a sequential decision model that has to continually choose an action $a$ among a set of actions $\mathcal{A}$ – a.k.a. arms. The selection of action $a\in \mathcal{A}$ in a trial $t$ brings out in a certain reward $R_t\left(a_t\right)\in \mathbb{R}$, which can be summarized as a real number. The main goal is to maximize the reward returned ${\sum }_{t=1}^T{R}_t\left(a_t\right)$ for $T$ trials. In the recommendation domain, items available are usually modeled as the arms to be pulled. Selecting an arm is equivalent to recommending an item, and the reward is the user’s response (e.g., clicks, acceptance, satisfaction, etc.). Similar to traditional RL scenarios, to achieve its goal, the bandit model should balance the exploitation and exploration dilemma. While exploitation just means pull arms with the highest rewards in the past, maximizing the system short-term reward, exploration is achieved by recommending other arms to improve the knowledge available about users and items to maximize the system long-term reward (Sanz-Cruzado et al., 2019, Zhao et al., 2013).

The MAB problem has attracted a lot of attention from both industry and academy in the recommendation field. In the academy, it is possible to notice that more than 50% of all publications about this topic was only proposed in the last five years, as shown in Fig. 1. Similarly, in the industry, recent talks of research leaders of Netflix, Pandora, and Spotify in the main conferences, such as ACM Recommender Systems (RecSys), ACM Conference on Research and Development in Information Retrieval (SIGIR), and Web Conference (WWW), have revealed the growing interest of companies on this topic to handle the online recommendation task, especially.

However, even with this growing number of new publications available, there is no work that aims to map the main advances in the area, explain the main concepts, and clarify the best practices. Therefore, this work performs a systematic literature review (SLR) about MAB in the recommendation field to achieve three main goals:

• Provide a summary overview of the most important research in this area;

• Highlight the most popular concepts and methods, their core characteristics, and their main limitations to provide future directions of the field to guide the next research questions;

• Discuss the applicability of MAB-based recommendation approaches in some traditional RSs’ challenges, such as the data sparsity, scalability, cold-start, privacy, and explainability.

Searching all conferences available in the Google Scholar from 2000 to 2020, our SLR identified 1327 articles based on three main strings designed according to our goals and research questions. Then, we conducted two main reading steps to filter and identify the most relevant studies. While the first step performs a short reading by analyzing titles, year, conference, and abstract, the second one performs a more complete analysis by reading the introduction, experimentation, and conclusion of each work. In the first reading, only 408 papers (30.75%) were selected for the second step. Then, in the second stage, other 178 papers were rejected and only 230 papers were selected as relevant studies about MAB in the recommendation field.¹ These works were deeply studied to achieve our three main goals. They were read to improve our knowledge but only those with an experimental setup were used to fill a data extraction form designed to catch the current practices in the literature.

In general, the application of our SLR provides distinct contributions for academia and industry. In this paper, we highlight: (1) the main conferences where MAB studies have been published; (2) the most usual scenarios simulated by the publications; (3) the datasets applied for these studies; (4) the main algorithms usually applied to address the MAB problem; and (5) the main advances by combining traditional bandit algorithms with concepts of recommendation systems. Our work also identifies several gaps in the current literature and proposes relevant future directions. For instance, we noticed the absence of a strict evaluation criteria that reflect the traditional RS goals. Relevant metrics usually related to user satisfaction or engagement have been neglected by only applying the traditional evaluation criteria of learning algorithms based on rewards (or regrets). Moreover, it is not clear how the most relevant challenges of the recommendation field can affect bandit algorithms. We discussed common problems that still are trends in the field, like sparsity, scalability, cold-start, privacy, and explainability, by pointing out the future directions for research in these topics.

The remainder of this paper is organized as follows. First, Section 2 highlights background concepts about the traditional MAB problem. Then, Section 3 presents the SLR process by showing each step performed by our inspection. Section 4 organizes the main discussion of our paper by highlighting the works developed so far, the current evaluation criteria, and how MAB has faced the current challenges of the field. These discussions allow us to point out the main future directions in Section 5. Finally, Section 6 presents our main conclusions.