On discovering non-obvious recommendations: using unexpectedness and neighborhood selection methods in collaborative filtering systems

This paper proposes a number of studies in order to move the field of recommender systems beyond the traditional paradigm and the classical perspective of rating prediction accuracy. We contribute to existing helpful but less explored recommendation strategies and propose new approaches targeting to more useful recommendations for both users and businesses. Working toward this direction, we discuss the studies we have conducted so far and present our future research plans. The overall goal of this research program is to expand our focus from even more accurate rating predictions toward a more holistic experience for the users, by providing them with non-obvious but high quality recommendations and avoiding the over-specialization and concentration bias problems. In particular, we propose a new probabilistic neighborhood-based method as an improvement of the standard $k$-nearest neighbors approach, alleviating some of the most common problems of collaborative filtering recommender systems, based on classical metrics of dispersion and diversity as well as some newly proposed metrics. Furthermore, we propose a concept of unexpectedness in recommender systems and operationalize it by suggesting various mechanisms for specifying the expectations of the users and proposing a recommendation method for providing the users with unexpected but high quality personalized recommendations that fairly match their interests. Besides, in order to generate utility-based recommendations for Massive Open Online Courses (MOOCs) that better serve the educational needs of students, we study the satisfaction of users with online courses vis-a-vis student retention. Finally, we summarize the conclusions of the conducted studies, discuss the limitations of our work and also outline the managerial implications of the proposed stream of research.