SDNN: Symmetric deep neural networks with lateral connections for recommender systems

Abstract

The recommender system is the key approach to alleviate the data explosion problem. Recently, with the rapid development of deep learning, there are several researches of employing deep neural networks (DNNs) on recommender systems. Most of these methods tend to capture the complex mapping relations between user-item representation and matching score via DNNs. These methods are mainly a pyramid structure which maps relations into low-dimensional space and then predicts the result by logistic regression. However, partial relations may be linearly indivisible in low-dimensional space. As we know, data that are hard to be separated in low-dimensional space can become much easier after being mapped into a high-dimensional space. Hence, motivated by the ladder network, we propose a Symmetric Deep Neural Networks (SDNN) with lateral connections, which can learn relations in both high-dimensional and low-dimensional spaces simultaneously. Moreover, considering that deep neural network is very inefficient in catching low-rank relations between users and items, we further combine SDNN with an improved deep matrix factorization model into a unified framework, and name this new model DualCF. Extensive experiments on three benchmark datasets are conducted and the results verify the effectiveness of SDNN and DualCF over state-of-the-art models for implicit feedback prediction.

Introduction

With the rapid development of Internet technology, we have entered the era of information explosion, where recommender system (RS) plays a vital role in helping consumers to find the most interesting and relevant items (e.g., products, service, movies) amongst a huge variety of options. Modern recommender systems are usually based on collaborative filtering (CF), which relies on historical behavioral data, such as ratings or clicks, to lean user preferences [30]. Among various CF techniques, matrix factorization (MF), which represents users and items by learning a latent space, provides superior performance over others and has become a de facto standard for latent factor model-based recommendation [1]. Earlier MF algorithms mainly rely on explicit ratings from users on items [28], [17]. Later, much research effort has been devoted to improving MF from different aspects, such as considering user or item biases [18], integrating extra data like social relations [32], and exploiting implicit feedback [25], [15].

In the past few years, deep learning has achieved great success on computer vision, speech recognition and nature language processing [4], [9]. Due to its good performance at representation learning, deep learning models have also been widely used in recommender systems, such as Restricted Boltzmann Machines [29], Autoencoders and the denoising autoencoders [19], [31]. With explicit historical ratings as the input, these methods tend to learn hidden structures that can reconstruct a user’s ratings. In addition to explicit ratings, implicit feedback has also been utilized in deep learning recommendation models. For example, Wu et al. [36] proposed a collaborative denoising autoencoder (CADE) for CF with implicit feedback, He et al. [11] proposed neural collaborative filtering (NCF) that models the interaction between user and item latent factors with a multi-layer perceptron neural network. Note that the above two works only exploit implicit feedback, in [37], by utilizing both explicit ratings and implicit feedback, a deep matrix factorization (DMF) model was proposed, which uses two distinct multi-layer perceptron (MLP) networks to learn user and item latent vectors and then models the user-item interaction by cosine similarity of the latent vectors. Recently, Deng et al. [5] categorized CF models into two types, i.e., representation learning-based CF and matching function learning-based CF, and proposed a Deep Collaborative Filtering (DeepCF) framework, which combines the strengths of these two types of CF models to achieve better performance.

The matching function learning-based CF method learns the relationship between user-item representation and matching score by stacked neural layers. As deep neural networks have proven to be capable of approximating any continuous function [13], such kind of method can capture complex relationship well. Nevertheless, deep neural networks applied on recommender systems are a pyramidal structure in general, which maps user and item latent vector into low-dimensional space to learn abstract and invariant feature. However, it may lead to partial features hard to be separated. Mapping to higher dimension may make it easier to classify those features that are complex in low-dimensional space. For example, as shown in Fig. 1, data represented in two-dimensional space are linearly indivisible, but when it is mapped into three dimensional-space, it can be divided by a plane. To better understand the usefulness of high-dimensional space representation, consider the example in Fig. 2, where the recommendation systems needs to rank three movies for a user for later recommendation. Suppose that a single dimension simply represents a kind of factor. The feature vector [0.7, 0.1, 0.9] indicates the user’s explicit preferences, including Prefer Detective Comics, Insensitive to Role and Like Adventure. Each movie is embed by a feature vector with different factors. When considering only two factors, Studio and Role, a ranking list can be obtained by conducting a dot product between user feature vector and item feature vector. It can be observed that the scores of The Dark Knight (0.96) and Superman Returns (0.95) are very close to each other, which makes it hard to make a proper recommendation (suppose Top-1 recommendation here). If we take one more factor, such as Genres, into consideration, another ranking list can be obtained and the score of different movies become easily distinguishable. We can simply choose the movie Superman Returns with the highest score of $1.95$ for recommendation. This example illustrates that high-dimensional space representation can help better differentiate data that are hard to be separated in low-dimensional space.

In this paper, we propose a novel Symmetric Deep Neural Networks (SDNN) with lateral connections to learn the complex relationship between user-item representation and matching score. SDNN contains two parts, a forward MLP is used to capture the invariant relations in low-dimensional space, while a reverse MLP combined with lateral connections is used to capture relations in higher dimension. Besides, with lateral connections, the reverse MLP can obtain the output of the hidden layer from the forward MLP layer by which allows the reverse MLP to include more information. Moreover, considering that deep neural network is very inefficient in catching low-rank relations between users and items, following the spirit of DeepCF, we combine SDNN with an improved deep matrix factorization model [5], to learn complex matching relations and low-rank relations between users and items jointly. We term the joint model as Dual Collaborative Filtering (DualCF). Our main contributions are summarized as follows,

• We propose a novel Symmetric Deep Neural Networks (SDNN) that can learn the mapping relations between user-item representation and matching score from two perspectives, i.e., high-dimensional space and low-dimensional space, to help improve the recommendation performance.

• We propose a novel framework, Dual Collaborative Filtering (DualCF), which combines SDNN with an improved deep matrix factorization model to learn complex mapping relations and low-rank relations between users and items jointly.

• Extensive experiments are conducted on three publicly available datasets and the experimental results demonstrate the effectiveness of our methods over several state-of-the-art models.

The remainder of this paper is organized as follows. Section 2 discusses related work; Section 3 formally defines the problem to be addressed in this work; Section 4 details our proposed methods SDNN and DualCF; Section 5 presents and analyzes the experimental results; Finally, Section 6 draws a conclusion.