Knowledge Graph enhanced Neural Collaborative Filtering with Residual Recurrent Network
Introduction
As the amount of data from different platforms grows at an unprecedented rate, there is an enormous demand for the recommendation system to provide relevant information tailored to users’ interests or preferences. For example, 11,914 movies are released around the world in 2018.1 It is impossible for users to watch all available movies to identify their interested ones. As a result, the recommender system is becoming more and more crucial to help users discover personalised content from these ever-growing corpus of data. Due to their extraordinary capability of exploiting collective wisdom and experiences, Collaborative Filtering (CF) algorithms have proven to be the most widely used recommendation technology [37]. Though widely studied in the past, traditional collaborative filtering, however, suffers from severe user-item interaction data sparse issues [8].
Inspired by the recent popularity of deep neural networks, Neural Collaborative Filtering (NCF) is proposed, as a new class of CF methods, cast the traditional MF (Matrix Factorization) algorithm into an overall neural framework [13], [12], [6], [20]. In general, learnable NCF models consist of two key components: 1) embedding, which converts users and items to vectorised embeddings, and 2) interaction modelling, which encodes historical interactions over the user/item embeddings. For example, in [13], the inner product of the MF interaction function is replaced by deep nonlinear networks. And the Euclidean distance metric is used in translation-based CF as the interaction function [34]. However, these methods lack the ability of producing high-quality user/item embeddings for collaborative filtering. One critical reason is that most existing methods build the embedding function with the descriptive features only (e.g., ID or categories). As a result, when the neural-based interaction function goes deeper to capture complex user-item relationship, these methods tend to overfit and make the sparsity problem even worse.
The rapid explosion of the online social network platforms offer new opportunities for CF. Therefore, researchers propose using auxiliary data in recommender system to enrich the semantics of user or item embedding, such as social networks, attributes, and multimedia data [28], [15], [18], [23], [27]. In previous works, these auxiliary information is usually organised in an unstructured manner, and they only available in specific platforms. Recently, Knowledge Graphs (KGs) have attracted increasing attention, which usually consist of fruitful connected facts about items organised in graph structure [22], [40], [3]. By introducing semantic relatedness among items, KG is a useful tool to help mine items’ hidden connections and provide better recommendation results. Extra user-item connectivity context information derived from KG endows recommender systems the ability of reasoning and alleviates sparsity [45]. Taking movie recommendation as an example in Fig. 1, a user Bob is connected to the movie “The Revenant”, since she likes the movie “Inception” by the same actor “Leonardo DiCaprio”. Such connectivity helps to reason about unseen user-item interactions (i.e., a potential recommendation) by synthesising information from long paths, which are complementary to sparse user-item interaction data.
There are mainly two kinds of knowledge graph enhanced recommender systems: path-based model and entity embedding based models (1) Path-based models usually use pre-defined meta-path with specific connectivity patterns to capture different semantics similarities between users and items carried in KGs. Meta-path exploits KG in an intuitive way, however, they heavily rely on manually-designed features to capture path semantics, and they fail to automatically uncover and reason on unseen connectivity patterns. More importantly, handcrafted features can hardly exploit all of the possible entity connections, thus limiting the recommendation results’ quality. (2) For KG embedding based models, the KG is usually pre-processed by Knowledge Graph Embedding (KGE) methods [42], such as TransE [2] and TransR [19], and then the learned entity embeddings are used to regularise the representations of items. KGE start with the assumption that similar entities are likely to have similar relational roles, where each triplet in KG is interpreted as , indicates a fact that head entity h and tail entity t have the relationship r. Here we use the boldfaces to denote their embeddings. Their primary focus, therefore, depends on learning from relational triplets of entities. However, triple-level learning has the limitation of low expressiveness. These methods only consider direct relations between entities, rather than the long-term path connectivity. Furthermore, simple entity embedding disregards the semantic relations of entities that are connected by paths, which has been widely used in path based methods.
To address these issues, we aim to find a new data-driven method that does not rely on manually designed meta-path features or simple entity embedding, yet can still obtain the hidden semantics of both entities and paths within KGs for the recommendation. Our main idea is to learn context-based path embedding that explicitly exploit long-term dependencies relational context in KG, which characterise a three-way interaction of the form: (user, KG, item) tailored for the recommendation task. Take the KG based movie recommendation, for example, where Bob’s preference can be inferred by the relational paths: 1) Bob Inception Fantasy Batman Begins, 2) Bob Inception Christopher Nolan Batman Begins. Learning the representation of above relational paths can exploit more relational dependencies than simple entity triplets, while still maintain the local relational information of entities. Moreover, without time-consuming manual meta-path design, path embedding can still exploit multi-hop relation among entities. And the low-dimension path embedding can also facilitate the downstream process with more flexibility. This example also highlights that different paths connecting a same entity pair often carry relations of different semantics. User behaviours have the polysemy property, since an action to an item may have a variety of meanings in different contextual scenarios. Typically, they are of different importance in characterising user tastes over items, i.e., certain paths can better describe user preferences than the others. In the example, Bob’s preference over Batman Begins may be driven more by his interest in the genre than by his favour for the director. To fully exploit paths in KGs for recommendation, it requires to capture not only the semantics of different paths but also their distinctive saliency in describing user preferences towards items.1.
Towards this end, we propose a two-channel neural interaction model by explicitly incorporating path based KG context into the sparse user-item interaction. (1) To construct the context-based path interaction, instead of using traditional Recurrent Neural Networks (RNNs), we propose Residual Recurrent Network (RRN), which is optimised for sequence modelling in KG. That is because RNNs predict the next element of the sequence only based on the current input and the previous hidden state, which is inappropriate for KG path modelling with the triplet structure. To overcome this weakness, the proposed RRN enable the output hidden states of relations to learn a residual [11] connection from their direct head entities when inferring tail entities in the relational sequence. A self-attention network is then applied to the context-based path embedding to capture the polysemy of user interaction behaviours. (2) To explicitly exploit the pairwise interaction between user and item, we propose to use outer product above the embedding layer. Outer product operation generates a two-dimensional interaction map that is more expressive and semantically plausible than simple concatenation or element-wise product. (3) Above these two interaction matrix channel, we propose to adopt a convolutional neural network (CNN) to learn the high-order correlations between users and items. Finally, a joint learning framework is designed to train the overall neural framework in an end-to-end manner, which makes the KG path embedding part and the neural collaborative recommendation (NCF) part to enhance the training process of each other mutually.
The main contributions of this paper are as follows:
- 1.
We are the first to combine KG structure information with the neural collaborative recommendation in an end-to-end neural style.
- 2.
We propose a new recommendation framework KGNCF-RRN, which capture both of 1) the KG’s long-term dependencies context encoded in path embedding and 2) complex matching interaction of user-item with a two-channel convolutional NCF.
- 3.
We conduct empirical studies on three large-scale real-world dataset. Extensive experiment results demonstrate the state-of-the-art performance of KGNCF-RRN and its effectiveness in improving the embedding and interaction quality for the final prediction.
Section snippets
Preliminary
Before introducing our proposed approach, we first introduce the notation and the definition knowledge graph enhanced recommendation task. Then we shortly recapitulate the standard neural collaborative filtering, highlighting its limitations for dealing with this task.
In the study of KG-enhanced recommendation, we have a target recommendation domain with user-item historical interaction data and an auxiliary knowledge graph domain data. For the data in the target recommendation task, let
The proposed approach
The overall framework of KGNCF-RRN is presented in Fig. 3, which have two major parts: (1) A residual recurrent network (RRN) for path embedding (2) a two-channel convolutional NCF that model both of the KG’s long-term dependencies encoded in path embedding and complex matching interaction of user-item. Given user-item interaction and KG data, our aim to capture the fruitful heterogeneous context information hidden in the KG to supplement sparse user-item interaction prediction.
Datasets
We use the following three datasets in our experiment for movie, book and music recommendation, respectively:
- •
MovieLens-1M2 dataset contains almost 1 million ratings (ranging from 1 to 5) on the MovieLens website, which is a widely used benchmark dataset in movie recommendations.
- •
Book-Crossing3 dataset contains 1 million explicit ratings (ranging from 0 to 10) of books in the Book-Crossing community.
- •
KG-based recommendation
We briefly introduce two main categories of knowledge-aware recommendation methods.
Conclusion
In this paper, we develop an end-to-end neural architecture KGNCF-RRN that jointly incorporates the knowledge graph structure and user-item interaction in a unified neural network model for recommendation. Specifically, we propose Residual Recur rent Network (RRN) to construct path embedding, which integrate recur rent neural networks (RNNs) with residual learning to efficiently bridge the gaps between entities and capture the long-term relational dependencies within KGs. Besides, a two channel
CRediT authorship contribution statement
Lei Sang: Conceptualization, Data curation, Software, Writing - original draft. Min Xu: Conceptualization, Methodology, Validation, Writing - review & editing. Shengsheng Qian: Methodology, Formal analysis, Supervision, Writing - review & editing. Xindong Wu: Resources, Project administration, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported in part by the programs for Innovative Research Team in University of the Ministry of Education under Grant IRT_17R32, National Key Research and Development Program of China, under grant 2016YFB1000901, and in part by the National Natural Science Foundation of China under Grant 61673152 and Grant 91746209.
Lei Sang is currently pursuing the Ph.D. degree with the School of Computer Science and Information Engineering, Hefei University of Technology, China, and also with the Faculty of Engineering and Information Technology, University of Technology, Sydney, Ultimo, NSW, Australia. His current research interests include natural language processing and recommender system.
References (54)
- et al.
Matrix factorization for recommendation with explicit and implicit feedback
Knowledge-Based Systems
(2018) - et al.
Resolving data sparsity by multi-type auxiliary implicit feedback for recommender systems
Knowledge-Based Systems
(2017) - et al.
Towards context-aware social recommendation via individual trust
Knowledge-Based Systems
(2017) - et al.
Hybrid neural recommendation with joint deep representation learning of ratings and reviews
Neurocomputing
(2020) - et al.
Semantics-aware recommender systems exploiting linked open data and graph-based features
Knowledge-Based Systems
(2017) A survey of transfer learning for collaborative recommendation with auxiliary data
Neurocomputing
(2016)- et al.
Multi-modal multi-view bayesian semantic embedding for community question answering
Neurocomputing
(2019) - et al.
Serendipitous recommendation in e-commerce using innovator-based collaborative filtering
IEEE Transactions on Cybernetics
(2018) - et al.
Enhancing recommendation on extremely sparse data with blocks-coupled non-negative matrix factorization
Neurocomputing
(2018) - et al.
A neural collaborative filtering model with interaction-based neighborhood
Translating embeddings for modeling multi-relational data
Learning to exploit long-term relational dependencies in knowledge graphs
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Deep residual learning for image recognition
Neural collaborative filtering
Leveraging meta-path based context for top-n recommendation with a neural co-attention model
Your neighbors affect your ratings: on geographical neighborhood influence to rating prediction
Matrix factorization techniques for recommender systems
Computer
An experimental study on implicit social recommendation
Factorization machines with libfm
ACM Transactions on Intelligent Systems and Technology (TIST)
Cited by (0)
Lei Sang is currently pursuing the Ph.D. degree with the School of Computer Science and Information Engineering, Hefei University of Technology, China, and also with the Faculty of Engineering and Information Technology, University of Technology, Sydney, Ultimo, NSW, Australia. His current research interests include natural language processing and recommender system.
Min Xu received the B.E. degree from University of Science and Technology of China, in 2000, M.S degree from National University of Singapore in 2004 and Ph.D. degree from University of Newcastle, Australia in 2010. She is currently a Senior Lecturer at University of Technology, Sydney. Her research interests include multimedia data analytics, pattern recognition and computer vision. She has published over 100 research papers in high quality international journals and conferences.
Shengsheng Qian received the B.E. degree from the Jilin University, Changchun, China, in 2012, and the Ph.D. degree in pattern recognition and intelligent systems from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 2017. He is currently an Assistant Professor with the Institute of Automation, Chinese Academy of Sciences. His current research interests include social media data mining and social event content analysis.
Xindong Wu received the Ph.D. degree in artificial intelligence from The University of Edinburgh, Edinburgh, U.K. He is a professor of computer science at the University of Louisiana at Lafayette, USA. His current research interests include data mining, knowledge-based systems, and Web information exploration. He is the Steering Committee chair of IEEE International Conference on Data Mining (ICDM). He is the editor-in-chief of Knowledge and Information Systems (KAIS) and ACM Transactions on Knowledge Discovery from Data (TKDD). He is a fellow of IEEE and the AAAS