Knowledge graphs (KGs) can provide rich, structured information for recommendation systems as well as increase accuracy and perform explicit reasoning. Deep reinforcement learning (RL) has also sparked great interest in personalized recommendations. The combination of the two holds promise in carrying out interpretable causal inference procedures and improving the performance of graph-structured recommendation. However, most KG-based recommendation focus on rich semantic relationships between entities in a heterogeneous knowledge graph, and thus fail to fully make use of the image information corresponding to an entity. In order to address these issues, we proposed a novel Multi-modal Knowledge-aware Reinforcement Learning Network (MKRLN), which couples recommendation and interpretability by providing actual paths in multi-modal KG (MKG). The MKRLN can generate path representation by composing the structural and visual information of entities, and infers the underlying rational of agent-MKG interactions by leveraging the sequential dependencies within a path from the MKG. In addition, as KGs have too many attributes and entities, their combination with RL leads to too many action spaces and states in the reinforcement learning space, which complicates the search of action spaces. Furthermore, in order to solve this problem, we proposed a new hierarchical attention-path, which makes users focus their attention on the items they are interested in. This reduces the relations and entities in the KGs, which in turn reduces the action space and state in RL, shortens the path to the target entity, and improves the accuracy of recommendation. Our model has explicit explanation ability in knowledge and images. Finally, we extensively evaluated our model on several large-scale real-world benchmark datasets, and it yielded favorable results compared with state-of-the-art methods.
Introduction
With the explosive growth of online content and services, recommendation systems are playing an increasingly important role in matching user needs with various online resources. Knowledge graphs (KGs) is used as an auxiliary resource in recommendation systems, and helps make better use of various types of structured information to improve recommendation performance and enhance the interpretability of the recommended model [1]. In general, the use of KG-awareness recommendation is two-fold: First, some approaches focuses on using knowledge-graph embedding to make personalized recommendation, such as TransE [2], node2vec [3], and Metapath2Vec [4]. These approaches adopt knowledge graph embedding to calculate the similarity between items and users for Top-N item recommendation [5], [6]. However, these methods match the similarity between a user and item, which do not produce the interpretable reasoning process. Second, as knowledge-graph embedding methods do not produce explanations for recommendation. Therefore, some recommendation systems can produce reasonable explanations through KGs. For example, Wang et al. [7] proposed the knowledge-aware path recurrent network (KPRN) to exploit effective reasoning on paths that further infer the underlying rationale of a user-item interaction. Xian et al. [8] further extended explainable recommendation by proposing policy-guided path reasoning, which formally defines and interprets the reasoning process.
As such, there is a great value in exploiting KGs for explainable recommendation systems. However, those methods inject KGs to enrich the representation of recommendation problems, but ignore the visual traits of items and multi-modal information. Further, with the growing volume of online images, content-based image decisions have come into play. For example, when browsing a movie from a website in Fig. 1, users typically look at the movie’s poster first; then they will read the text content and learn that the film is a story between a white driver, Tony, and black musician, Don. Then they will decide about whether to watch the movie. This indicates that images share a great deal of latent knowledge-level connections, which benefits recommendation systems.
Therefore, it is natural to combine the semantic and image modalities. Indeed, many efforts linking image and text have shown promising results and can be applied to recommendation systems, such as visual-semantic embedding and multi-modal correlation learning. Recently, researchers have explored the potential of multi-modal recommendation in greater depth. For example, Yu et al. [9] proposed the vision-language recommendation model, which enables users to provide natural language feedback on visual products. Zhang et al. [10] proposed Joint Representation Learning (JRL) heterogeneous recommendation systems. However, these models use the representation of text and images, which cannot produce the interpretable reasoning process.
In contrast to existing single-modal recommendation methods, such as pure KGs [5], [11], we propose a multi-modal method for explaining recommendation systems. The agent starts with a user and conducts a multi-hop logical path over the MKGs so as to discover suitable items for recommendation to the target user. If the agent recommends items for the user based on a logical path, it will be easy to interpret the reasoning process over the MKG that leads to each recommendation. Thus, the system can provide two aspects of causal evidence, visuals and knowledge, in support of the recommended items. Accordingly, the aim of our system is not only to recommend the candidate items for the user, but also to provide the corresponding explaining logic paths in the MKG. The logic paths contain visuals and knowledge that serve as interpretable evidence for why a given recommendation is made.
Considering the shortcomings of previous work and inspired by the wide application of images, KGs, and deep reinforcement learning (RL), we proposed a multi-modal knowledge graph incorporating the reinforcement learning network (MKRLN) model, that is, a deep RL model incorporating multiple modes for multi-dimension explanation and reasoning. Then, we designed a novel hierarchy attention-path over KGs, which can largely decrease the action spaces and filter noise. The designed recommendation approach has three advantages. First, we can provide explanations from both visual and knowledge aspects, which are complementary. The agent starts from a user (i.e., is linked to an entity) and performs searches so as to discover suitable items along the paths over the KGs. These multi-step paths can provide logical reasons and deep explanations as to how to recommend items for the user. The image can explain the reason from a visual perspective, and knowledge can explain the recommendation from an external knowledge perspective. Second, in a typical KG, one entity can be linked to a large number of neighbors with the same attributes. In this regard, we propose using attention neighbors and attention-paths to greatly reduce the number of large action spaces and the entities. Third, faced with a large number of items, users have difficulty focusing on the items they care most about, so attention-path mechanisms can explore the user’s really preference and filter out redundant information.
Our contributions are summarized as below:
•
We proposed a multi-modal KG combined with deep RL for personalized recommendation. The model can explain logical reasoning from both visual and knowledge aspects, making the explanation multi-dimensional.
•
We designed a novel approach hierarchy attention-path over multi-modal KGs, which can greatly reduce the number of action spaces, entities, and filter out noise, allowing the user to focus on items they care about most.
•
We highlighted the significance that multi-modal KGs for the purpose of evaluating the performance of recommendation systems capable of recommendation with a higher knowledge level and more explicit reasoning about image contents using external information.
Section snippets
Recommendation with knowledge graph
In recent years, researchers have explored the potential of knowledge graph reasoning in recommendation systems. A series of studies focused on the use of knowledge-graph embedding models to make recommendation [4], [5]. Another research direction is to make interpretable recommendation based on the entity and path information in a knowledge graph. For example, Ai et al. [12] proposed a collaborative filtering (CF) method over knowledge-graph embedding for improving personalized recommendation.
Framework
A multi-modal knowledge graph is defined as , where is entity set and is the relation set, represents the KG triples. Each triple represents a facet of the relation , from the head entity, , to the tail entity, . Let be the set of images, with an image being . Each entity is associated with the corresponding image . The images describe the appearances of the entity, which enriches the entities’ representation with their hidden semantics.
Experiments
We extensively evaluated the performance of our model on real world datasets. We first introduced the benchmarks for our experiments and the corresponding experiments settings. Then, we quantitatively compared the effectiveness of our model with other state-of-the-art methods, and conducted ablation studies to show how parameter variations influence our model.
Conclusion
We believe that in the future intelligent agents should have the ability to perform explicit reasoning over knowledge and images for decision-making. In this paper, we proposed an end-to-end framework based on the interaction of deep reinforcement learning and multi-modal knowledge graph to automatically model the recommendation system for recommendation with interpretation. To achieve this, we built a multi-modal knowledge graph and learned the representations of the entities and images within
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work is supported by the National Key R&D Program of China under Grant 2017YFB0802000, the National Natural Science Foundation of China under Grant U1736111, the Plan For Scientific Innovation Talent of Hen’an Province, China under Grant 184100510012, Key Technologies R&D Program of He’nan Province, China under Grant 212102210084, and Innovation Scientists and Technicians Troop Construction Projects of He’nan Province Key Technologies R & D Program, China
of Henan Province under Grant NO.
References (28)
ZhangY.-F. et al.
Explainable recommendation: A survey and new perspectives
(2018)
A. Bordes, N. Usunier, A.G. Duran, J. Weston, et al. Translating Embeddings for Modeling Multi-relational data, in:...
A. Grover, J. Leskovec, node2vec: Scalable feature learning for networks, in: Proceedings of the 22nd ACM SIGKDD...
Y.-X. Dong, N.-V. Chawla, A. Swami, metapath2vec: Scalable representation learning for heterogeneous networks, in:...
F.-Z. Zhang, N.-J. Yuan, D. Lian, X. Xie, et al. Collaborative knowledge base embedding for recommender systems, in:...
E. Palumbo, G. Rizzo, R. Troncy, Entity2rec: Learning user-item relatedness from knowledge graphs for top-N item...
WangX. et al.
Explainable reasoning over knowledge graphs for recommendation
(2018)
Y.-K. Xian, Z.-H. Fu, S. Muthukrishnan, G.-D. Melo, et al. Reinforcement knowledge graph reasoning for explainable...
Y. Tong, Y.-L. Lin, R.-Y. Zhang, X.-Y. Zeng, et al. Vision-language recommendation via attribute augmented multimodal...
Y.-F. Zhang, Q.-Y. Ai, X. Chen, W.-B. Cro, Joint representation learning for top-N recommendation with heterogeneous...
X.-T. Wang, Y.-R. Chen, J. Yang, L. Wu, et al. A reinforcement learning framework for explainable recommendation, in:...
AiQ.-Y. et al.
Learning heterogeneous knowledge base embeddings for explainable recommendation, algorithms
Algorithms
(2018)
WangX. et al.
Explainable reasoning over knowledge graphs for recommendation
(2018)
L.-Z. Liao, Y.-S. Ma, X.-N. He, R.-C. Hong, et al. Knowledge-aware multimodal dialogue systems, in: Proceedings of the...
The software knowledge community provides software developers with valuable knowledge of technologies, activities, tools and project management related to software development. However, a large number of unresolved questions and the lack of expert participation have become evident and critical challenges for the software knowledge community. To address the problems of label dependence, interactive data’s sparsity and unassociated knowledge in community-based software expert recommendation, we propose an Expertise Preference-Aware Network model for Software Expert Recommendation (EPAN-SERec) with knowledge graph. Firstly, the software knowledge graph is utilized as an auxiliary resource to provide domain knowledge representation. Secondly, we devise an expertise preference-learning framework by means of deep reinforcement learning that models the historical interactive information of experts and generate the expertise preference weight graph. To better learn expertise preference features, a graph convolutional network (GCN) model with integrated graph self-supervised learning is proposed to optimize the features representation. Finally, software knowledge entity embeddings with semantic information are obtained by exploiting the graph-embedding model, and the final features of question to be answered are obtained by fusing the expertise preference of experts. Extensive experiments on the dataset based on StackOverflow demonstrate that our approach achieves a better outcome than baseline models.
Existing approaches to learning path recommendation for online learning communities mainly rely on the individual characteristics of users or the historical records of their learning processes, but pay less attention to the semantics of users’ postings and the context. To facilitate the knowledge understanding and personalized learning of users in online learning communities, it is necessary to conduct a fine-grained analysis of user data to capture their dynamical learning characteristics and potential knowledge levels, so as to recommend appropriate learning paths. In this paper, we propose a fine-grained and multi-context-aware learning path recommendation model for online learning communities based on a knowledge graph. First, we design a multidimensional knowledge graph to solve the problem of monotonous and incomplete entity information presentation of the single layer knowledge graph. Second, we use the topic preference features of users’ postings to determine the starting point of learning paths. We then strengthen the distant relationship of knowledge in the global context using the multidimensional knowledge graph when generating and recommending learning paths. Finally, we build a user background similarity matrix to establish user connections in the local context to recommend users with similar knowledge levels and learning preferences and synchronize their subsequent postings. Experiment results show that the proposed model can recommend appropriate learning paths for users, and the recommended similar users and postings are effective.
Understanding tourists’ decision-making processes, in which many factors ranging from functional attributes to geographical configurations are highly intertwined, has long been a crux for tourism management. Existing studies are typically based on manual surveys that extract the intricate psychological or behavioural mechanisms, but the huge expense of the required samplings limits the generalization and comprehensiveness of the findings. This study proposes a novel explainable recommendation method—Knowledge-Graph-aware Disentangled Auto-Encoder (KGDAE)—to automatically unravel the tourists’ decision processes from massive historical behaviour data. Based on the constructed tourism-KG that integrates multidimensional factors into 23 types of entities corresponding to 37 semantic and geographic relationships, KGDAE realizes a macro-micro supervised disentangled learning for the interaction of multiple determinants. Macroscopically, the hierarchical attention mechanisms are designed to distinguish the dominance of either functional or geographical factors, and capture the effect of the residential environment; microscopically, the preference-propagation-based technique is introduced to infer the fine-grained characteristics and relations of tourist interests on the tourism-KG. Extensive experiments show that KGDAE can effectively restore tourists’ decision processes according to two empirical studies while boosting the recommendation performance compared to multiple state-of-the-art methods with an increase of 1∼19%. Furthermore, the advantaged interpretability also guarantees the robustness of sparse recommendation scenario to achieve the lowest degradation at 7.8%.
Recommender systems (RSs) have been employed for many real-world applications including search engines, social networks, and information retrieval systems as powerful intelligent techniques for creating personalized recommendations. RSs mainly utilize different data resources such as user-item ratings, tag data, and social relationships in their recommendation process to overcome the data sparsity issue. One of the main challenges is how to integrate the results obtained by these data resources. Moreover, most of the existing RSs focus on the recommendation accuracy, and do not consider the reliability of the recommendations. To address these challenges, in this study, a reliable recommendation method is developed, which employs deep neural networks and reinforcement learning. The proposed method firstly uses a stacked denoising autoencoder model to obtain the latent features from the rating, tag, and trust data. The similarity value between each pair of users is then calculated in three different aspects according to the extracted latent features. An ensemble approach is developed by integrating the three similarities using reinforcement learning to determine the final similarity value. The ensemble similarity values are used in the rating prediction process, and the reliability of the predicted ratings is evaluated using an effective measure. Finally, a recommendation strategy is developed based on the integration of the predicted ratings and their reliability values. We evaluated the effectiveness of the proposed method by conducting numerous experiments on three datasets where the results show that the proposed method is far superior to other recommendation models in making more accurate recommendations.
In recent years, RL has been widely introduced into interaction recommendation systems, such as ad recommendation [26], movie recommendation [27], music recommendation [28], etc., as its advantages of considering users’ long-term feedbacks. For example, Tao et al. [29] proposed an end-to-end framework based on the multi-modal knowledge-aware reinforcement learning network for explainable recommendation. Huang et al. [30] modeled the recommendation process with MDP and then introduced a top-N deep reinforcement learning-based approach to handle the long-term recommendation problem.
Effective reuse of historical process knowledge has been one of the most important technical roadmaps driving traditional human experience-based process planning all the way to intelligent process planning in the context of Industry 4.0. However, effective knowledge reuse in process planning is still made difficult nowadays due to the lack of means to capture the technologists’ preference embedded in the interaction history of technologists with the system, and the mutual influence of process decision-making activities for future knowledge recommendations. To bridge the gap, this work takes in-depth integration of deep learning and reinforcement learning in process planning and proposes a novel dynamic interaction recommendation approach for knowledge aided intelligent process planning (KAiPP) towards Industry 4.0. This work first formalizes KAiPP as a sequential interaction knowledge recommendation scenario, to which a Markov decision process is employed to model the KAiPP scenario. On that basis, a knowledge-aware reinforcement learning model is designed to learn a KAiPP agent through interacting with the KAiPP environment for effective interaction knowledge recommendation in process planning. Here, the technologists’ preference-task information model and dynamic knowledge sub-graph are elaborated as the KAiPP environment to capture the preference of the technologist and mutual influence of process decision-making activities. Finally, experimental results show that the proposed approach achieves a very high recommendation performance that outperforms the state-of-the-art approaches in the process planning domain. In addition, a thus developed KAiPP prototype system provides a reference for industrial implementation of the approach.
To tackle this problem, RL algorithms are preferred solutions. Differing from the recommendation models based on traditional machine learning methods [53–55] and deep learning methods [56–59], the explainable recommendation models with KG and RL [60] not only make high-quality recommendations but also provide explanations, which contribute to the effectiveness and trustworthiness of the recommender systems. For example, Park et al. [61]
To improve the recommendation accuracy and offer explanations for recommendations, Reinforcement Learning (RL) has been applied to path reasoning over knowledge graphs. However, in recommendation tasks, most existing RL methods learn the path-finding policy using only a short-term or single reward, leading to a local optimum and losing some potential paths. To address these issues, we propose a Self-Supervised Reinforcement Learning (SSRL) framework combined with dual-reward for knowledge-aware recommendation reasoning over knowledge graphs. Then, we improve Actor–Critic algorithm by using a dual-reward driven strategy, which combines short-term reward with long-term incremental evaluation. The improved algorithm helps the policy guide path reasoning in an overall situation. In addition, to find the most potential paths, in the improved Actor–Critic algorithm, a loss constraint of each sample is used as a reinforced signal to update the gradients. With some improvements against baselines, experimental results demonstrate the effectiveness of our framework.
