Skip to main content
SpringerLink
Log in

Fusing heterogeneous information for multi-modal attributed network embedding

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In the real world, networks with many types of nodes and edges are complex, forming a heterogeneous network. For instance, film networks contain different node types, such as directors, films and actors, as well as different types of edge and multimodal attributes. Most existing attribution network embedding algorithms cannot flexibly capture the impact of multimodal attributes on the topology. Premature fusion of multimodal features encodes different attribute information into the representation embedding, while the later fusion strategy ignores the interaction between different modes, both of which affect the modeling of graph embedding.To solve this problem, we propose a multimodal attribute network representation learning algorithm based on heterogeneity information fusion, named FHIANE. It extracts features from multimodal information sources through deep heterogeneous convolutional networks and projects them into a consistent semantic space while maintaining structural information. In addition, we design a modality fusion network based on an extended attention mechanism that takes full advantage of the consistency and complementarity of multimodal information. We evaluate the performance of the FHIANE algorithm on several real datasets through challenging tasks such as link prediction and node classification. The experimental results show that FHIANE outperforms other baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data Availability

The datasets that support the findings of this study are available in https://github.com/yangjieyi123/FHIANE

Notes

  1. https://grouplens.org/datasets/movielens/

  2. https://nijianmo.github.io/amazon/index.html

  3. https://www.kaggle.com/CooperUnion/anime-recommendations-database

References

  1. Wei Y, Wang X, Nie L, He X, Hong R, Chua T-S (2019) MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video. In: Proceedings of the 27th ACM International Conference on Multimedia, Nice, pp. 1437–1445

  2. Liang J, Jacobs P, Sun J, Parthasarathy S (2018) Semi-supervised embedding in attributed networks with outliers. In: Proceedings of the 2018 SIAM International Conference on Data Mining, San Diego, pp. 153 – 161 . https://doi.org/10.1137/1.9781611975321.18

  3. Bandyopadhyay S, Vivek SV, Murty MN (2020) Outlier resistant unsupervised deep architectures for attributed network embedding. In:Proceedings of the international conference on web search and data mining, Houston, pp 25 – 33

  4. Zhao Z, Li C, Zhang X, Chiclana F, Viedma EH (2019) An incremental method to detect communities in dynamic evolving social networks. Knowledge-based systems 163:404 – 415

  5. Liu J, Shi C, Yang C, Lu Z, Philip SY (2022) A survey on heterogeneous information network based recommender systems: Concepts, methods, applications and resources. AI Open 3:40 – 57

  6. Gao Y, Gong M, Xie Y, Zhong H (2020) Community-oriented attributed network embedding. Knowl Based Syst 193:105418. https://doi.org/10.1016/j.knosys.2019.105418

    Article  Google Scholar 

  7. Chunaev P (2020) Community detection in node-attributed social networks: a survey. Comput Sci Rev 37:100286

    Article  MathSciNet  MATH  Google Scholar 

  8. Khan W,(2021) An exhaustive review on state-of-the-art techniques for anomaly detection on attributed networks. Turkish J Comput Math Educ (TURCOMAT)12(10):6707–6722

  9. Afyouni I, Al Aghbari Z, Razack RA (2022) Multi-feature, multi-modal, and multi-source social event detection: A comprehensive survey information fusion 79:279–308

  10. Zhang C, Song D, Huang C, Swami A, Chawla NV(2019) Heterogeneous graph neural network. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, pp 793 – 803

  11. Jangra A, Mukherjee S, Jatowt A, Saha S, Hasanuzzaman M (2021) A survey on multi-modal summarization. ACM Computing Surveys

  12. Wang Y (2021) Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 17(1s), 1 – 25

  13. Perozzi B, Al-Rfou R, Skiena S (2014) Deepwalk Online learning of social representations. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, pp 701 – 710

  14. Kipf TN, Welling M (2017) Semi-Supervised Classification with Graph Convolutional Networks. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. https://openreview.net/forum?id=SJU4ayYgl

  15. Velickovic P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph Attention Networks. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada .https://openreview.net/forum?id=rJXMpikCZ

  16. Ying C, Cai T, Luo S, Zheng S, Ke G, He D, Shen Y, Liu T-Y (2021) Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems 34, 28877 – 28888

  17. [17] Waswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, Kaiser L, Polosukhin I (2017) Attention is all you need. In: NIPS

  18. Dong Y, Chawla NV, Swami A (2017) metapath2vec: Scalable Representation Learning for Heterogeneous Networks In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 135 – 144 https://doi.org/10.1145/3097983.3098036

  19. Wang X, Ji H, Shi C, Wang B, Ye Y, Cui P, Yu PS (2019) Heterogeneous Graph Attention Network In: The World Wide Web Conference, WWW 2019, pp 2022 – 2032 https://doi.org/10.1145/3308558.3313562

  20. Liu J, Song L, Wang G, Shang X (2023) Meta-hgt: Metapath-aware hypergraph transformer for heterogeneous information network embedding Neural Networks 157, pp 65 – 76

  21. Wang R, Shi C, Zhao T, Wang X, Ye YF (2021) Heterogeneous information network embedding with adversarial disentangler. IEEE Transactions on Knowledge and Data Engineering

  22. Huang X, Song Q, Li Y, Hu X (2019) Graph Recurrent Networks With Attributed RandomWalks. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, pp 732 – 740 https://doi.org/10.1145/3292500.3330941

  23. Zhou X, Su L, Li X, Zhao Z, Li C (2023) Community detection based on unsupervised attributed network embedding. Expert Syst Appl 213:118937

    Article  Google Scholar 

  24. Zhang C, Song D, Huang C, Swami A, Chawla NV (2019) Heterogeneous graph neural network In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining pp 793 – 803

  25. Li P-H, Fu T-J, Ma W-Y (2020) Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, pp. 8236 – 8244 https://aaai.org/ojs/index.php/AAAI/article/view/6338

  26. Chen J, Zhang A (2020) HGMF: Heterogeneous Graph-based Fusion for Multimodal Data with Incompleteness In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Virtual Event, pp 1295 – 1305

  27. Wang J, Hu J, Qian S, Fang Q, Xu C (2020) Multimodal graph convolutional networks for high quality content recognition. Neurocomputing 412:42 – 51 https://doi.org/10.1016/j.neucom.2020.04.145

  28. Wang Y, Qian S, Hu J, Fang Q, Xu C (2020) Fake News Detection via Knowledge-driven Multimodal Graph Convolutional Networks In: Proceedings of the 2020 on International Conference on Multimedia Retrieval, pp 540 – 547 https://doi.org/10.1145/3372278.3390713

  29. Tao Z, Wei Y, Wang X, He X, Huang X, Chua T-S (2020) MGAT: Multimodal Graph Attention Network for Recommendation Inf Process Manag 57(5):102277 https://doi.org/10.1016/j.ipm.2020.102277

  30. Zhang J, Zhu Y, Liu Q, Wu S, Wang S, Wang L (2021) Mining Latent Structures for Multimedia Recommendation In: MM ’21: ACM Multimedia Conference, pp 3872 – 3880 https://doi.org/10.1145/3474085.3475259

  31. Jia X, Jiang M, Dong Y, Zhu F, Lin H, Xin Y, Chen H (2022) Multimodal heterogeneous graph attention network, Neural Computing and Applications, 1 – 16

  32. Cai H, Zheng VW, Chang K C-C (2018) A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Transactions on Knowledge and Data Engineering 30(9): 1616 – 1637

  33. Gao H, Huang H (2018) Deep Attributed Network Embedding, In: Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, pp 3364 – 3370 https://doi.org/10.24963/ijcai.2018/467

  34. Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q (2015) Line: Large-scale information network embedding In: Proceedings of the 24th International Conference on World Wide Web, Florence, pp 1067–1077

  35. Zhang Z, Yang H, Bu J, Zhou S, Yu P, Zhang J, Ester M, Wang C, (2018) ANRL: Attributed Network Representation Learning via Deep Neural Networks. In: Proceedings of the International Joint Conference on Artificial Intelligence, 18 Stockholm, pp 3155 – 3161

  36. Yang C, Liu Z, Zhao D, Sun M, Chang EY, (2015) Network representation learning with rich text information In: Proceedings of the International Joint Conference on Artificial Intelligence, vol. 2015. Melbourne, pp 2111 – 2117

  37. Hamilton WL, Ying Z, Leskovec J, (2017) Inductive Representation Learning on Large Graphs. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, pp 1024 – 1034 http://papers.nips.cc/paper/6703-inductive-representation-learning-on-large-graphs

  38. Velickovic P, Fedus W, Hamilton WL, Li‘o P, Bengio Y, Hjelm RD (2019) Deep Graph Infomax. In: ICLR (Poster)

  39. Hong H, Guo H, Lin Y, Yang X, Li Z, Ye J, (2020) An attentionbased graph neural network for heterogeneous structural learning In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, pp 4132 – 4139

  40. Lv Q, Ding M, Liu Q, Chen Y, Feng W, He S, Zhou C, Jiang J, Dong Y, Tang J, (2021) Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp 1150 – 1160

  41. Davis J, Goadrich M, (2006) The relationship between Precision-Recall and ROC curves In: Proceedings of the 23rd International Conference on Machine Learning, pp 233 – 240

  42. Wang H, Zhang F, Hou M, Xie X, Guo M, Liu Q, (2018) Shine: Signed heterogeneous information network embedding for sentiment link prediction In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp 592 – 600

  43. Chen H, Yin H, Wang W, Wang H, Nguyen Q.V.H, Li X, (2018) PME: projected metric embedding on heterogeneous networks for link prediction In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 1177 – 1186

  44. Li M.W, Xu D.Y, Geng J, Hong W.C, (2022) A hybrid approach for forecasting ship motion using cnn-gru-am and gcwoa, Applied Soft Computing (114-), 114

  45. Van der Maaten L, Hinton G, (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11)

  46. Fu X, Zhang J, Meng Z, & King I, Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference (2020) pp 2331–2341

  47. Bandyopadhyay, S, Biswas, A, Kara, H, Murty, MN (2020) A Multilayered Informative Random Walk for Attributed Social Network Embedding. In: 24th European Conference on Artificial Intelligence, pp 1738–1745. https://doi.org/10.3233/FAIA200287

  48. Kakisim, AG (2022). Enhancing attributed network embedding via enriched attribute representations. Applied Intelligence, 52(2), 1566–1580

Download references

Acknowledgements

This study was funded by National Natural Science Foundation of China (No.62271274), Natural Science Foundation of Zhejiang Province (No.LY20F020009,LZ20F020001) and Natural Science Foundation of Ningbo (No.202003N4086). (Corresponding author: Yihong Dong.)

Author information

Authors and Affiliations

  1. Faculty of electrical engineering and computer science, Ningbo University, 315211, Ningbo, China

    Yang Jieyi, Zhu Feng, Dong Yihong & Qian Jiangbo

Authors
  1. Yang Jieyi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhu Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dong Yihong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qian Jiangbo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dong Yihong.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jieyi, Y., Feng, Z., Yihong, D. et al. Fusing heterogeneous information for multi-modal attributed network embedding. Appl Intell 53, 22328–22347 (2023). https://doi.org/10.1007/s10489-023-04675-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04675-5

Keywords

Search

Navigation