research-article

Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding

Authors:

Chun ChenAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 1631 - 1639

https://doi.org/10.1145/3485447.3512209

Published: 25 April 2022 Publication History

Abstract

Learning low-dimensional representations for Heterogeneous Information Networks (HINs) has drawn increasing attention recently for its effectiveness in real-world applications. Compared with homogeneous networks, HINs are characterized by meta-paths connecting different types of nodes with semantic meanings. Existing methods mainly follow the prototype of independently learning meta-path-based embeddings and integrating them into a unified embedding. However, meta-paths in a HIN are inherently correlated since they reflect different perspectives of the same object. If each meta-path is treated as an isolated semantic data resource and the correlations among them are disregarded, sub-optimality in the both the meta-path based embedding and final embedding will be resulted. To address this issue, we make the first attempt to explicitly model the correlation among meta-paths by proposing Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding (CKD). More specifically, we model the knowledge in each meta-path with two different granularities: regional knowledge and global knowledge. We learn the meta-path-based embeddings by collaboratively distill the knowledge from intra-meta-path and inter-meta-path simultaneously. Experiments conducted on six real-world HIN datasets demonstrates the effectiveness of the CKD method.

References

[1]

Rohan Anil, Gabriel Pereyra, Alexandre Passos, Róbert Ormándi, George E. Dahl, and Geoffrey E. Hinton. 2018. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations.

[2]

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. 2018. Mine: mutual information neural estimation. ArXiv preprint (2018).

[3]

Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019. Representation learning for attributed multiplex heterogeneous network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1358–1368.

Digital Library

[4]

Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. 2020. Online Knowledge Distillation with Diverse Peers. In Proceedings of the AAAI Conference on Artificial Intelligence. 3430–3437.

[5]

Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. 2021. Cross-Layer Distillation with Semantic Calibration. In Proceedings of the AAAI Conference on Artificial Intelligence. 7028–7036.

[6]

Hongxu Chen, Hongzhi Yin, Weiqing Wang, Hao Wang, Quoc Viet Hung Nguyen, and Xue Li. 2018. PME: projected metric embedding on heterogeneous networks for link prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1177–1186.

Digital Library

[7]

Xia Chen, Guoxian Yu, Jun Wang, Carlotta Domeniconi, Zhao Li, and Xiangliang Zhang. 2019. Activehne: Active heterogeneous network embedding. arXiv preprint arXiv:1905.05659(2019).

[8]

Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017. metapath2vec: Scalable representation learning for heterogeneous networks. In SIGKDD. 135–144.

[9]

Yuxiao Dong, Ziniu Hu, Kuansan Wang, Yizhou Sun, and Jie Tang. 2020. Heterogeneous Network Representation Learning. In IJCAI, Vol. 20. 4861–4867.

[10]

Dániel Fogaras, Balázs Rácz, Károly Csalogány, and Tamás Sarlós. 2005. Towards scaling fully personalized pagerank: Algorithms, lower bounds, and experiments. Internet Mathematics(2005), 333–358.

[11]

Tao-yang Fu, Wang-Chien Lee, and Zhen Lei. 2017. Hin2vec: Explore meta-paths in heterogeneous information networks for representation learning. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1797–1806.

[12]

Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020. MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding. In The Web Conference. 2331–2341.

[13]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In NeuraIPS. 1024–1034.

[14]

Yu He, Yangqiu Song, Jianxin Li, Cheng Ji, Jian Peng, and Hao Peng. 2019. Hetespaceywalk: A heterogeneous spacey random walk for heterogeneous information network embedding. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 639–648.

Digital Library

[15]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531(2015).

[16]

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. ArXiv preprint (2018).

[17]

Huiting Hong, Hantao Guo, Yucheng Lin, Xiaoqing Yang, Zang Li, and Jieping Ye. 2020. An attention-based graph neural network for heterogeneous structural learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 4132–4139.

[18]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous graph transformer. In The Web Conference. 2704–2710.

Digital Library

[19]

Xiao Huang, Qingquan Song, Fan Yang, and Xia Hu. 2019. Large-scale heterogeneous feature embedding. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 3878–3885.

Digital Library

[20]

Rana Hussein, Dingqi Yang, and Philippe Cudré-Mauroux. 2018. Are meta-paths necessary? Revisiting heterogeneous graph embeddings. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 437–446.

[21]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. ArXiv preprint (2016).

[22]

Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. ArXiv preprint (2016).

[23]

Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. 2019. Diffusion improves graph learning. In NeuraIPS. 13354–13366.

[24]

Yuanfu Lu, Chuan Shi, Linmei Hu, and Zhiyuan Liu. 2019. Relation structure-aware heterogeneous information network embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 4456–4463.

Digital Library

[25]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781(2013).

[26]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. NeuraIPS (2013), 3111–3119.

[27]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748(2018).

[28]

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. 2020. Graph Representation Learning via Graphical Mutual Information Maximization. In The Web Conference. 259–270.

Digital Library

[29]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 701–710.

Digital Library

[30]

Yuxiang Ren, Bo Liu, Chao Huang, Peng Dai, Liefeng Bo, and Jiawei Zhang. 2019. Heterogeneous deep graph infomax. ArXiv preprint (2019).

[31]

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In Proceedings of the International Conference on Learning Representations.

[32]

Chuan Shi, Yuanfu Lu, Linmei Hu, Zhiyuan Liu, and Huadong Ma. 2020. RHINE: Relation structure-aware heterogeneous information network embedding. IEEE Transactions on Knowledge and Data Engineering (2020).

[33]

Jiaming Song and Stefano Ermon. 2019. Understanding the limitations of variational mutual information estimators. ArXiv preprint (2019).

[34]

Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S Yu, and Tianyi Wu. 2011. Pathsim: Meta path-based top-k similarity search in heterogeneous information networks. VLDB (2011), 992–1003.

Digital Library

[35]

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015. Line: Large-scale information network embedding. In The Web Conference. 1067–1077.

Digital Library

[36]

Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive representation distillation. In Proceedings of the International Conference on Learning Representations.

[37]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. ArXiv preprint (2017).

[38]

Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019. Deep graph infomax. (2019).

[39]

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019. Heterogeneous graph attention network. In The Conference. 2022–2032.

Digital Library

[40]

Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. 2021. Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning. arXiv preprint arXiv:2105.09111(2021).

[41]

Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. 2020. Heterogeneous Network Representation Learning: Survey, Benchmark, Evaluation, and Beyond. TKDE (2020).

[42]

Sergey Zagoruyko and Nikos Komodakis. 2017. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In Proceedings of the International Conference on Learning Representations.

[43]

Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. 2019. Heterogeneous graph neural network. In SIGKDD. 793–803.

[44]

Wentao Zhang, Yuan Fang, Zemin Liu, Min Wu, and Xinming Zhang. 2020. mg2vec: Learning Relationship-Preserving Heterogeneous Graph Representations via Metagraph Embedding. TKDE (2020).

[45]

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4320–4328.

[46]

Jianan Zhao, Xiao Wang, Chuan Shi, Zekuan Liu, and Yanfang Ye. 2020. Network Schema Preserving Heterogeneous Information Network Embedding.

[47]

Sheng Zhou, Jiajun Bu, Xin Wang, Jiawei Chen, and Can Wang. 2019. Hahe: Hierarchical attentive heterogeneous information network embedding. arXiv preprint arXiv:1902.01475(2019).

[48]

Sheng Zhou, Xin Wang, Jiajun Bu, Martin Ester, Pinggang Yu, Jiawei Chen, Qihao Shi, and Can Wang. 2020. DGE: Deep Generative Network Embedding Based on Commonality and Individuality. In AAAI.

[49]

Sheng Zhou, Yucheng Wang, Defang Chen, Jiawei Chen, Xin Wang, Can Wang, and Jiajun Bu. 2021. Distilling Holistic Knowledge with Graph Neural Networks. In International Conference on Computer Vision.

[50]

Sheng Zhou, Hongxia Yang, Xin Wang, Jiajun Bu, Martin Ester, Pinggang Yu, Jianwei Zhang, and Can Wang. 2018. Prre: Personalized relation ranking embedding for attributed networks. In CIKM. 823–832.

Digital Library

Cited By

Xi XYuan JLu SHe J(2025)Synergistic Multi-Drug Combination Prediction Based on Heterogeneous Network Representation Learning with Contrastive LearningTsinghua Science and Technology10.26599/TST.2023.901014930:1(215-233)Online publication date: Feb-2025
https://doi.org/10.26599/TST.2023.9010149
Tian YPei SZhang XZhang CChawla N(2025)Knowledge Distillation on Graphs: A SurveyACM Computing Surveys10.1145/3711121Online publication date: 30-Jan-2025
https://doi.org/10.1145/3711121
Meng XXie CLi HZeng GChen K(2025)Research on Optimization of Large-Scale Heterogeneous Combat Network Based on Graph EmbeddingIEEE Access10.1109/ACCESS.2025.352665013(5773-5784)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3526650
Show More Cited By

Index Terms

Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information systems applications

Index terms have been assigned to the content through auto-classification.

Recommendations

Heterogeneous Information Network Embedding with Meta-path Based Graph Attention Networks
Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions
Abstract
Network embedding is an emerging research field which aims at projecting network elements into lower dimensional spaces. However, most network embedding algorithms focus on homogeneous networks, thus cannot be directly applied to the Heterogeneous ...
Easing Embedding Learning by Comprehensive Transcription of Heterogeneous Information Networks
KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Heterogeneous information networks (HINs) are ubiquitous in real-world applications. In the meantime, network embedding has emerged as a convenient tool to mine and learn from networked data. As a result, it is of interest to develop HIN embedding ...
Task-Guided and Path-Augmented Heterogeneous Network Embedding for Author Identification
WSDM '17: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining

In this paper, we study the problem of author identification under double-blind review setting, which is to identify potential authors given information of an anonymized paper. Different from existing approaches that rely heavily on feature engineering, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Natural Science Foundation of China

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
610
Total Downloads

Downloads (Last 12 months)95
Downloads (Last 6 weeks)11

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xi XYuan JLu SHe J(2025)Synergistic Multi-Drug Combination Prediction Based on Heterogeneous Network Representation Learning with Contrastive LearningTsinghua Science and Technology10.26599/TST.2023.901014930:1(215-233)Online publication date: Feb-2025
https://doi.org/10.26599/TST.2023.9010149
Tian YPei SZhang XZhang CChawla N(2025)Knowledge Distillation on Graphs: A SurveyACM Computing Surveys10.1145/3711121Online publication date: 30-Jan-2025
https://doi.org/10.1145/3711121
Meng XXie CLi HZeng GChen K(2025)Research on Optimization of Large-Scale Heterogeneous Combat Network Based on Graph EmbeddingIEEE Access10.1109/ACCESS.2025.352665013(5773-5784)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3526650
Liu FLi CWu B(2025)Community-oriented multi-scale heterogeneous community detection using weighted positives and debiased negativesKnowledge-Based Systems10.1016/j.knosys.2024.112934310(112934)Online publication date: Feb-2025
https://doi.org/10.1016/j.knosys.2024.112934
Mo YShen HZhu X(2025)Efficient self-supervised heterogeneous graph representation learning with reconstructionInformation Fusion10.1016/j.inffus.2024.102846117(102846)Online publication date: May-2025
https://doi.org/10.1016/j.inffus.2024.102846
Tang JWei WXia LHuang CSerra ESpezzano F(2024)EasyST: A Simple Framework for Spatio-Temporal PredictionProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679749(2220-2229)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679749
Chen LGuo HLei YLi YLiu Z(2024)Coarse-to-Fine Robust Heterogeneous Network Representation Learning Without MetapathIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.344572411:6(5773-5789)Online publication date: Nov-2024
https://doi.org/10.1109/TNSE.2024.3445724
Zhang HYang XBai LLiang J(2024)Enhancing Drug Recommendations Via Heterogeneous Graph Representation Learning in EHR NetworksIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.332902536:7(3024-3035)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2023.3329025
Zhao HZeng ZWang YYe DMiao C(2024)HGAttack: Transferable Heterogeneous Graph Adversarial Attack2024 IEEE International Conference on Agents (ICA)10.1109/ICA63002.2024.00028(100-105)Online publication date: 4-Dec-2024
https://doi.org/10.1109/ICA63002.2024.00028
Lv SLi XLi QZhao C(2024)Semantic Structure Encoding Enhancement for Heterogeneous Graph Neural Networks2024 17th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI)10.1109/CISP-BMEI64163.2024.10906266(1-6)Online publication date: 26-Oct-2024
https://doi.org/10.1109/CISP-BMEI64163.2024.10906266
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten