Yifei Zhang
ABSTRACT
Graph representation learning via Contrastive Learning (GCL) has drawn considerable attention recently. Efforts are mainly focused on gathering more global information via contrasting on a single high-level graph view, which, however, underestimates the inherent complex and hierarchical properties in many real-world networks, leading to sub-optimal embeddings. To incorporate these properties of a complex graph, we propose Cross-Scale Contrastive Graph Knowledge Synergy (CGKS), a generic feature learning framework, to advance graph contrastive learning with enhanced generalization ability and the awareness of latent anatomies. Specifically, to maintain the hierarchical information, we create a so-call graph pyramid (GP) consisting of coarse-grained graph views. Each graph view is obtained via the careful design topology-aware graph coarsening layer that extends the Laplacian Eigenmaps with negative sampling. To promote cross-scale information sharing and knowledge interactions among GP, we propose a novel joint optimization formula that contains a pairwise contrastive loss between any two coarse-grained graph views. This synergy loss not only promotes knowledge sharing that yields informative representations, but also stabilizes the training process. Experiments on various downstream tasks demonstrate the substantial improvements of the proposed method over its counterparts.
Supplemental Material
- Amir Hosein Khas Ahmadi, Kaveh Hassani, Parsa Moradi, Leo Lee, and Quaid Morris. 2020. Memory-Based Graph Networks. In Proc. of ICLR.Google Scholar
- Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. 2020. Spectral Clustering with Graph Neural Networks for Graph Pooling. In Proc. of ICML.Google Scholar
- Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine.Google Scholar
- Yankai Chen, Yixiang Fang, Yifei Zhang, and Irwin King. 2023 a. Bipartite Graph Convolutional Hashing for Effective and Efficient Top-N Search in Hamming Space. In Proceedings of the ACM Web Conference 2023 (WebConf).Google ScholarDigital Library
- Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, and Irwin King. 2022a. Learning binarized graph representations with multi-faceted quantization reinforcement for top-k recommendation. In Proc. of KDD.Google ScholarDigital Library
- Yankai Chen, Menglin Yang, Yingxue Zhang, Mengchen Zhao, Ziqiao Meng, Jianye Hao, and Irwin King. 2022c. Modeling scale-free graphs with hyperbolic geometry for knowledge-aware recommendation. In Proc. of WSDM.Google ScholarDigital Library
- Yankai Chen, Yaming Yang, Yujing Wang, Jing Bai, Xiangchen Song, and Irwin King. 2022b. Attentive knowledge-aware graph convolutional networks with collaborative guidance for personalized recommendation. In 2022 IEEE 38th International Conference on Data Engineering (ICDE).Google ScholarCross Ref
- Yankai Chen, Yifei Zhang, Huifeng Guo, Ruiming Tang, and Irwin King. 2022d. An Effective Post-training Embedding Binarization Approach for Fast Online Top-K Passage Matching. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. 102--108.Google Scholar
- Yankai Chen, Yifei Zhang, Menglin Yang, Zixing Song, Chen Ma, and Irwin King. 2023 b. WSFE: Wasserstein Sub-graph Feature Encoder for Effective User Segmentation in Collaborative Filtering. arXiv preprint arXiv:2305.04410 (2023).Google Scholar
- Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable Feature Learning for Networks. In Proc. of KDD.Google ScholarDigital Library
- Michael Gutmann and Aapo Hyv"arinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS.Google Scholar
- Hakim Hafidi, Mounir Ghogho, Philippe Ciblat, and Ananthram Swami. 2020. Graphcl: Contrastive self-supervised learning of graph representations. ArXiv preprint.Google Scholar
- William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In Proc. of NeurIPS.Google Scholar
- Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. 2022. Vision GNN: An Image is Worth Graph of Nodes. ArXiv preprint.Google Scholar
- Kaveh Hassani and Amir Hosein Khas Ahmadi. 2020. Contrastive Multi-View Representation Learning on Graphs. In Proc. of ICML.Google Scholar
- Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. ArXiv preprint.Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In Proc. of ICLR.Google Scholar
- Bolian Li, Baoyu Jing, and Hanghang Tong. 2022. Graph Communal Contrastive Learning. In Proceedings of the ACM Web Conference 2022.Google ScholarDigital Library
- Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning. In Proc. of AAAI.Google ScholarCross Ref
- Shuai Lin, Chen Liu, Pan Zhou, Zi-Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. 2022. Prototypical graph contrastive learning. IEEE Transactions on Neural Networks and Learning Systems.Google ScholarCross Ref
- Shuai Lin, Pan Zhou, Zi-Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. 2021. Prototypical graph contrastive learning. ArXiv preprint.Google Scholar
- Jan R Magnus. 1998. HANDBOOK OF MATRICES: H. Lütkepohl, John Wiley and Sons, 1996. Econometric Theory.Google ScholarCross Ref
- Costas Mavromatis and George Karypis. 2021. Graph InfoClust: Maximizing Coarse-Grain Mutual Information in Graphs. In Proc. of KDD.Google ScholarDigital Library
- Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-Based Recommendations on Styles and Substitutes. In Proc. of SIGIR.Google Scholar
- Péter Mernyei and Cua tua lina Cangea. 2020. Wiki-cs: A wikipedia-based benchmark for graph neural networks. ArXiv preprint.Google Scholar
- Diego P. P. Mesquita, Amauri H. Souza Jr., and Samuel Kaski. 2020. Rethinking pooling in graph neural networks. In Proc. of NeurIPS.Google Scholar
- Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. 2017. graph2vec: Learning Distributed Representations of Graphs. ArXiv preprint.Google Scholar
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kö pf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. of NeurIPS.Google Scholar
- Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learning of social representations. In Proc. of KDD.Google ScholarDigital Library
- Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020. GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. In Proc. of KDD.Google ScholarDigital Library
- Marian-Andrei Rizoiu, Timothy Graham, Rui Zhang, Yifei Zhang, Robert Ackland, and Lexing Xie. 2018. # debatenight: The role and influence of socialbots on twitter during the 1st 2016 us presidential debate. In Proc. of AAAI.Google ScholarCross Ref
- Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. 2009. Efficient graphlet kernels for large graph comparison. In Proc. of AISTATS.Google Scholar
- David I Shuman, Benjamin Ricaud, and Pierre Vandergheynst. 2012. A windowed graph Fourier transform. In 2012 IEEE Statistical Signal Processing Workshop (SSP).Google ScholarCross Ref
- Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (mas) and applications. In Proc. of WWW.Google ScholarDigital Library
- Zixing Song and Irwin King. 2022. Hierarchical Heterogeneous Graph Attention Network for Syntax-Aware Summarization. In Proc. of AAAI.Google ScholarCross Ref
- Zixing Song, Ziqiao Meng, Yifei Zhang, and Irwin King. 2021. Semi-supervised multi-label learning for graph-structured data. In Proc. of CIKM.Google ScholarDigital Library
- Zixing Song, Yifei Zhang, and Irwin King. 2022. Towards an optimal asymmetric graph structure for robust semi-supervised node classification. In Proc. of KDD.Google ScholarDigital Library
- Dawei Sun, Anbang Yao, Aojun Zhou, and Hao Zhao. 2019. Deeply-Supervised Knowledge Synergy. In Proc. of CVPR.Google ScholarCross Ref
- Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. 2020. InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. In Proc. of ICLR.Google Scholar
- Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. 2019. Deep Graph Infomax. In Proc. of ICLR.Google Scholar
- Menglin Yang, Zhihao Li, Min Zhou, Jiahong Liu, and Irwin King. 2022a. Hicf: Hyperbolic informative collaborative filtering. In Proc. of KDD.Google ScholarDigital Library
- Menglin Yang, Min Zhou, Marcus Kalander, Zengfeng Huang, and Irwin King. 2021. Discrete-time temporal network embedding via implicit hierarchical learning in hyperbolic space. In Proc. of KDD.Google ScholarDigital Library
- Menglin Yang, Min Zhou, Jiahong Liu, Defu Lian, and Irwin King. 2022b. HRCF: Enhancing collaborative filtering via hyperbolic geometric regularization. In Proceedings of the ACM Web Conference 2022.Google ScholarDigital Library
- Menglin Yang, Min Zhou, Hui Xiong, and Irwin King. 2022c. Hyperbolic Temporal Network Embedding. IEEE Transactions on Knowledge and Data Engineering.Google ScholarDigital Library
- Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. 2018. Hierarchical Graph Representation Learning with Differentiable Pooling. In Proc. of NeurIPS.Google Scholar
- Yifei Zhang, Dun Zeng, Jinglong Luo, Zenglin Xu, and Irwin King. 2023. A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy. ArXiv preprint.Google Scholar
- Yifei Zhang and Hao Zhu. 2019. Doc2hash: Learning Discrete Latent variables for Documents Retrieval. In Proc. of NAACL.Google Scholar
- Yifei Zhang and Hao Zhu. 2020a. Additively homomorphical encryption based deep neural network for asymmetrically collaborative machine learning. arXiv preprint arXiv:2007.06849 (2020).Google Scholar
- Yifei Zhang and Hao Zhu. 2020b. Discrete Wasserstein Autoencoders for Document Retrieval. In Proc. of ICASSP.Google ScholarCross Ref
- Yifei Zhang, Hao Zhu, Ziqiao Meng, Piotr Koniusz, and Irwin King. 2022a. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022.Google ScholarDigital Library
- Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. 2022b. COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning. In Proc. of KDD.Google ScholarDigital Library
- Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. 2022c. Spectral Feature Augmentation for Graph Contrastive Learning and Beyond. ArXiv preprint.Google Scholar
- Gangming Zhao, Weifeng Ge, and Yizhou Yu. 2021. GraphFPN: Graph Feature Pyramid Network for Object Detection. In Proc. of ICCV.Google ScholarCross Ref
- Hao Zhu and Piotr Koniusz. 2021. Simple spectral graph convolution. In International conference on learning representations.Google Scholar
- Hao Zhu and Piotr Koniusz. 2022a. EASE: Unsupervised discriminant subspace learning for transductive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9078--9088.Google ScholarCross Ref
- Hao Zhu and Piotr Koniusz. 2022b. Generalized laplacian eigenmaps. Advances in Neural Information Processing Systems , Vol. 35 (2022), 30783--30797.Google Scholar
- Hao Zhu, Ke Sun, and Peter Koniusz. 2021a. Contrastive Laplacian Eigenmaps. In Proc. of NeurIPS.Google Scholar
- Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. 2021b. An Empirical Study of Graph Contrastive Learning. ArXiv preprint.Google Scholar
- Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep graph contrastive representation learning. ArXiv preprint.Google Scholar
- Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2021c. Graph contrastive learning with adaptive augmentation. In Proc. of WWW. ioGoogle ScholarDigital Library
Index Terms
- Contrastive Cross-scale Graph Knowledge Synergy
Recommendations
Cross-view graph contrastive learning with hypergraph
AbstractGraph contrastive learning (GCL) provides a new perspective to alleviate the reliance on labeled data for graph representation learning. Recent efforts on GCL leverage various graph augmentation strategies, i.e., node dropping and edge masking, ...
Highlights- We proposed that hypergraphs are used as a paradigm to enhance graph contrastive learning.
- We propose a novel diffusion model-based fusion mechanism that aligns the positive examples.
- Our experimental results all exceed existing ...
Cross-scale contrastive triplet networks for graph representation learning
AbstractGraph representation learning aims to learn low-dimensional representation for the graph, which has played a vital role in real-world applications. Without requiring additional labeled data, contrastive learning based graph representation ...
Highlights- A contrastive learning framework for extracting multiple information is proposed.
- A combination of contextual and intrinsic information of graph data is used for node representation.
- The experimental results superior to the latest ...
Community-aware graph contrastive learning for collaborative filtering
AbstractRecently, graph neural networks have demonstrated superior performance in the field of collaborative filtering (CF). The graph collaborative filtering (GCF) method learns the interactions between users and items, whose performance is susceptible ...
Comments