Yang Liu
Chuan Zhou
ABSTRACT
Graph neural networks (GNNs) are powerful models to handle graph data and can achieve state-of-the-art in many critical tasks including node classification and link prediction. However, existing graph neural networks still face both challenges of over-smoothing and over-squashing based on previous literature. To this end, we propose a new Curvature-based topology-aware Dropout sampling technique named CurvDrop, in which we integrate the Discrete Ricci Curvature into graph neural networks to enable more expressive graph models. Also, this work can improve graph neural networks by quantifying connections in graphs and using structural information such as community structures in graphs. As a result, our method can tackle the both challenges of over-smoothing and over-squashing with theoretical justification. Also, numerous experiments on public datasets show the effectiveness and robustness of our proposed method. The code and data are released in https://github.com/liu-yang-maker/Curvature-based-Dropout.
- Uri Alon and Eran Yahav. 2020. On the Bottleneck of Graph Neural Networks and its Practical Implications. In International Conference on Learning Representations.Google Scholar
- Shuliang Bai, An Huang, Linyuan Lu, and Shing-Tung Yau. 2020. On the sum of Ricci-curvatures for weighted graphs. arXiv preprint arXiv:2001.01776 (2020).Google Scholar
- Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, and Stephan Günnemann. 2018. Netgan: Generating graphs via random walks. In International conference on machine learning. PMLR, 610–619.Google Scholar
- Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 3438–3445.Google ScholarCross Ref
- Jie Chen, Tengfei Ma, and Cao Xiao. 2018. FastGCN: Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR.Google Scholar
- Weilin Cong, Rana Forsati, Mahmut Kandemir, and Mehrdad Mahdavi. 2020. Minimal variance sampling with provable guarantees for fast training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1393–1403.Google ScholarDigital Library
- Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. 2017. Protein Interface Prediction Using Graph Convolutional Networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6533–6542.Google ScholarDigital Library
- Yang Gao, Peng Zhang, Zhao Li, Chuan Zhou, Yongchao Liu, and Yue Hu. 2021. Heterogeneous Graph Neural Architecture Search. In 2021 IEEE International Conference on Data Mining (ICDM). IEEE, 1066–1071.Google Scholar
- Yang Gao, Peng Zhang, Chuan Zhou, Hong Yang, Zhao Li, Yue Hu, and Philip S. Yu. 2023. HGNAS++: Efficient Architecture Search for Heterogeneous Graph Neural Networks. IEEE Transactions on Knowledge and Data Engineering (2023), 1–14. https://doi.org/10.1109/TKDE.2023.3239842Google ScholarDigital Library
- Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. 2019. Predict then Propagate: Graph Neural Networks meet Personalized PageRank. In International Conference on Learning Representations (ICLR).Google Scholar
- Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In International conference on machine learning. PMLR, 1263–1272.Google Scholar
- William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NIPS.Google Scholar
- Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. 2018. Adaptive sampling towards fast graph representation learning. Advances in neural information processing systems 31 (2018).Google Scholar
- Anees Kazi, Shayan Shekarforoush, S Arvind Krishna, Hendrik Burwinkel, Gerome Vivar, Karsten Kortüm, Seyed-Ahmad Ahmadi, Shadi Albarqouni, and Nassir Navab. 2019. InceptionGCN: receptive field aware graph convolutional network for disease prediction. In International Conference on Information Processing in Medical Imaging. Springer, 73–85.Google ScholarCross Ref
- Mark Kempton, Gabor Lippner, and Florentin Münch. 2020. Large scale Ricci curvature on graphs. Calculus of variations and partial differential equations 59, 5 (2020), 1–17.Google Scholar
- Thomas N Kipf and Max Welling. 2016. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations.Google Scholar
- Chang Li and Dan Goldwasser. 2019. Encoding Social Information with Graph Convolutional Networks forPolitical Perspective Detection in News Media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 2594–2604. https://doi.org/10.18653/v1/P19-1247Google ScholarCross Ref
- Haifeng Li, Jun Cao, Jiawei Zhu, Yu Liu, Qing Zhu, and Guohua Wu. 2022. Curvature graph neural network. Information Sciences 592 (2022), 50–66. https://doi.org/10.1016/j.ins.2021.12.077Google ScholarDigital Library
- Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. 2019. Actional-structural graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3595–3603.Google ScholarCross Ref
- Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning. In AAAI. AAAI Press, 3538–3545.Google Scholar
- Yong Lin, Linyuan Lu, and Shing-Tung Yau. 2011. Ricci curvature of graphs. Tohoku Mathematical Journal, Second Series 63, 4 (2011), 605–627.Google ScholarCross Ref
- Yong Lin and Shing-Tung Yau. 2010. Ricci curvature and eigenvalue estimate on locally finite graphs. Mathematical research letters 17, 2 (2010), 343–356.Google Scholar
- László Lovász. 1993. Random walks on graphs. Combinatorics, Paul erdos is eighty 2, 1-46 (1993), 4.Google Scholar
- Chien-Chun Ni, Yu-Yao Lin, Feng Luo, and Jie Gao. 2019. Community detection on networks with Ricci flow. Scientific reports 9, 1 (2019), 1–12.Google Scholar
- Yann Ollivier. 2009. Ricci curvature of Markov chains on metric spaces. Journal of Functional Analysis 256, 3 (2009), 810–864.Google ScholarCross Ref
- Kenta Oono and Taiji Suzuki. 2019. On asymptotic behaviors of graph cnns from dynamical systems perspective. (2019).Google Scholar
- Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2020. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. In International Conference on Learning Representations. https://openreview.net/forum¿id=Hkx1qkrKPrGoogle Scholar
- Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine 29, 3 (2008), 93–93.Google Scholar
- Marco Serafini and Hui Guan. 2021. Scalable graph neural network training: The case for sampling. ACM SIGOPS Operating Systems Review 55, 1 (2021), 68–76.Google ScholarDigital Library
- Weijing Shi and Raj Rajkumar. 2020. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1711–1719.Google ScholarCross Ref
- Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Jiuxin Cao, Yingxia Shao, and Nguyen Quoc Viet Hung. 2021. Heterogeneous hypergraph embedding for graph classification. In Proceedings of the 14th acm international conference on web search and data mining. 725–733.Google ScholarDigital Library
- Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Qing Meng, Wang Han, and Jiuxin Cao. 2021. Multi-level hyperedge distillation for social linking prediction on sparsely observed networks. In Proceedings of the Web Conference 2021. 2934–2945.Google ScholarDigital Library
- Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. 2022. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations.Google Scholar
- Guangtao Wang, Rex Ying, Jing Huang, and Jure Leskovec. 2021. Multi-hop Attention Graph Neural Networks. In IJCAI.Google Scholar
- Haibo Wang, Chuan Zhou, Xin Chen, Jia Wu, Shirui Pan, and Jilong Wang. 2020. Graph Stochastic Neural Networks for Semi-supervised Learning. In The Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS-20).Google Scholar
- Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018. Representation Learning on Graphs with Jumping Knowledge Networks. In ICML.Google Scholar
- Ze Ye, Kin Sum Liu, Tengfei Ma, Jie Gao, and Chao Chen. 2020. Curvature Graph Network. In International Conference on Learning Representations. https://openreview.net/forum¿id=BylEqnVFDBGoogle Scholar
- Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. 2020. Deep graph neural networks with shallow subgraph samplers. arXiv preprint arXiv:2012.01380 (2020).Google Scholar
- Kai Zhang, Yaokang Zhu, Jun Wang, and Jie Zhang. 2020. Adaptive Structural Fingerprints for Graph Attention Networks. In International Conference on Learning Representations. https://openreview.net/forum¿id=BJxWx0NYPrGoogle Scholar
- Shichao Zhu, Shirui Pan, Chuan Zhou, Jia Wu, Yanan Cao, and Bin Wang. 2020. Graph Geometry Interaction Learning. In The Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS-20).Google Scholar
- Fariba Zohrizadeh, Mohsen Kheirandishfard, Farhad Kamangar, and Ramtin Madani. 2019. Non-smooth Optimization over Stiefel Manifolds with Applications to Dimensionality Reduction and Graph Clustering.. In IJCAI. 1319–1326.Google Scholar
Index Terms
- CurvDrop: A Ricci Curvature Based Approach to Prevent Graph Neural Networks from Over-Smoothing and Over-Squashing
Recommendations
On the Trade-off between Over-smoothing and Over-squashing in Deep Graph Neural Networks
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge ManagementGraph Neural Networks (GNNs) have succeeded in various computer science applications, yet deep GNNs underperform their shallow counterparts despite deep learning's success in other domains. Over-smoothing and over-squashing are key challenges when ...
Alleviating over-smoothing via graph sparsification based on vertex feature similarity
AbstractIn recent years, graph neural networks (GNNs) have developed rapidly. However, GNNs are difficult to deepen because of over-smoothing. This limits their applications. Starting from the relationship between graph sparsification and over-smoothing, ...
Inflation Improves Graph Neural Networks
WWW '22: Proceedings of the ACM Web Conference 2022Graph neural networks (GNNs) have gained significant success in graph representation learning and become the go-to approach for many graph-based tasks. Despite their effectiveness, the performance of GNNs is known to decline gradually as the number of ...
Comments