Abstract
Graph neural networks (GNNs) have achieved significant success in various applications. Most GNNs learn the node features with information aggregation of its neighbors and feature transformation in each layer. However, the node features become indistinguishable after many layers, leading to performance deterioration: a significant limitation known as over-smoothing. Past work adopted various techniques for addressing this issue, such as normalization and skip-connection of layer-wise output. After the study, we found that the information aggregations in existing work are all contracted aggregations, with the intrinsic property that features will inevitably converge to the same single point after many layers. To this end, we propose the aggregation over compacted manifolds method (ACM) that replaces the existing information aggregation with aggregation over compact manifolds, a special type of manifold, which avoids contracted aggregations. In this work, we theoretically analyze contracted aggregation and its properties. We also provide an extensive empirical evaluation that shows ACM can effectively alleviate over-smoothing and outperforms the state-of-the-art. The code can be found in https://github.com/DongzhuoranZhou/ACM.git.
D. Zhou and H. Yang—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In Fig. 1b, the unit circle is used as the embedding space. The aggregation of points is defined under polar coordinates. For example, the aggregation of two points in the unit circle with polar coordinate \((1, \theta ), (1, \phi )\) is \((1, \frac{\theta +\phi }{2})\).
References
Bachmann, G., Bécigneul, G., Ganea, O.: Constant curvature graph convolutional networks. In: ICML, pp. 486–496. PMLR (2020)
Balazevic, I., Allen, C., Hospedales, T.M.: Multi-relational poincaré graph embeddings. In: NeurIPS, pp. 4465–4475 (2019)
Cai, C., Wang, Y.: A note on over-smoothing for graph neural networks. CoRR abs/ arXiv: 2006.13318 (2020)
Chami, I., et al: Hyperbolic graph convolutional neural networks. In: NeurIPS, pp. 4869–4880 (2019)
Chen, D., Lin, Y., et al.: Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. CoRR abs/ arXiv: 1909.03211 (2019)
Chen, M., Wei, Z., Huang, Z., Ding, B., Li, Y.: Simple and deep graph convolutional networks. In: ICML, pp. 1725–1735. PMLR (2020)
Chien, E., Peng, J., Li, P., Milenkovic, O.: Adaptive universal generalized pagerank graph neural network. In: ICLR (2021)
Ganea, O., Bécigneul, G., Hofmann, T.: Hyperbolic neural networks. In: NeurIPS, pp. 5350–5360 (2018)
Gao, H., Wang, Z., Ji, S.: Large-scale learnable graph convolutional networks. In: KDD, pp. 1416–1424. ACM (2018)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS, pp. 249–256 (2010)
Gülçehre, Ç., et al.: Hyperbolic attention networks. In: ICLR (2019)
Hamilton, W.L.: Graph Representation Learning. Synthesis Lect. Artifi. Intell. Mach. Learn. (2020)
Hamilton, W.L., et al.: Inductive representation learning on large graphs. In: NIPS, pp. 1024–1034 (2017)
He, K., et al.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE Computer Society (2016)
Hou, Y., Zhang, J., et al.: Measuring and improving the use of graph information in graph neural networks. In: ICLR (2020)
Huang, W., et al.: Tackling over-smoothing for general graph convolutional networks. CoRR (2020)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456 (2015)
Jin, W., Et al.: Feature overcorrelation in deep graph neural networks: a new perspective. In: KDD, pp. 709–719. ACM (2022)
Khrulkov, V., Et al.: Hyperbolic image embeddings. In: CVPR, pp. 6417–6427. Computer Vision Foundation/IEEE (2020)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. CoRR abs/ arxiv: 1609.02907 (2016)
Klicpera, J., Et al.: Predict then propagate: graph neural networks meet personalized pagerank. In: ICLR (2019)
. Klicpera, J., et al.: Predict then propagate: graph neural networks meet personalized pagerank. In: ICLR (2019)
Lee, J.: Introduction to Smooth Manifolds. Graduate Texts in Mathematics
Li, G., Müller, M., et al.: Deepgcns: can gcns go as deep as cnns? In: ICCV, pp. 9266–9275. IEEE (2019)
Li, Q., Han, Z., Wu, X.M.: Deeper insights into graph convolutional networks for semi-supervised learning. In: AAAI, pp. 3538–3545 (2018)
Liu, M., Gao, H., Ji, S.: Towards deeper graph neural networks. In: KDD, pp. 338–348. ACM (2020)
Mendelson, B.: Introduction to topology (1990)
Oono, K., Suzuki, T.: Graph neural networks exponentially lose expressive power for node classification. In: ICLR (2020)
Pei, H., et al.: Geom-gcn: Geometric graph convolutional networks. In: ICLR (2020)
Rashid, A.M., Karypis, G., et al.: Learning preferences of new users in recommender systems: an information theoretic approach. SIGKDD Explor., 90–100 (2008)
Rong, Y., Huang, W., Xu, T., Huang, J.: Dropedge: towards deep graph convolutional networks on node classification. In: ICLR (2020)
Shchur, O., Mumme, M., Bojchevski, A., Günnemann, S.: Pitfalls of graph neural network evaluation. CoRR abs/ arXiv: 1811.05868 (2018)
Srivastava, N., et al.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Ungar, A.A.: Barycentric calculus in Euclidean and hyperbolic geometry: a comparative introduction (2010)
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. CoRR abs/ arXiv: 1710.10903 (2017)
Wu, F., et al.: Simplifying graph convolutional networks. In: ICML, vol. 97, pp. 6861–6871. PMLR (2019)
Xu, K., Hu, W., et al.: How powerful are graph neural networks? In: ICLR (2019)
Xu, K., Li, C., Tian, Y., et al.: Representation learning on graphs with jumping knowledge networks. In: ICML, pp. 5449–5458. PMLR (2018)
Yang, Z., et al.: Revisiting semi-supervised learning with graph embeddings. In: ICML. JMLR Workshop and Conference Proceedings, vol. 48, pp. 40–48 (2016)
Zhao, L., Akoglu, L.: Pairnorm: tackling oversmoothing in gnns. In: ICLR (2020)
Zhou, J., Cui, G., et al.: Graph neural networks: a review of methods and applications. AI Open 1, 57–81 (2020)
Zhou, K., et al.: Towards deeper graph neural networks with differentiable group normalization. In: NeurIPS (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhou, D., Yang, H., Xiong, B., Ma, Y., Kharlamov, E. (2024). Alleviating Over-Smoothing via Aggregation over Compact Manifolds. In: Yang, DN., Xie, X., Tseng, V.S., Pei, J., Huang, JW., Lin, J.CW. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2024. Lecture Notes in Computer Science(), vol 14646. Springer, Singapore. https://doi.org/10.1007/978-981-97-2253-2_31
Download citation
DOI: https://doi.org/10.1007/978-981-97-2253-2_31
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-2252-5
Online ISBN: 978-981-97-2253-2
eBook Packages: Computer ScienceComputer Science (R0)