Skip to main content
Springer Nature Link
Log in

Alleviating over-smoothing via graph sparsification based on vertex feature similarity

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In recent years, graph neural networks (GNNs) have developed rapidly. However, GNNs are difficult to deepen because of over-smoothing. This limits their applications. Starting from the relationship between graph sparsification and over-smoothing, for the problems existing in current graph sparsification methods, we propose a novel graph sparsification method SimSparse based on vertex feature similarity and theoretically prove that it can help to relieve the over-smoothing. Furthermore, we also propose its derivatives SimSparse-G (SimSparse with G umbel-Softmax) and SimSparse-GC (SimSparse-G with memory occupancy reduC tion), which can achieve end-to-end learning. Moreover, SimSparse-GC can apply to larger graphs. Based on the three graph sparsification methods, we further propose a general sparse-convolution block SparseConvBlock with a sparsification layer and a graph convolutional layer to construct deep GNNs in which over-smoothing can be better alleviated. Extensive experiments show that the GNNs constructed by SparseConvBlock have better performance when the numbers of the network layers increase (i.e., in deep GNNs). Furthermore, we verify that our graph sparsification methods help to relieve over-smoothing in deep GNNs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Algorithm 2
Fig. 1
Algorithm 3
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Batson JD, Spielman DA, Srivastava N et al (2013) Spectral sparsification of graphs: theory and algorithms. Commun ACM 56(8):87–94. https://doi.org/10.1145/2492007.2492029

    Article  Google Scholar 

  2. Bruna J, Zaremba W, Szlam A et al (2014) Spectral networks and locally connected networks on graphs. In: 2nd international conference on learning representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings

  3. Cai C, Wang Y (2020) A note on over-smoothing for graph neural networks. arXiv:2006.13318 [cs.LG]

  4. Chen J, Ma T, Xiao C (2018) Fastgcn: fast learning with graph convolutional networks via importance sampling. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings

  5. Chen M, Wei Z, Huang Z et al (2020) Simple and deep graph convolutional networks. In: Proceedings of the 37th international conference on machine learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, vol 119. PMLR, pp 1725–1735

  6. Chung F (1997) Spectral graph theory. Published for the Conference Board of the mathematical sciences by the American Mathematical Society, Providence, R.I

  7. Cohen E (1998) Fast algorithms for constructing t-spanners and paths with stretch t. SIAM J Comput 28(1):210–236. https://doi.org/10.1137/s0097539794261295

    Article  MathSciNet  MATH  Google Scholar 

  8. Corso G, Cavalleri L, Beaini D et al (2020) Principal neighbourhood aggregation for graph nets. In: Larochelle H, Ranzato M, Hadsell R et al (eds) Advances in Neural Information Processing Systems, vol 33. Curran Associates Inc. pp 13,260–13,271. https://proceedings.neurips.cc/paper/2020/file/99cad265a1768cc2dd013f0e740300ae-Paper.pdf

  9. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in neural information processing systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10 2016, Barcelona, Spain, pp 3837–3845

  10. Duvenaud D, Maclaurin D, Aguilera-Iparraguirre J et al (2015) Convolutional networks on graphs for learning molecular fingerprints. In: Advances in neural information processing systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12 2015 Montreal, Quebec, Canada, pp 2224–2232

  11. Gilmer J, Schoenholz SS, Riley PF et al (2017) Neural message passing for quantum chemistry. In: Proceedings of the 34th international conference on machine learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, Proceedings of Machine Learning Research, vol 70. PMLR, pp 1263–1272

  12. Hamilton WL, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs. In: Advances in neural information processing systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9 2017 Long Beach, CA, USA, pp 1024–1034

  13. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2016.90

  14. Hu W, Fey M, Zitnik M et al (2020) Open graph benchmark: Datasets for machine learning on graphs. In: Larochelle H, Ranzato M, Hadsell R et al (eds) Advances in Neural Information Processing Systems, vol 33. Curran Associates Inc, pp 22,118–22,133. https://proceedings.neurips.cc/paper/2020/file/fb60d411a5c5b72b2e7d3527cfc84fd0-Paper.pdf

  15. Huang G, Liu Z, Maaten LVD et al (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2017.243

  16. Jang E, Gu S, Poole B (2017) Categorical reparameterization with gumbel-softmax. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=rkE3y85ee

  17. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings

  18. Klicpera J, Bojchevski A, Günnemann S (2019) Predict then propagate: Graph neural networks meet personalized pagerank. In: 7th international conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6-9 2019

  19. Levie R, Monti F, Bresson X et al (2019) Cayleynets: graph convolutional neural networks with complex rational spectral filters. IEEE Trans Signal Process 67(1):97–109. https://doi.org/10.1109/TSP.2018.2879624

    Article  MathSciNet  MATH  Google Scholar 

  20. Li G, Muller M, Thabet A et al (2019) DeepGCNs: can GCNs go as deep as CNNs?. In: IEEE/CVF International Conference on Computer Vision (ICCV). IEEE. https://doi.org/10.1109/iccv.2019.00936

  21. Li Q, Han Z, Wu X (2018) Deeper insights into graph convolutional networks for semi-supervised learning. In: Proceedings of the thirty-second AAAI conference on artificial intelligence, (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. AAAI Press, pp 3538–3545

  22. Louizos C, Welling M, Kingma DP (2018) Learning sparse neural networks through l_0 regularization. In: International conference on learning representations. https://openreview.net/forum?id=H1Y8hhg0b

  23. Luo D, Cheng W, Yu W et al (2021) Learning to drop: robust graph neural network via topological denoising. In: Proceedings of the 14th ACM international conference on web search and data mining. ACM. https://doi.org/10.1145/3437963.3441734

  24. Min Y, Wenkel F, Wolf G (2020) Scattering GCN: overcoming oversmoothness in graph convolutional networks. In: Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

  25. Monti F, Boscaini D, Masci J et al (2017) Geometric deep learning on graphs and manifolds using mixture model cnns. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, pp 5425–5434. https://doi.org/10.1109/CVPR.2017.576

  26. Oono K, Suzuki T (2020) Graph neural networks exponentially lose expressive power for node classification. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30 2020

  27. Pan S, Wu J, Zhu X et al (2016) Tri-party deep network representation. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016. IJCAI/AAAI Press, pp 1895–1901

  28. Rong Y, Huang W, Xu T et al (2020) Dropedge: towards deep graph convolutional networks on node classification. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

  29. Sen P, Namata G, Bilgic M et al (2008) Collective classification in network data. AI Mag 29(3):93–106. https://doi.org/10.1609/aimag.v29i3.2157

    Google Scholar 

  30. Spielman DA, Srivastava N (2011) Graph sparsification by effective resistances. SIAM J Comput 40(6):1913–1926. https://doi.org/10.1137/080734029

    Article  MathSciNet  MATH  Google Scholar 

  31. Velickovic P, Cucurull G, Casanova A et al (2018) Graph attention networks. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings

  32. Voudigari E, Salamanos N, Papageorgiou T et al (2016) Rank degree: an efficient algorithm for graph sampling. In: Kumar R, Caverlee J, Tong H (eds) 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2016, San Francisco, CA, USA, August 18-21, 2016. IEEE Computer Society, pp 120–129. https://doi.org/10.1109/ASONAM.2016.7752223

  33. Wang J, Deng Z (2021) A deep graph wavelet convolutional neural network for semi-supervised node classification. In: 2021 international joint conference on neural networks (IJCNN). IEEE. https://doi.org/10.1109/ijcnn52387.2021.9533634

  34. Wang K, Shen Z, Huang C et al (2020) Microsoft academic graph: when experts are not enough. Quantitative Science Studies 1(1):396–413. https://doi.org/10.1162/qss_a_00021

    Article  Google Scholar 

  35. Wu F, AHS Jr, Zhang T et al (2019) Simplifying graph convolutional networks. In: Proceedings of the 36th international conference on machine learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research, vol 97. PMLR, pp 6861–6871

  36. Xu J (2017) Representing big data as networks: new methods and insights. PhD thesis. https://curate.nd.edu/show/q524jm2466t

  37. Xu K, Li C, Tian Y et al (2018) Representation learning on graphs with jumping knowledge networks. In: Dy JG, Krause A (eds) Proceedings of the 35th international conference on machine learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, Proceedings of Machine Learning Research, vol 80. PMLR, pp 5449–5458. http://proceedings.mlr.press/v80/xu18c.html

  38. Zhao L, Akoglu L (2020) Pairnorm: Tackling oversmoothing in gnns. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30 2020

  39. Zheng C, Zong B, Cheng W et al (2020) Robust graph representation learning via neural sparsification. In: III HD, Singh A (eds) Proceedings of the 37th international conference on machine learning, Proceedings of Machine Learning Research, vol 119. PMLR, pp 11,458–11,468. http://proceedings.mlr.press/v119/zheng20d.html

  40. Zhou K, Huang X, Li Y et al (2020) Towards deeper graph neural networks with differentiable group normalization. In: Larochelle H, Ranzato M, Hadsell R et al (eds) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. https://proceedings.neurips.cc/paper/2020/hash/33dd6dba1d56e826aac1cbf23cdcca87-Abstract.html

  41. Zhu H, Koniusz P (2021) Simple spectral graph convolution. In: 9th international conference on learning representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=CYO5T-YjWZV

Download references

Acknowledgements

This study was supported by Science Research Project of Liaoning Department of Education of China (No. LJKZ0008) and Science and Technology Development Project of Liaoning Province of China (No. 2021JH6/10500127).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Northeastern University, Chuangxin Road, Shenyang, 110169, Liaoning, China

    Gongce Wu, Shukuan Lin, Yilin Zhuang & Jianzhong Qiao

Authors
  1. Gongce Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shukuan Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yilin Zhuang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Jianzhong Qiao
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Shukuan Lin.

Ethics declarations

Conflict of Interests

The authors declare no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, G., Lin, S., Zhuang, Y. et al. Alleviating over-smoothing via graph sparsification based on vertex feature similarity. Appl Intell 53, 20223–20238 (2023). https://doi.org/10.1007/s10489-023-04537-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04537-0

Keywords