Skip to main content
Springer Nature Link
Log in

A Comprehensive Review of the Oversmoothing in Graph Neural Networks

  • Conference paper
  • First Online:
Computer Supported Cooperative Work and Social Computing (ChineseCSCW 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2012))

  • 875 Accesses

Abstract

There are many ways to process graph data in deep learning, among which Graph Neural Network(GNN) is an effective and popular deep learning model. However, GNN also has some problems. For example, after multiple layers of neural networks, the features between nodes will become more and more similar, so that the model identifies two completely different nodes as one type. For example, when two nodes with different structural information output, they are almost the same at the feature level and thus difficult to be distinguished, and this phenomenon is called oversmoothing. For example, in node classification, two completely different types of nodes obtain highly similar node features after model training. How to alleviate and solve the oversmoothing problem has become an emerging hot research topic in graph research. However, there has yet to be an extensive investigation and evaluation of this topic. This paper aims to summarize different approaches to mitigate the oversmoothing phenomenon by providing a detailed research survey. We analyze and summarize proposed research schemes from three aspects currently: topological perturbation, message passing, and adaptive learning, and evaluate the strengths and limitations of existing research by outlining oversmoothing evaluation methods. In addition, we predict and summarize promising and possible research paths in the future. In doing so, this paper contributes to the development of GNN and provides insightful information for practitioners working with GNN and graph data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bhagat, S., Cormode, G.: Node classification in social networks. In: Proceedings of the 2011 SIAM International Conference on Data Mining, pp. 201–212 (2011)

    Google Scholar 

  2. Chen, D., Lin, Y., Li, W., Li, P., Zhou, J., Sun, X.: Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7–12, 2020, pp. 3438–3445. AAAI Press (2020). https://ojs.aaai.org/index.php/AAAI/article/view/5747

  3. Chen, H., Li, Y.: Node-smoothness based adaptive initial residual deep graph convolutional network (2022). Available at SSRN 4254779

    Google Scholar 

  4. Chen, J., Zhong, M., Li, J., Wang, D., Qian, T., Tu, H.: Effective deep attributed network representation learning with topology adapted smoothing. IEEE Trans. Cybern. 52(7), 5935–5946 (2022). https://doi.org/10.1109/TCYB.2021.3064092

    Article  Google Scholar 

  5. Chen, M., Wei, Z., Huang, Z., Ding, B., Li, Y.: Simple and deep graph convolutional networks. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 1725–1735. PMLR (2020). http://proceedings.mlr.press/v119/chen20v.html

  6. Chen, Y., Li, Z., Xiao, X., Zhang, K., Lu, H.: Simple and effective graph convolutional networks with graph attention convolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11459–11466 (2020)

    Google Scholar 

  7. Danescu-Niculescu-Mizil, C., Lee, L., Pang, B., Kleinberg, J.: Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 123–132. ACM, New York, NY, USA (2014). https://doi.org/10.1145/2623330.2623623, https://www.cs.cornell.edu/ cristian/Cornell_Movie-Dialogs_Corpus.html

  8. Debnath, A.K., Lopez de Compadre, R., Debnath, G.C., Shusterman, A., Hansch, C.: Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. J. Med. Chem. 34, 786–797 (1991)

    Google Scholar 

  9. Ding, H., Takigawa, I., Mamitsuka, H., Zhu, S.: Similarity-based machine learning methods for predicting drug-target interactions: a brief review. Brief. Bioinform. 15(5), 734–747 (2014)

    Article  Google Scholar 

  10. Do, T.H., Nguyen, D.M., Bekoulis, G., Munteanu, A., Deligiannis, N.: Graph convolutional neural networks with node transition probability-based message passing and DropNode regularization. Expert Syst. Appl. 174, 114711 (2021). https://doi.org/10.1016/j.eswa.2021.114711

    Article  Google Scholar 

  11. Fang, T., Xiao, Z., Wang, C., Xu, J., Yang, X., Yang, Y.: DropMessage: unifying random dropping for graph neural networks. CoRR abs/2204.10037 (2022). https://doi.org/10.48550/arXiv.2204.10037

  12. Giles, C.L., Bollacker, K.D., Lawrence, S.: CiteSeer: an automatic citation indexing system. In: Proceedings of the 3rd ACM Conference on Digital Libraries, pp. 89–98. ACM (1998)

    Google Scholar 

  13. Guo, H., Sun, S.: SoftEdge: regularizing graph classification with random soft edges. CoRR abs/2204.10390 (2022). https://doi.org/10.48550/arXiv.2204.10390

  14. Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems, pp. 1024–1034 (2017)

    Google Scholar 

  15. Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: data, models and evaluation metrics. J. Artif. Intell. Res. 47, 853–899 (2013)

    Article  MathSciNet  Google Scholar 

  16. Hu, W., et al.: Open graph benchmark: datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 (2021)

  17. Jin, W., Barz, B., Li, M.: Graphdiff: differential privacy for graph neural networks via topology change. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11024–11031 (2020)

    Google Scholar 

  18. Klicpera, J., Bojchevski, A., Günnemann, S.: Diffusion improves graph learning. In: Proceedings of the 36th International Conference on Machine Learning (ICML), vol. 97, pp. 3651–3661 (2019)

    Google Scholar 

  19. Koishekenov, Y.: Reducing over-smoothing in graph neural networks using relational embeddings. CoRR abs/2301.02924 (2023). https://doi.org/10.48550/arXiv.2301.02924

  20. Li, H., et al.: DeeperGCN: all you need to train deeper GCNs. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3332–3342. ACM (2020)

    Google Scholar 

  21. Li, Q., Han, Z., Wu, X.: Deeper insights into graph convolutional networks for semi-supervised learning. CoRR abs/1801.07606 (2018). http://arxiv.org/abs/1801.07606

  22. Luo, D., et al.: Learning to drop: robust graph neural network via topological denoising. In: Lewin-Eytan, L., Carmel, D., Yom-Tov, E., Agichtein, E., Gabrilovich, E. (eds.) WSDM 2021, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8–12, 2021, pp. 779–787. ACM (2021). https://doi.org/10.1145/3437963.3441734

  23. Mai, S., Zheng, S., Sun, Y., Zeng, Y., Yang, Y., Hu, H.: Dynamic graph dropout for subgraph-based relation prediction. Knowl. Based Syst. 250, 109172 (2022). https://doi.org/10.1016/j.knosys.2022.109172

    Article  Google Scholar 

  24. Papp, P.A., Martinkus, K., Faber, L., Wattenhofer, R.: DropGNN: random dropouts increase the expressiveness of graph neural networks. In: Ranzato, M., Beygelzimer, A., Dauphin, Y.N., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021(December), pp. 6–14, 2021. virtual, vol. 34, pp. 21997–22009 (2021). https://proceedings.neurips.cc/paper/2021/hash/b8b2926bd27d4307569ad119b6025f94-Abstract.html

  25. Perozzi, B., Al-Rfou, R., Skiena, S.: DeepWalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 701–710 (2014)

    Google Scholar 

  26. Rong, Y., Huang, W., Xu, T., Huang, J.: DropEdge: towards deep graph convolutional networks on node classification. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net (2020). https://openreview.net/forum?id=Hkx1qkrKPr

  27. Rusch, T.K., Bronstein, M.M., Mishra, S.: A survey on oversmoothing in graph neural networks. CoRR abs/2303.10993 (2023). https://doi.org/10.48550/arXiv.2303.10993

  28. Sayers, E.W., et al.: PubBed: a resource for curated biomedical literature. Nucleic Acids Res. 39(Database issue), D1268–D1271 (2011)

    Google Scholar 

  29. Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., Eliassi-Rad, T.: Collective classification in network data. In: AI Magazine, vol. 29, pp. 93–93 (2008)

    Google Scholar 

  30. Shervashidze, N., Schweitzer, P., van Leeuwen, E.J., Mehlhorn, K., Borgwardt, K.M.: Weisfeiler-Lehman graph kernels. J. Mach. Learn. Res. 12(Sep), 2539–2561 (2011)

    MathSciNet  Google Scholar 

  31. Spinelli, I., Scardapane, S., Hussain, A., Uncini, A.: FairDrop: biased edge dropout for enhancing fairness in graph representation learning. IEEE Trans. Artif. Intell. 3(3), 344–354 (2022). https://doi.org/10.1109/TAI.2021.3133818

    Article  Google Scholar 

  32. Stark, C., Breitkreutz, B.J., Reguly, T., Boucher, L., Breitkreutz, A., Tyers, M.: BioGRID: a general repository for interaction datasets. Nucleic Acids Res. 34(Database issue), D535–D539 (2006)

    Article  Google Scholar 

  33. Street, W.N., Wolberg, W.H., Mangasarian, O.L.: Wisconsin (Original) Breast Cancer Dataset. UCI Machine Learning Repository (1995)

    Google Scholar 

  34. Sun, F.: Over-smoothing effect of graph convolutional networks. CoRR abs/2201.12830 (2022). https://arxiv.org/abs/2201.12830

  35. Tan, J., Chang, S., Zettlemoyer, L.S.: Linguistically informed character-level language models for unsupervised named entity recognition. arXiv preprint arXiv:1505.05008 (2015)

  36. Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., Mei, Q.: Line: large-scale information network embedding. In: Proceedings of the 24th International Conference on World Wide Web, pp. 1067–1077 (2015)

    Google Scholar 

  37. Tang, L., Liu, H.: Relational learning via latent social dimensions. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 817–826. ACM (2009)

    Google Scholar 

  38. Toivonen, H., Kaski, S., Nikkilä, J., Vähänikkilä, M., Hautaniemi, S.: Statistical evaluation of term occurrences for discovering differentially expressed genes. J. Comput. Biol. 10(4), 447–464 (2003)

    Google Scholar 

  39. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (2018)

    Google Scholar 

  40. Wale, N., Karypis, G.: Comparison of descriptor spaces for chemical compound retrieval and classification. Knowl. Inf. Syst. 14(3), 347–375 (2008)

    Article  Google Scholar 

  41. Wang, H., Wang, J., Wang, J., Zhao, M., Zhang, W.: Knowledge-aware graph neural networks with label smoothness regularization for recommender systems. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1221–1230 (2019)

    Google Scholar 

  42. Wikipedia Contributors: Wikipedia (2023)

    Google Scholar 

  43. Wu, F., Zhang, T., Souza, A., Fifty, C., Yu, T., Weinberger, K.Q.: Simplifying graph convolutional networks. In: Proceedings of the 36th International Conference on Machine Learning, pp. 6861–6871 (2019)

    Google Scholar 

  44. Wu, Q., Zhao, W., Li, Z., Wipf, D., Yan, J.: NodeFormer: a scalable graph structure learning transformer for node classification. CoRR abs/2306.08385 (2023). https://doi.org/10.48550/arXiv.2306.08385

  45. Xu, K., Li, C., Tian, Y., Sonobe, T., Kawarabayashi, K.i., Jegelka, S.: Representation learning on graphs with jumping knowledge networks. In: Proceedings of the 35th International Conference on Machine Learning (ICML), vol. 80, pp. 5421–5430 (2018)

    Google Scholar 

  46. Xu, K., Li, C., Tian, Y., Sonobe, T., Kawarabayashi, K.i., Jegelka, S.: How powerful are graph neural networks? In: International Conference on Learning Representations (2019)

    Google Scholar 

  47. Yan, Y., Hashemi, M., Swersky, K., Yang, Y., Koutra, D.: Two sides of the same coin: heterophily and oversmoothing in graph convolutional neural networks. In: Zhu, X., Ranka, S., Thai, M.T., Washio, T., Wu, X. (eds.) IEEE International Conference on Data Mining, ICDM 2022, Orlando, FL, USA, November 28 - Dec. 1, 2022, pp. 1287–1292. IEEE (2022). https://doi.org/10.1109/ICDM54844.2022.00169

  48. Yanardag, P., Vishwanathan, S.: Deep graph kernels. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 1365–1374 (2015)

    Google Scholar 

  49. Ye, Y., Ji, S.: Sparse graph attention networks. IEEE Trans. Knowl. Data Eng. 35(1), 905–916 (2023)

    MathSciNet  Google Scholar 

  50. Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W.L., Leskovec, J.: Graph convolutional neural networks for web-scale recommender systems. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2018)

    Google Scholar 

  51. Zeng, S., Yang, J., Liu, W., Liu, Q.: GraphSAINT: graph sampling based inductive learning method. In: International Conference on Learning Representations (ICLR) (2020)

    Google Scholar 

  52. Zhang, S., Zhu, F., Yan, J., Zhao, R., Yang, X.: DOTIN: dropping task-irrelevant nodes for GNNs. CoRR abs/2204.13429 (2022). https://doi.org/10.48550/arXiv.2204.13429

  53. Zhang, S., Du, L., Li, F., Yu, G., Chen, M.: Propagate deeper and adaptive graph convolutional networks. In: ICLR (2023)

    Google Scholar 

  54. Zhang, X., Li, Y., Zhuang, Y., Zhou, Q.: Towards deeper graph neural networks with differentiable group normalization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4609–4616 (2019)

    Google Scholar 

  55. Zheng, C., et al.: Robust graph representation learning via neural sparsification. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 11458–11468. PMLR (2020). http://proceedings.mlr.press/v119/zheng20d.html

  56. Zhou, X., Wu, O.: Drop “noise” edge: an approximation of the Bayesian GNNs. In: Wallraven, C., Liu, Q., Nagahara, H. (eds.) Pattern Recognition - 6th Asian Conference, ACPR 2021, Jeju Island, South Korea, November 9–12, 2021, Revised Selected Papers, Part II. Lecture Notes in Computer Science, vol. 13189, pp. 59–72. Springer (2021). https://doi.org/10.1007/978-3-031-02444-3_5

  57. Zhu, J., Mao, G., Jiang, C.: DII-GCN: dropedge based deep graph convolutional networks. Symmetry 14(4), 798 (2022). https://doi.org/10.3390/sym14040798

    Article  Google Scholar 

  58. Zou, X., Li, K., Chen, C., Yang, X., Wei, W., Li, K.: DGSLN: differentiable graph structure learning neural network for robust graph representations. Inf. Sci. 626, 94–113 (2023). https://doi.org/10.1016/j.ins.2023.01.059

    Article  Google Scholar 

Download references

Acknowledgements

This paper is partly supported by National Key R &D Program of China No.2021YFF0900800, NSFC No.62202279, Shandong Provincial Key Research and Development Program (Major Scientific and Technological Innovation Project) No.2021CXGC010108, Shandong Provincial Natural Science Foundation No. ZR2022QF018, Shandong Provincial Outstanding Youth Science Foundation No. 2023HWYQ-039, Fundamental Research Funds of Shandong University.

Author information

Authors and Affiliations

  1. School of Software, Shandong University, Jinan, 250101, China

    Xu Zhang, Wei He, Wei Guo & Lizhen Cui

  2. Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University, Jinan, 250101, China

    Yonghui Xu & Lizhen Cui

Authors
  1. Xu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonghui Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lizhen Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lizhen Cui .

Editor information

Editors and Affiliations

  1. Shandong University, Jinan, China

    Yuqing Sun

  2. Fudan University, Shanghai, China

    Tun Lu

  3. Harbin Engineering University, Harbin, China

    Tong Wang

  4. Tongji University, Shanghai, China

    Hongfei Fan

  5. Guangdong University of Technology, Guangzhou, China

    Dongning Liu

  6. Tongji University, Shanghai, China

    Bowen Du

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, X., Xu, Y., He, W., Guo, W., Cui, L. (2024). A Comprehensive Review of the Oversmoothing in Graph Neural Networks. In: Sun, Y., Lu, T., Wang, T., Fan, H., Liu, D., Du, B. (eds) Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2023. Communications in Computer and Information Science, vol 2012. Springer, Singapore. https://doi.org/10.1007/978-981-99-9637-7_33

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-9637-7_33

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-9636-0

  • Online ISBN: 978-981-99-9637-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships