Skip to main content
SpringerLink
Log in

Multi-constraints in deep graph convolutional networks with initial residual

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Graph Convolutional Networks (GCN) can effectively extract rich information from non-structured data. However, in deep GCN models, the iterative propagation and updating of node information will lead to severe over-smoothing, which hampers the model’s performance. In our view, when the model suffers from over-smoothing, there will be little difference between the node features before and after updating. This paper proposes a more comprehensive smoothness metric according to nodes themselves, which considers both numerical and directional differences between nodes. Furthermore, (1) adding a similarity constraint between the initial features and the current layer features, which ensures the nodes’ representations avoid moving away from the initial features during the updating process; and (2) introducing a disparity constraint to the features of the nodes at each GCN layer to slow down the speed of node features becoming similar between before and after updating. We conduct extensive experiments on models with initial residual and achieve state-of-the-art results on several standard datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Lu M, Xiao Z, Li H, Zhang Y, Xiong NN (2022) Feature pyramid-based graph convolutional neural network for graph classification. J Syst Archit 128:102562

    Article  Google Scholar 

  2. Ghnn (2022) Graph harmonic neural networks for semi-supervised graph-level classification. Neural Netw 151:70–79

    Article  Google Scholar 

  3. Tang R, Jiang S, Chen X, Wang W, Wang W (2022) Network structural perturbation against interlayer link prediction. Knowl-Based Syst 250:109095

    Article  Google Scholar 

  4. Maurya SK, Liu X, Murata T (2022) Simplifying approach to node classification in graph neural networks. J Comput Sci 62:101695

    Article  Google Scholar 

  5. Abu-El-Haija S, Kapoor A, Perozzi B, Lee J (2019) N-GCN: multi-scale graph convolution for semi-supervised node classification. In: Amir Globerson and Ricardo Silva, editors, Proceedings of the thirty-fifth conference on uncertainty in artificial intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, vol 115 of Proceedings of machine learning research, pp 841–851. AUAI Press

  6. Cao M, Yang M, Qin C, Zhu X, Chen Y, Wang J, Liu T (2021) Using deepgcn to identify the autism spectrum disorder from multi-site resting-state data. Biomed Signal Process. Control 70:103015

    Article  Google Scholar 

  7. Chen D, Lin Y, Li W, Li P, Zhou J, Sun X (2020) Relieving the over-smoothing problem for graph neural networks from the topological view. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI Measuring New York, NY, USA, February 7-12, 2020, pp 3438–3445, AAAI Press, p 2020

  8. Chen H, Huang Z, Xu Y, Deng Z, Huang F, He P, Li Z (2022) Neighbor enhanced graph convolutional networks for node classification and recommendation. Knowl Based Syst 246:108594

    Article  Google Scholar 

  9. Chen H, Mao H, Li Y (2022) Elliptical convolution kernel: more real visual field. Neurocomputing 492:107–116

    Article  Google Scholar 

  10. Chen J, Wang X, Xu X (2021) Gc-lstm: graph convolution embedded lstm for dynamic network link prediction. Applied Intelligence, pp 1–16

  11. Chen M, Wei Z, Huang Z, Ding B, Li Y (2020) Simple and deep graph convolutional networks. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18, July 2020, Virtual event volume 119 of Proceedings of Machine Learning Research, pp 1725–1735, PMLR

  12. Chen YL, Hsiao CH, Wu CC (2022) An ensemble model for link prediction based on graph embedding. Decis Support Syst 157:113753

    Article  Google Scholar 

  13. Chong Y, Ding Y, Yan Q, Pan S (2020) Graph-based semi-supervised learning: a review. Neurocomputing 408:216–230

    Article  Google Scholar 

  14. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, volume 15 of JMLR Proceedings, pp 315–323, JMLR.org

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition, CVPR 2016 Las Vegas, NV, USA, June 27-30, 2016, pp 770–778, IEEE Computer Society

  16. Huang G, Liu Z, van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on computer vision and pattern recognition, CVPR 2017 Honolulu, HI, USA, July 21-26, 2017, pp 2261–2269, IEEE Computer Society

  17. Huang Z, Tang Y, Chen Y (2022) A graph neural network-based node classification model on class-imbalanced graph data. Knowl Based Syst 244:108538

    Article  Google Scholar 

  18. Kingma Diederik P, Adam JB (2015) A method for stochastic optimization. In: ICLR (Poster)

  19. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th International conference on learning representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference track proceedings. OpenReview.net

  20. Klicpera J, Bojchevski A, Günnemann S (2019) Predict then propagate: graph neural networks meet personalized pagerank. In: 7th International conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net

  21. Li B, Li Z, Yang Y (2021) Residual attention graph convolutional network for web services classification. Neurocomputing 440:45–57

    Article  Google Scholar 

  22. Li G, Müller M, Qian G, Perez ICD, Abualshour A, Thabet AK, Ghanem B (2021) Deepgcns: making gcns go as deep as cnns. IEEE Transactions on Pattern Analysis and Machine Intelligence

  23. Li K, Ye W (2022) Semi-supervised node classification via graph learning convolutional neural network. Applied Intelligence, pp 1–13

  24. Li MW, Xu DY, Geng J, Hong WC (2022) A ship motion forecasting approach based on empirical mode decomposition method hybrid deep learning network and quantum butterfly optimization algorithm. Nonlinear Dyn 107(3):2447–2467

    Article  Google Scholar 

  25. Li Q, Han Z, Wu XM (2018) Deeper insights into graph convolutional networks for semi-supervised learning. In: Sheila, A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp 3538–3545. AAAI Press

  26. Liu M, Gao H, Ji S (2020) Towards deeper graph neural networks. In: Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp 338–348. ACM

  27. Lu W, Zhan Y, Guan Z, Liu L, Yu B, Wei Z, Yang Y, Tao D (2021) Skipnode: on alleviating over-smoothing for deep graph convolutional networks. arXiv:2112.11628

  28. Ma T, Pan Q, Wang H, Shao W, Tian Y, Al-Nabhan N (2020) Graph classification algorithm based on graph structure embedding. Expert Syst. Appl. 161:113715

    Article  Google Scholar 

  29. Ma T, Wang H, Zhang L, Tian Y, Al-Nabhan N (2021) Graph classification based on structural features of significant nodes and spatial convolutional neural networks. Neurocomputing 423:639–650

    Article  Google Scholar 

  30. Nasiri E, Berahmand K, Rostami M, Dabiri M (2021) A novel link prediction algorithm for protein-protein interaction networks by attributed graph embedding. Comput Biol Medicine 137:104772

    Article  Google Scholar 

  31. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kȯpf A, Yang EZ, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: an imperative style, high-performance deep learning library. In: NeurIPS, pp 8024–8035

  32. Qi C, Zhang J, Jia H, Mao Q, Wang L, Song H (2021) Deep face clustering using residual graph convolutional network. Knowl Based Syst 211:106561

    Article  Google Scholar 

  33. Rong Y, Huang W, Xu T, Huang J (2020) Dropedge: towards deep graph convolutional networks on node classification. In: 8th International conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net

  34. Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in network data. AI Mag 29(3):93–106

    Google Scholar 

  35. Wang Z, Li W, Su H (2021) Hierarchical attention link prediction neural network. Knowl Based Syst 232:107431

    Article  Google Scholar 

  36. Wu Y, Song Y, Huang H, Ye F, Xie X, Jin H (2021) Enhancing graph neural networks via auxiliary training for semi-supervised node classification. Knowl Based Syst 220:106884

    Article  Google Scholar 

  37. Yang Z, Cohen W, Salakhudinov R (2016) Revisiting semi-supervised learning with graph embeddings. In: ICML, vol 48 of JMLR Workshop and conference proceedings, pp 40–48. JMLR.org

  38. Yin R, Li K, Zhang G, Lu J (2019) A deeper graph neural network for recommender systems. Knowl Based Syst, p 185

  39. Yuan H, Ji S (2020) Structpool: structured graph pooling via conditional random fields. In: 8th International conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net

  40. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2021) Residual dense network for image restoration. IEEE Trans Pattern Anal Mach Intell 43(7):2480–2495

    Article  Google Scholar 

  41. Zhao L, Akoglu L (2020) Pairnorm: tackling oversmoothing in gnns. In: 8th International conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net

  42. Zhou K, Huang X, Li Y, Zha D, Chen R, Hu X (2020) Towards deeper graph neural networks with differentiable group normalization. In: Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in neural information processing systems 33: Annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

Download references

Acknowledgements

This work was supported by the State Grid Corporation Headquarters Science and Technology Project under Grant 5700-202199539A-0-5-ZN.

Author information

Authors and Affiliations

  1. School of Control and Computer Engineering, North China Electric Power University, Beijing, 102206, China

    Hui Chen & Yuancheng Li

Authors
  1. Hui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuancheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuancheng Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Hyper-parameters details

Appendix A: Hyper-parameters details

Our parameters mainly consist of the balance coefficients of the similarity constraint and disparity constraint. Other parameters remain the same as the original papers, and we use grid search method to select suitable parameters. We let \({\mathscr{L}}_{cons}, {\mathscr{L}}_{disp} \in \) [5e-6, 1e-6, 5e-5, 1e-5, 5e-4, 1e-4, 5e-3, 1e-3, 5e-2, 1e-2, 0.1], μ ∈[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. And in Table 4, we summarise the training configurations to reproduce the results of Table 2.

Table 4 The hyper-parameters for Table 2
Full size table

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Li, Y. Multi-constraints in deep graph convolutional networks with initial residual. Appl Intell 53, 13608–13620 (2023). https://doi.org/10.1007/s10489-022-04222-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04222-8

Keywords

Search

Navigation