Skip to main content
SpringerLink
Log in

Graph representation learning via simple jumping knowledge networks

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Recent graph neural networks for graph representation learning depend on a neighborhood aggregation process. Several works focus on simplifying the neighborhood aggregation process and model structures. However, as the depth of the models increases, the simplified models will encounter oversmoothing, resulting in a decrease in model performance. Several works leverage sophisticated learnable neighborhood aggregation algorithms to learn more accurate graph representations. However, the high computational cost limits the depth of these models and the ability to tackle large graphs. In this paper, we propose simple jumping knowledge networks (SJK-Nets), which first leverage a simple no-learning method to complete the neighborhood aggregation process, and then utilize a jumping architecture to combine the different neighborhood ranges of each node to achieve a better structure-aware representation. Under such circumstances, first, we use a simple neighborhood aggregation algorithm to reduce computational complexity of the model. Then, we aggregate the features of high-order neighboring nodes to learn more informative node feature representations. Finally, by combining the above methods, the oversmoothing problem of the deep graph neural networks is alleviated. Our experimental evaluation demonstrates that SJK-Nets achieve or match state-of-the-art results in node classification tasks, text classification tasks, and community prediction tasks. Moreover, since SJK-Nets’ neighborhood aggregation is a no-learning process, SJK-Nets are successfully extended to node clustering tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th international conference on learning representations

  2. Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: 6th international conference on learning representations

  3. Zhang X, Liu H, Li Q, Wu XM (2019) Attributed graph clustering via adaptive graph convolution. In: Proceedings of the Twenty-Eighth international joint conference on artificial intelligence, pp 4327–4333

  4. Zhu H, Koniusz P (2021) Simple spectral graph convolution. In: 9th international conference on learning representations

  5. Roy I, De A, Chakrabarti S (2021) Adversarial permutation guided node representations for link prediction. In: Proceedings of the AAAI conference on artificial intelligence, vol 35, pp 9445–9453

  6. Zhang Q, Wang R, Yang J, Xue L (2021) Knowledge graph embedding by translating in time domain space for link prediction. Knowl-Based Syst 212. https://doi.org/10.1016/j.knosys.2020.106564

  7. Bruna J, Zaremba W, Szlam A, LeCun Y (2014) Spectral networks and locally connected networks on graphs. In: 2th international conference on learning representations

  8. Atwood J, Towsley D (2015) Deep convolutional networks on graph-structured data. arXiv:150605163

  9. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in neural information processing systems, pp 3844–3852

  10. Wu F, Zhang T (2019) Simplifying graph convolutional networks. In: Proceedings of the 36th international conference on machine learning, vol 97, pp 6861–6871

  11. Xu K, Li C, Tian Y, Sonobe T, Kawarabayashi K, Jegelka S (2018) Representation learning on graphs with jumping knowledge networks. In: Proceedings of the 35th international conference on machine learning

  12. Abu-El-Haija S, Perozzi B, Kapoor A, Alipourfard N, Lerman K, Harutyunyan H, Steeg GV, Galstyan A (2019) MixHop: higher-order graph convolutional architectures via sparsified neighborhood mixing. In: Proceedings of the 36th international conference on machine learning, pp 21–29

  13. Klicpera J, Bojchevski A, Günnemann S (2019a) Predict then Propagate: Graph neural networks meet personalized pagerank. In: 7th International conference on learning representations

  14. Klicpera J, Weißenberger S, Günnemann S (2019b) Diffusion improves graph learning

  15. Page L, Brin S, Motwani R, Winograd T (1999) The pagerank citation ranking: bringing order to the web. In: Proceedings of the WWW: the Web Conference

  16. Chien E, Peng J, Li P, Milenkovic O (2021) Adaptive Universal Generalized Pagerank Graph Neural Network. In: 9th international conference on learning representations

  17. Sun K, Zhu Z, Lin Z (2021) Adagcn: Adaboosting graph convolutional networks into deep models. In: 9th international conference on learning representations

  18. Li R, Wang S, Zhu F, Huang J (2018) Adaptive graph convolutional neural networks. In: Proceedings of the AAAI conference on artificial intelligence

  19. Yang F, Zhang H, Tao S (2021) Simplified multilayer graph convolutional networks with dropout. Applied Intelligence. https://doi.org/10.1007/s10489-021-02617-7

  20. NT H, Maehara T (2019) Revisiting Graph Neural Networks: All We Have is Low-Pass Filters. arXiv:190509550

  21. Duvenaud D, Maclaurin D, Aguilera-Iparraguirre J, Gomez-Bombarelli R, Hirzel T, Aspuru-Guzik A, Adams RP (2015) Convolutional Networks on graphs for learning molecular fingerprints. In: Advances in neural information processing systems, pp 2224–2232

  22. Atwood J, Towsley D (2016) Diffusion-convolutional neural networks. In: Advances in neural information processing systems, pp 1993–2001

  23. Niepert M, Ahmed M, Kutzkov K (2016) Learning convolutional neural networks for graphs. In: Proceedings of The 33rd international conference on machine learning, vol 48, pp 2014–2023

  24. Hamilton WL, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. In: Advances in neural information processing systems, pp 1024–1034

  25. Monti F, Boscaini D, Masci J, Rodola E, Svoboda J, Bronstein MM (2017) Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2017.576. IEEE, Honolulu, pp 5425–5434

  26. Gao H, Wang Z, Ji S (2018) large-Scale learnable graph convolutional networks. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. https://doi.org/10.1145/3219819.3219947. ACM, London, pp 1416–1424

  27. Chen J, Ma T, Xiao C (2018) FastGCN: Fast learning with graph convolutional networks via importance sampling. In: 6th international conference on learning representations

  28. Huang W, Zhang T, Rong Y, Huang J (2018) Adaptive sampling towards fast graph representation learning. In: Advances in neural information processing systems

  29. Thekumparampil KK, Wang C, Oh S, Li LJ (2018) Attention-based Graph Neural Network for Semi-supervised Learning. arXiv:180303735

  30. Lee J, Lee I, Kang J (2019) Self-attention graph pooling. In: Proceedings of the 36th international conference on machine learning

  31. Zhang K, Zhu Y, Wang J, Zhang J (2020) Adaptive structural fingerprints for graph attention networks. In: 8th international conference on learning representations

  32. Chiang WL, Liu X, Si S, Li Y, Bengio S, Hsieh CJ (2019) Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, Anchorage, pp 257–266. https://doi.org/10.1145/3292500.3330925

  33. Li G, Muller M, Thabet A, Ghanem B (2019) DeepGCNs: Can GCNs Go As Deep As CNNs?. In: 2019 IEEE/CVF International conference on Computer Vision (ICCV). IEEE, Korea, pp 9266–9275. https://doi.org/10.1109/ICCV.2019.00936

  34. Loukas A (2020) What graph neural networks cannot learn: depth vs width. In: 8th international conference on learning representations

  35. Oono K, Suzuki T (2020) Graph neural networks exponentially lose expressive power for node classification. In: 8th international conference on learning representations

  36. Zhou K, Huang X, Li Y, Zha D, Chen R, Hu X (2020) Towards deeper graph neural networks with differentiable group normalization. In: Advances in Neural Information Processing Systems, vol 33

  37. Rong Y, Huang W, Xu T, Huang J (2020) Dropedge: towards deep graph convolutional network on node classification. In: Proceedings of the 37th international conference on machine learning

  38. Chen M, Wei Z, Huang Z, Ding B, Li Y (2020) Simple and deep graph convolutional networks. In: Proceedings of the 37th international conference on machine learning, pp 1725–1735

  39. Li Q, Han Z, Wu XM (2018) Deeper insights into graph convolutional networks for semi-supervised learning. In: Proceedings of the AAAI conference on artificial intelligence, pp 3538–3545

  40. Zhou J, Cui G, Zhang Z, Yang C, Liu Z, Wang L, Li C, Sun M (2019) Graph neural networks: a review of methods and applications. In: 7th international conference on learning representations

  41. Chen L, Wu L, Hong R, Zhang K, Wang M (2020) Revisiting graph based collaborative filtering: a linear residual graph convolutional network approach. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 27–34. https://doi.org/10.1609/aaai.v34i01.5330

  42. Wu Z, Pan S, Chen F, Long G, Zhang C, Yu PS (2021) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24. https://doi.org/10.1109/TNNLS.2020.2978386

    Article  MathSciNet  Google Scholar 

  43. Veli,̌ cković P, Fedus W, Hamilton WL, Liò P, Bengio Y, Hjelm RD (2019) Deep Graph Infomax. In: 7th International Conference on Learning Representations

  44. Xu B, Shen H, Cao Q, Qiu Y, Cheng X (2019) Graph wavelet neural network. In: 7th international conference on learning representations

  45. Zhu H, Feng F, He X, Wang X, Li Y, Zheng K, Zhang Y (2020) Bilinear Graph Neural Network with Neighbor Interactions. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, Yokohama, Japan, vol 5, pp 1452–1458. https://doi.org/10.24963/ijcai.2020/202

  46. Gao H, Ji S (2019) Graph U-Nets. In: Proceedings of the 36th international conference on machine learning

  47. Bianchi FM, Grattarola D, Alippi C (2020) Spectral clustering with graph neural networks for graph pooling. In: Proceedings of the 37th international conference on machine learning

  48. Ying R, You J, Morris C, Ren X, Hamilton WL, Leskovec J (2018) Hierarchical Graph Representation Learning with Differentiable Pooling. In: Advances in Neural Information Processing Systems

  49. Cai D, Lam W (2020) Graph Transformer for graph-to-Sequence Learning. In: Proceedings of the AAAI Conference on Artificial Intelligence

  50. Xhonneux LPAC, Qu M, Tang J (2020) Continuous graph neural networks. In: Proceedings of the 37th international conference on machine learning, pp 10432–10411

  51. Avelar PHC, Tavares AR, Gori M, Lamb LC (2021) Discrete and Continuous Deep Residual Learning Over Graphs. In: Proceedings of the 13th international conference on agents and artificial intelligence, vol 2, pp 119–131. https://doi.org/10.5220/0010231501190131

  52. Chung FR, Graham FC (1997) Spectral Graph Theory. American Mathematical Soc

  53. Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in Network Data. AI Mag 29(3):93. https://doi.org/10.1609/aimag.v29i3.2157

    Google Scholar 

  54. Yang Z, Cohen WW, Salakhutdinov R (2016) Revisiting Semi-Supervised Learning with Graph Embeddings. In: Proceedings of the 33th International Conference on Machine Learning, pp 40–48

  55. Pei H, Wei B, Chang KCC, Lei Y, Yang B (2020) Geom-GCN: geometric graph convolutional networks. In: 8th international conference on learning representations

  56. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: 3th international conference on learning representations

  57. Xu K, Hu W, Leskovec J, Jegelka S (2019) How Powerful Are Graph Neural Networks?. In: 7th international conference on learning representations

  58. Li Q, Wu XM, Liu H, Zhang X, Guan Z (2019) Label Efficient Semi-Supervised Learning via Graph Filtering. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, USA, pp 9574–9583. https://doi.org/10.1109/CVPR.2019.00981

  59. Perozzi B, Al-Rfou R, Skiena S (2014) DeepWalk: Online Learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. USA, ACM Press, pp 701–710. https://doi.org/10.1145/2623330.2623732

  60. Cao S, Lu W, Xu Q (2016) Deep neural networks for learning graph representations. In: Proceedings of the AAAI conference on artificial intelligence, pp 1145–1152

  61. Kipf TN, Welling M (2016) Variational Graph Auto-Encoders. In: NIPS Workshop on Bayesian Deep Learning

  62. Wang C, Pan S, Long G, Zhu X, Jiang J (2017) MGAE: Marginalized Graph Autoencoder for Graph Clustering. In: Proceedings of the 2017 ACM Conference on Information and Knowledge Management, pp 889–898

  63. Pan S, Hu R, Long G, Jiang J, Yao L, Zhang C (2018) Adversarially Regularized Graph Autoencoder for Graph Embedding. In: Proceedings of the Twenty-Seven International Joint Conference on Artificial Intelligence, pp 2609–2615

  64. Liu DC, Nocedal J (1989) On the limited memory BFGS method for large scale optimization. Math Program 45(1-3):503–528. https://doi.org/10.1007/BF01589116

    Article  MathSciNet  Google Scholar 

  65. Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of the AAAI conference on artificial intelligence

Download references

Acknowledgements

This document is the results of the research project funded by the National Natural Science Foundation of China (Grant No. 61772386), the Open Fund of Key Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources (Grant No. KF-2020-05-014), and the Fundamental Research Funds for the Central Universities (Grant No. KJ02072021-0119).

Author information

Authors and Affiliations

  1. School of Computer Science, Wuhan University, Wuhan, China

    Fei Yang, Huyin Zhang & Shiming Tao

  2. Key Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources, Shenzhen, China

    Huyin Zhang

  3. School of Computer Science, Central China Normal University, Wuhan, China

    Sheng Hao

Authors
  1. Fei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huyin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shiming Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sheng Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huyin Zhang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, F., Zhang, H., Tao, S. et al. Graph representation learning via simple jumping knowledge networks. Appl Intell 52, 11324–11342 (2022). https://doi.org/10.1007/s10489-021-02889-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02889-z

Keywords

Search

Navigation