Skip to main content

Advertisement

Log in

Dirichlet stochastic weights averaging for graph neural networks

  • Published:
Applied Intelligence Aims and scope Submit manuscript

A Correction to this article was published on 08 February 2025

A Correction to this article was published on 06 December 2024

This article has been updated

Abstract

The popularity of Graph Neural Networks (GNNs) has grown significantly because GNNs handle relational datasets such as social networks and citation networks. However, the usual relational dataset is sparse, and GNNs are easy to overfit to the dataset. To alleviate the overfitting problems, model ensemble methods are widely studied and adopted. However, model ensemble methods for GNNs are not well explored. In this study, we propose simple but effective model ensemble methods for GNNs. This is the first study that adopts stochastic weights averaging (SWA) for GNNs. Furthermore, we propose a new model ensemble method, Dirichlet stochastic weighs averaging (DSWA). DSWA adopts the running averages of the trained weights with random proportions sampled by Dirichlet distributions. DSWA provides the diverse model and its ensembles on inference time without the training time increases. We validate our models on the Cora, the Citeseer, and Pubmed datasets under usual settings and few-shot learning settings. We observe that the performance of current GNNs deteriorates when the number of specified data is limited. DSWA improves the performance of few-shot node classification tasks as well as the general node classification tasks.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets analyzed during the current study are available in the pytorch_geometric repository, https://github.com/pyg-team/pytorch_geometric

Change history

Notes

  1. https://github.com/pyg-team/pytorch_geometric

References

  1. Chakraborty M, Byshkin M, Crestani F (2020) Patent citation network analysis: a perspective from descriptive statistics and ergms. PLoS ONE 15(12):0241797

    Article  Google Scholar 

  2. Ebesu T, Fang Y (2017) Neural citation network for context-aware citation recommendation. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. pp 1093–1096

  3. Laghari AA, Laghari MA (2021) Quality of experience assessment of calling services in social network. ICT Express. 7(2):158–161

    Article  MATH  Google Scholar 

  4. Nilashi M, Abumalloh RA, Almulihi A, Alrizq M, Alghamdi A, Ismail MY, Bashar A, Zogaan WA, Asadi S (2023) big social data analysis for impact of food quality on travelers’ satisfaction in eco-friendly hotels. ICT Express. 9(2):182–188

    Article  Google Scholar 

  5. Xu H, Jiang C, Liang X, Li Z (2019) Spatial-aware graph relation network for large-scale object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 9298–9307

  6. Wang Y, Kitani K, Weng X (2021) Joint object detection and multi-object tracking with graph neural networks. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp 13708–13715

  7. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv:1609.02907

  8. Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: International conference on learning representations

  9. Bianchi FM, Grattarola D, Livi L, Alippi C (2021) Graph neural networks with convolutional arma filters. IEEE Trans Pattern Anal Mach Intell 44(7):3496–3507

    Google Scholar 

  10. Wu F, Souza A, Zhang T, Fifty C, Yu T, Weinberger K (2019) Simplifying graph convolutional networks. In: International conference on machine learning. PMLR, p 6861–6871

  11. Chen M, Wei Z, Huang Z, Ding B, Li Y (2020) Simple and deep graph convolutional networks. In: International conference on machine learning. PMLR, pp 1725–1735

  12. Dvornik N, Schmid C, Mairal J (2019) Diversity with cooperation: Ensemble methods for few-shot classification. In: Proceedings of the IEEE/CVF international conference on computer vision. pp 3723–3731

  13. Jiang W, Zhang Y, Kwok JT (2021) Seen: few-shot classification with self-ensemble. In: 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, p 1–8

  14. Izmailov P, Wilson A, Podoprikhin D, Vetrov D, Garipov T (2018) Averaging weights leads to wider optima and better generalization. In: 34th Conference on uncertainty in artificial intelligence 2018, UAI 2018. p 876–885

  15. Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE (2017) Neural message passing for quantum chemistry. In: International conference on machine learning. PMLR, p 1263–1272

  16. Wilson AG, Izmailov P (2020) Bayesian deep learning and a probabilistic perspective of generalization. Adv Neural Inf Process Syst 33:4697–4708. arXiv:2002.08791

  17. Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight uncertainty in neural network. In: International conference on machine learning. PMLR, p 1613–1622

  18. Gal Y, Ghahramani Z (2016) Dropout as a bayesian approximation: representing model uncertainty in deep learning. In: International conference on machine learning. PMLR, pp 1050–1059

  19. Daxberger E, Kristiadi A, Immer A, Eschenhagen R, Bauer M, Hennig P (2021) Laplace redux-effortless bayesian deep learning. Adv Neural Inf Process Syst 34:20089–20103

    Google Scholar 

  20. Zhang Y, Pal S, Coates M, Ustebay D (2019) Bayesian graph convolutional neural networks for semi-supervised classification. Proc AAAI Conf Artif Intell 33:5829–5836

    MATH  Google Scholar 

  21. Hasanzadeh A, Hajiramezanali E, Boluki S, Zhou M, Duffield N, Narayanan K, Qian X (2020) Bayesian graph neural networks with adaptive connection sampling. In: International conference on machine learning. PMLR, pp 4094–4104

  22. Papp PA, Martinkus K, Faber L, Wattenhofer R (2021) dropgnn: random dropouts increase the expressiveness of graph neural networks. Adv Neural Inf Process Syst 34:21997–22009

    MATH  Google Scholar 

  23. Rong Y, Huang W, Xu T, Huang J (2019) dropedge: towards deep graph convolutional networks on node classification. In: International conference on learning representations

  24. Wang Y, Karaletsos T (2021) Stochastic aggregation in graph neural networks. arXiv:2102.12648

  25. Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in network data. AI Mag 29(3):93–93

    Google Scholar 

  26. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  27. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. Adv Neural Inf Process Syst 29:3844–3852

    MATH  Google Scholar 

  28. Klicpera J, Bojchevski A, Günnemann S (2018) Predict then propagate: graph neural networks meet personalized pagerank. In: International conference on learning representations

  29. Fey M, Lenssen JE (2019) Fast graph representation learning with pytorch geometric. arXiv:1903.02428

  30. Maaten L, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(11)

Download references

Acknowledgements

This work was supported by the Basic Study and Interdisciplinary R&D Foundation Fund of the University of Seoul (2021).

Funding

The authors declare that there is no funding issue to disclose.

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Rakwoo Chang or Kyungwoo Song.

Ethics declarations

Conflicts of Interest

The authors declare that there is no conflict of interest or competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The incorrect graphical abstract was captured.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, M., Chang, R. & Song, K. Dirichlet stochastic weights averaging for graph neural networks. Appl Intell 54, 10516–10524 (2024). https://doi.org/10.1007/s10489-024-05708-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05708-3

Keywords

Navigation