Skip to main content
SpringerLink
Log in

Graph Contrastive Learning with Constrained Graph Data Augmentation

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Studies on graph contrastive learning, which is an effective way of self-supervision, have achieved excellent experimental performance. Most existing methods generate two augmented views, and then perform feature learning on the two views through maximizing semantic consistency. Nevertheless, it is still challenging to generate optimal views to facilitate the graph construction that can reveal the essential association relations among nodes by graph contrastive learning. Considering that the extremely high mutual information between views is prone to have a negative effect on model training, a good choice is to add constraints to the graph data augmentation process. This paper proposes two constraint principles, low dissimilarity priority (LDP) and mutual exclusion (ME), to mitigate the mutual information between views and compress redundant parts of mutual information between views. LDP principle aims to reduce the mutual information between views at global scale, and ME principle works to reduce the mutual information at local scale. They are opposite and appropriate in different situations. Without loss of generality, the two proposed principles are applied to two well-performed graph contrastive methods, i.e. GraphCL and GCA, and experimental results on 20 public benchmark datasets show that the models with the aid of the two proposed constraint principles achieve higher recognition accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Algorithm 2
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. https://github.com/CRIPAC-DIG/GCA.

  2. https://github.com/Shen-Lab/GraphCL.

  3. https://github.com/xushaowu/LDP_ME/.

References

  1. Caron M, Misra I, Mairal J, Goyal P, Bojanowski P, Joulin A (2020) Unsupervised learning of visual features by contrasting cluster assignments. Adv Neural Inf Process Syst 33:9912–9924

    Google Scholar 

  2. He K, Fan H, Wu Y, Xie S, Girshick R (2020) Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 9729–9738

  3. Tian Y, Krishnan D, Isola P (2020) Contrastive multiview coding. In: European conference on computer vision. pp 776–794. Springer

  4. Qian R, Meng T, Gong B, Yang M-H, Wang H, Belongie S, Cui Y (2021) Spatiotemporal contrastive video representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 6964–6974

  5. Chi Z, Dong L, Wei F, Yang N, Singhal S, Wang W, Song X, Mao X-L, Huang H, Zhou M (2021) Infoxlm: an information-theoretic framework for cross-lingual language model pre-training. In: Proceedings of the conference of the North American chapter of the association for computational linguistics: human language technologies. pp 3576–3588

  6. Fang H, Wang S, Zhou M, Ding J, Xie P (2020) Cert: contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766

  7. Giorgi J, Nitski O, Wang B, Bader G (2021) Declutr: deep contrastive learning for unsupervised textual representations. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (Volume 1: Long Papers). pp 879–895

  8. Wang C, Liu Z (2021) Learning graph representation by aggregating subgraphs via mutual information maximization. arXiv preprint arXiv:2103.13125

  9. Jin M, Zheng Y, Li Y.-F, Gong C, Zhou C, Pan S (2021) Multi-scale contrastive siamese networks for self-supervised graph representation learning. In: International joint conference on artificial intelligence 2021, pp 1477–1483. Association for the Advancement of Artificial Intelligence (AAAI)

  10. Li S, Zhou J, Xu T, Dou D, Xiong H (2022) Geomgcl: Geometric graph contrastive learning for molecular property prediction. Proc AAAI Conf Artif Intell 36:4541–4549

    Google Scholar 

  11. Chen M, Huang C, Xia L, Wei W, Xu Y, Luo R (2023) Heterogeneous graph contrastive learning for recommendation. In: Proceedings of the sixteenth ACM international conference on web search and data mining, pp 544–552

  12. Linsker R (1988) Self-organization in a perceptual network. Computer 21(3):105–117

    Article  Google Scholar 

  13. Xiao T, Wang X, Efros AA, Darrell T (2021) What should not be contrastive in contrastive learning. In: International conference on learning representations

  14. Chen B, Zhang J, Zhang X, Dong Y, Song J, Zhang P, Xu K, Kharlamov E, Tang J (2022) Gccad: Graph contrastive learning for anomaly detection. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2022.3200459

    Article  Google Scholar 

  15. Shi S, Xie P, Luo X, Qiao K, Wang L, Chen J, Yan B (2022) Adaptive multi-layer contrastive graph neural networks. Neural Process Lett. 1–20

  16. Sun Q, Li J, Peng H, Wu J, Ning Y, Yu P.S, He L (2021) Sugar: subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism. In: Proceedings of the web conference 2021. pp 2081–2091

  17. Zhao H, Yang X, Wang Z, Yang E, Deng C (2021) Graph debiased contrastive learning with joint representation clustering. In: IJCAI international joint conference on artificial intelligence. pp 3434–4440

  18. Shuai J, Zhang K, Wu L, Sun P, Hong R, Wang M, Li Y (2022) A review-aware graph contrastive learning framework for recommendation. In: Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. pp 1283–1293

  19. Zhu Y, Xu Y, Yu F, Liu Q, Wu S, Wang L (2021) Graph contrastive learning with adaptive augmentation. In: Proceedings of the web conference 2021. pp 2069–2080

  20. Wu M, Zhuang C, Mosse M, Yamins D, Goodman N (2020) On mutual information in contrastive learning for visual representations. arXiv preprint arXiv:2005.13149

  21. Tian Y, Sun C, Poole B, Krishnan D, Schmid C, Isola P (2020) What makes for good views for contrastive learning? Adv Neural Inf Process Syst 33:6827–6839

    Google Scholar 

  22. Xu D, Cheng W, Luo D, Chen H, Zhang X (2021) Infogcl: Information-aware graph contrastive learning. Adv Neural Inf Process Syst 36:30414–30425

    Google Scholar 

  23. You Y, Chen T, Sui Y, Chen T, Wang Z, Shen Y (2020) Graph contrastive learning with augmentations. Adv Neural Inf Process Syst 33:5812–5823

    Google Scholar 

  24. Zheng Y, Pan S, Lee V, Zheng Y, Yu PS (2022) Rethinking and scaling up graph contrastive learning: an extremely efficient approach with group discrimination. Adv Neural Inf Process Syst 35:10809–10820

    Google Scholar 

  25. Yu W, Wan S, Li G, Yang J, Gong C (2023) Hyperspectral image classification with contrastive graph convolutional network. IEEE Trans Geosci Remote Sens 61:1–15

    Google Scholar 

  26. Tsai Y-H, Wu Y, Salakhutdinov R, Morency L-P (2021) Self-supervised learning from a multi-view perspective. In: International conference on learning representations

  27. Wan S, Pan S, Yang J, Gong C (2021) Contrastive and generative graph convolutional networks for graph-based semi-supervised learning. Proc AAAI Conf Artif Intell 35:10049–10057

    Google Scholar 

  28. Yang Y, Huang C, Xia L, Li C (2022) Knowledge graph contrastive learning for recommendation. In: Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. pp 1434–1443

  29. Wu L, Lin H, Tan C, Gao Z, Li S.Z (2021) Self-supervised learning on graphs: contrastive, generative, or predictive. IEEE Trans Knowl Data Eng

  30. Poole B, Ozair S, Van Den Oord A, Alemi A, Tucker G (2019) On variational bounds of mutual information. In: International conference on machine learning. pp 5171–5180. PMLR

  31. Oord AVD, Li Y, Vinyals O (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

  32. Grover A, Leskovec J (2016) node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp 855–864

  33. Hamilton W.L, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. In: Advances in neural information processing systems. pp 1025–1035

  34. Kipf TN, Welling M (2016) Variational graph auto-encoders. Advances in neural information processing systems

  35. Perozzi B, Al-Rfou R, Skiena S (2014) Deepwalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining. pp 701–710

  36. Qiu J, Dong Y, Ma H, Li J, Wang K, Tang J (2018) Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In: Proceedings of the eleventh ACM international conference on web search and data mining. pp 459–467

  37. Veličković P, Fedus W, Hamilton WL, Liò P, Bengio Y, Hjelm RD (2019) Deep graph infomax. In: International Conference on Learning Representations

  38. Qiu J, Chen Q, Dong Y, Zhang J, Yang H, Ding M, Wang K, Tang J (2020) Gcc: graph contrastive coding for graph neural network pre-training. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery and data mining. pp 1150–1160

  39. Sun F-Y, Hoffman J, Verma V, Tang J (2020) Infograph: unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In: International conference on learning representations

  40. Hassani K, Khasahmadi AH (2020) Contrastive multi-view representation learning on graphs. In: International conference on machine learning. pp 4116–4126 . PMLR

  41. Wan S, Zhan Y, Liu L, Yu B, Pan S, Gong C (2021) Contrastive graph poisson networks: semi-supervised learning with extremely limited labels. Adv Neural Inf Process Syst 34:6316–6327

    Google Scholar 

  42. Ribeiro LF, Saverese PH, Figueiredo DR (2017) struc2vec: learning node representations from structural identity. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. pp 385–394

  43. Freeman LC (1978) Centrality in social networks conceptual clarification. Soc Netw 1(3):215–239

    Article  Google Scholar 

  44. Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp 1597–1607 . PMLR

  45. Morris C, Kriege N.M, Bause F, Kersting K, Mutzel P, Neumann M (2020) Tudataset: a collection of benchmark datasets for learning with graphs. In: International conference on machine learning

  46. Mernyei P, Cangea C (2020) Wiki-cs: a wikipedia-based benchmark for graph neural networks. In: International conference on machine learning

  47. Shchur O, Mumme M, Bojchevski A, Günnemann S (2018) Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868

  48. Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in network data. AI Mag 29(3):93–93

    Google Scholar 

  49. Pan S, Wu J, Zhu X, Zhang C, Wang Y (2016) Tri-party deep network representation. In: IJCAI international joint conference on artificial intelligence. pp 1895–1901

  50. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp 249–256

  51. Diederik KP, Jimmy B (2015) A method for stochastic optimization. In: International conference on learning representations

  52. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations

  53. Veličković P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y (2018) Graph attention networks. In: International conference on learning representations

  54. van der Maaten L, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9:2579–2605

    MATH  Google Scholar 

Download references

Acknowledgements

This work is supported by Beijing Natural Science Foundation (No.4202004). Consulting Project for Major Strategic Decision making for Serving Capital in 2022 from Beijing University of Technology.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Shaowu Xu, Luo Wang & Xibin Jia

Authors
  1. Shaowu Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xibin Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SX wrote the main manuscript text. All authors reviewed the manuscript.

Corresponding author

Correspondence to Xibin Jia.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, S., Wang, L. & Jia, X. Graph Contrastive Learning with Constrained Graph Data Augmentation. Neural Process Lett 55, 10705–10726 (2023). https://doi.org/10.1007/s11063-023-11346-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11346-6

Keywords

Search

Navigation