Abstract
We present InfoMotif, a new semi-supervised, motif-regularized, learning framework over graphs. We overcome two key limitations of message passing in popular graph neural networks (GNNs): localization (a k-layer GNN cannot utilize features outside the k-hop neighborhood of the labeled training nodes) and over-smoothed (structurally indistinguishable) representations. We formulate attributed structural roles of nodes based on their occurrence in different network motifs, independent of network proximity. Network motifs are higher-order structures indicating connectivity patterns between nodes and are crucial to the organization of complex networks. Two nodes share attributed structural roles if they participate in topologically similar motif instances over covarying sets of attributes. InfoMotif achieves architecture-agnostic regularization of arbitrary GNNs through novel self-supervised learning objectives based on mutual information maximization. Our training curriculum dynamically prioritizes multiple motifs in the learning process without relying on distributional assumptions in the underlying graph or the learning task. We integrate three state-of-the-art GNNs in our framework, to show notable performance gains (3–10% accuracy) across nine diverse real-world datasets spanning homogeneous and heterogeneous networks. Notably, we see stronger gains for nodes with sparse training labels and diverse attributes in local neighborhood structures.
Similar content being viewed by others
Notes
The terms network motif, graphlet, and induced subgraph are used interchangeably in graph mining literature.
References
Albert R, Barabási AL (2001) Statistical mechanics of complex networks. CoRR, cond-mat/0106096
Albert R, Albert-László B (2002) Statistical mechanics of complex networks. Rev Mod Phys 74(1):47
Bachman P, Hjelm RD, Buchwalter W (2019) Learning representations by maximizing mutual information across views. In: Wallach HM, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox EB, Garnett R (eds) Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, NeurIPS 2019, December 8–14, 2019, Vancouver, BC, Canada, pp 15509–15519
Belghazi MI, Baratin A, Rajeswar S, Ozair S, Bengio Y, Hjelm RD, Courville AC (2018) Mutual information neural estimation. In: Dy JC, Krause A (eds) Proceedings of the 35th international conference on machine learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, volume 80 of Proceedings of Machine Learning Research. PMLR, pp 530–539
Beyer L, Zhai X, Oliver A, Kolesnikov A (2019) S4L: self-supervised semi-supervised learning. In: 2019 IEEE/CVF international conference on computer vision, ICCV 2019, Seoul, Korea (South), October 27–November 2, 2019, IEEE, pp 1476–1485
Bruna J, Zaremba W, Szlam A, Yann L (2014) Spectral networks and locally connected networks on graphs. In: Bengio Y, LeCun Y (eds) 2nd international conference on learning representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, Conference Track Proceedings
Cui Y, Jia M, Lin T-Y, Song Y, Belongie SJ (2019) Class-balanced loss based on effective number of samples. In: IEEE conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16–20, 2019. Computer Vision Foundation/IEEE, pp 9268–9277
Dareddy MR, Das M, Yang H (2019) motif2vec: Motif aware node representation learning for heterogeneous networks. In: Baru C, Huan J, Khan L, Hu X, Ak R, Tian Y, Barga RS, Zaniolo C, Lee K, Ye YF (eds) 2019 IEEE international conference on big data (IEEE BigData), Los Angeles, CA, USA, December 9–12, 2019. IEEE, pp 1052–1059
Dauphin YN, Fan A, Auli M, Grangier D (2017) Language modeling with gated convolutional networks. In: Precup D, Teh YW (eds) Proceedings of the 34th international conference on machine learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, volume 70 of Proceedings of Machine Learning Research. PMLR, pp 933–941
Dong Y, Chawla NV, Swami A (2017) metapath2vec: scalable representation learning for heterogeneous networks. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, Halifax, NS, Canada, August 13–17, 2017. ACM, pp 135–144
Donnat C, Zitnik M, Hallac D, Leskovec J (2018) Learning structural node embeddings via diffusion wavelets. In: Guo Y, Farooq F (eds) Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2018, London, UK, August 19–23, 2018. ACM, pp 1320–1329
Fu T-Y, Lee W-C, Lei Z (2017) Hin2vec: explore meta-paths in heterogeneous information networks for representation learning. In: Lim F-P, Winslett M, Sanderson M, Fu AW-C, Sun J, Shane Culpepper J, Lo E, Ho JC, Donato D, Agrawal R, Zheng Y, Castillo C, Sun A, Tseng VC, Li C (eds) Proceedings of the 2017 ACM on conference on information and knowledge management, CIKM 2017, Singapore, November 06–10, 2017. ACM, pp 1797–1806
Fu X, Zhang J, Meng Z, King I (2020) MAGNN: metapath aggregated graph neural network for heterogeneous graph embedding. In: Huang Y, King I, Liu T-Y , van Steen M (eds) WWW ’20: the web conference 2020, Taipei, Taiwan, April 20–24, 2020. ACM / IW3C2, pp 2331–2341
Grover A, Leskovec J (2016) node2vec: Scalable feature learning for networks. In: Krishnapuram B, Shah M, Smola AJ, Aggarwal CC, Shen D, Rastogi R (eds) Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, CA, USA, August 13–17, 2016. ACM, pp 855–864
Hamilton WL, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp 1024–1034
Harper FM, Konstan JA (2016) The movielens datasets: history and context. ACM Trans Interact Intell Syst 5(4):19:1-19:19
Henderson K, Gallagher B, Eliassi-Rad T, Tong H, Basu S, Akoglu L, Koutra D, Faloutsos C, Li L (2012) Rolx: structural role extraction & mining in large graphs. In: Yang Q, Agarwal D, Pei J (eds) The 18th ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’12, Beijing, China, August 12–16, 2012. ACM, pp 1231–1239
Hjelm RD, Fedorov A, Lavoie-Marchildon S, Grewal K, Bachman P, Trischler A, Bengio Y (2019) Learning deep representations by mutual information estimation and maximization. In: 7th international conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. OpenReview.net
Hu Z, Dong Y, Wang K, Sun Y (2020) Heterogeneous graph transformer. In: Huang Y, King I, Liu T-Y, van Steen M (eds) WWW ’20: the web conference 2020, Taipei, Taiwan, April 20–24, 2020. ACM/IW3C2, pp 2704–2710
Jin W, Derr T, Liu H, Wang Y, Wang S, Liu Z, Tang J (2020) Self-supervised learning on graphs: deep insights and new direction. CoRR, arXiv:2006.10141
Jin W, Derr T, Wang Y, Ma Y, Liu Z, Tang J (2021) Node similarity preserving graph convolutional networks. In: Lewin-Eytan L, Carmel D, Yom-Tov E, Agichtein E, Gabrilovich E (eds) WSDM ’21, The fourteenth ACM international conference on web search and data mining, virtual event, Israel, March 8–12, 2021. ACM, pp 148–156
Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net
Krishnan A, Cheruvu H, Cheng T, Sundaram H (2019) A modular adversarial approach to social recommendation. In: Zhu W, Tao D, Cheng X, Cui P, Rundensteiner EA, Carmel D, He Q, Yu JX (eds) Proceedings of the 28th ACM international conference on information and knowledge management, CIKM 2019, Beijing, China, November 3–7, 2019. ACM, pp 1753–1762
Lee JB, Ryan RA, Kong X, Kim S, Koh E, Rao A (2019) Graph convolutional networks with motif-based attention. In: Zhu W, Tao D, Cheng X, Cui P, Rundensteiner EA, Carmel D, He Q, Yu JX (eds) Proceedings of the 28th ACM international conference on information and knowledge management, CIKM 2019, Beijing, China, November 3–7, 2019. ACM, pp 499–508
Li Q, Han Z, Wu X-M (2018) Deeper insights into graph convolutional networks for semi-supervised learning. In: McIlraith SA, Weinberger KQ (eds) Proceedings of the thirty-second AAAI conference on artificial intelligence, (AAAI-18), the 30th innovative applications of artificial intelligence (IAAI-18), and the 8th AAAI symposium on educational advances in artificial intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2–7, 2018. AAAI Press, pp 3538–3545
Liu Y, Pan S, Jin M, Zhou C, Xia F, Yu PS (2021) Graph self-supervised learning: asurvey. CoRR, arXiv:2103.00111
McCallum A, Nigam K, Rennie J, Seymore K (2000) Automating the construction of internet portals with machine learning. Inf Retr 3(2):127–163
Milo R, Shen-Orr S, Itzkovitz S, Kashtan N, Chklovskii D, Alon U (2002) Network motifs: simple building blocks of complex networks. Science 298(5594):824–827
Narang K, Yang C, Krishnan A, Wang J, Sundaram H, Sutter C (2019) An induced multi-relational framework for answer selection in community question answer platforms. CoRR, arXiv:1911.06957
Paranjape A, Benson AR, Leskovec J (2017) Motifs in temporal networks. In: de Rijke M, Shokouhi M, Tomkins A, Zhang M (eds) Proceedings of the Tenth ACM international conference on web search and data mining, WSDM 2017, Cambridge, United Kingdom, February 6–10, 2017. ACM, pp 601–610
Peel L, Delvenne J-C, Lambiotte R (2017) Multiscale mixing patterns in networks. CoRR, arXiv:1708.01236
Peng Z, Dong Y, Luo M, Wu X-M, Zheng Q (2020) Self-supervised graph representation learning via global context prediction. CoRR, arXiv:2003.01604
Peng Z, Huang W, Luo M, Zheng Q, Rong Y, Xu T, Huang J (2020) Graph representation learning via graphical mutual information maximization. In: Huang Y, King I, Liu T-Y, van Steen M (eds) WWW ’20: The Web conference 2020, Taipei, Taiwan, April 20–24, 2020. ACM/IW3C2, pp 259–270
Qiu J, Chen Q, Dong Y, Zhang J, Yang H, Ding M, Wang K, Tang J (2020) GCC: graph contrastive coding for graph neural network pre-training. In: Gupta R, Liu Y, Tang J, Aditya Prakash B (eds) KDD ’20: The 26th ACM SIGKDD conference on knowledge discovery and data mining, virtual event, CA, USA, August 23–27, 2020. ACM, pp 1150–1160
Ren M, Zeng W, Yang B, Urtasun R (2018) Learning to reweight examples for robust deep learning. In: Dy JC, Krause A (eds) Proceedings of the 35th international conference on machine learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, volume 80 of Proceedings of Machine Learning Research. PMLR, pp 4331–4340
Ribeiro LFR, Saverese PHP, Figueiredo DR (2017) struc2vec: learning node representations from structural identity. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, Halifax, NS, Canada, August 13–17, 2017. ACM, pp 385–394
Ribeiro P, Paredes P, Silva MEP, Aparício D, Fernando SMA (2021) A survey on subgraph counting: concepts, algorithms, and applications to network motifs and graphlets. ACM Comput Surv 54(2):28:1-28:36
Rossi RA, Ahmed NK (2015) Role discovery in networks. IEEE Trans Knowl Data Eng 27(4):1112–1131
Rossi RA, Ahmed NK, Carranza AG, Arbour D, Rao A, Kim S, Koh E (2019) Heterogeneous network motifs. CoRR, arXiv:1901.10026
Rossi RA, Ahmed NK, Koh E, Kim S, Rao A, Abbasi-Yadkori Y (2020) A structural graph representation learning framework. In: Caverlee J, Hu XB, Lalmas M, Wang W (eds) WSDM ’20: The Thirteenth ACM international conference on web search and data mining, Houston, TX, USA, February 3–7, 2020. ACM, pp 483–491
Rossi RA, Jin D, Kim S, Ahmed NK, Koutra D, Boaz Lee J (2019) From community to role-based graph embeddings. CoRR, arXiv:1908.08572
Rossi RA, Rong Z, Ahmed NK (2019) Estimation of graphlet counts in massive networks. IEEE Trans Neural Netw Learn Syst 30(1):44–57
Sankar A (2022) Sparsity-aware neural user behavior modeling in online interaction platforms. Preprint arXiv:2202.13491
Sankar A, Liu Y, Yu J, Shah N (2021) Graph neural networks for friend ranking in large-scale social platforms. In: Leskovec J, Grobelnik M, Najork M, Tang J, Zia L (eds) WWW ’21: The Web Conference 2021, Virtual Event/Ljubljana, Slovenia, April 19–23, 2021. ACM/IW3C2, pp 2535–2546
Sankar A, Wang J, Krishnan A, Sundaram H (2021) Protocf: Prototypical collaborative filtering for few-shot recommendation. In: Jesús Corona Pampín H, Larson MA, Willemsen MC, Konstan JA, McAuley JJ, Garcia-Gathright J, Huurnink B, Oldridge E (eds) RecSys ’21: Fifteenth ACM conference on recommender systems, Amsterdam, The Netherlands, 27 September 2021–1 October 2021. ACM, pp 166–175
Sankar A, Wu Y, Gou L, Zhang W, Yang H (2020) Dysat: Deep neural representation learning on dynamic graphs via self-attention networks. In: Caverlee J, Hu XB, Lalmas M, Wang W (eds) WSDM ’20: the thirteenth ACM international conference on web search and data mining, Houston, TX, USA, February 3–7, 2020. ACM, pp 519–527
Sankar A, Wu Y, Wu Y, Zhang W, Yang H, Sundaram H (2020) Groupim: a mutual information maximization framework for neural group recommendation. In: Huang J, Chang Y, Cheng X, Kamps J, Murdock V, Wen J-R, Liu Y (eds) Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25–30, 2020. ACM, pp 1279–1288
Sankar A, Zhang X, Chen-Chuan Chang K (2017) Motif-based convolutional neural network on graphs. CoRR, arXiv:1711.05697
Sankar A, Zhang X, Chen-Chuan Chang K (2019) Meta-gnn: metagraph neural network for semi-supervised learning in attributed heterogeneous information networks. In: Spezzano F, Chen W, Xiao X (eds) ASONAM ’19: international conference on advances in social networks analysis and mining, Vancouver, British Columbia, Canada, 27–30 August, 2019. ACM, pp 137–144
Sankar A, Zhang X, Krishnan A, Han J (2020) Inf-vae: a variational autoencoder framework to integrate homophily and influence in diffusion prediction. In: Caverlee J, Hu XB, Lalmas M, Wang W (eds) WSDM ’20: the thirteenth ACM international conference on web search and data mining, Houston, TX, USA, February 3–7, 2020. ACM, pp 510–518
Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in network data. AI Mag 29(3):93–106
Shchur O, Mumme M, Bojchevski A, Günnemann S (2018) Pitfalls of graph neural network evaluation. CoRR, arXiv:1811.05868
Shi Y, Gui H, Zhu Q, Lance KM, Han J (2018) Aspem: Embedding learning by aspects in heterogeneous information networks. In: Ester M, Pedreschi D (eds) Proceedings of the 2018 SIAM international conference on data mining, SDM 2018, May 3–5, 2018, San Diego Marriott Mission Valley, San Diego, CA, USA. SIAM, pp 144–152
Sun Y, Han J, Yan X, Yu PS, Wu T (2011) Pathsim: meta path-based top-k similarity search in heterogeneous information networks. Proc VLDB Endow 4(11):992–1003
Tu K, Cui P, Wang X, Philip YS, Zhu W (2018) Deep recursive network embedding with regular equivalence. In: Guo Y, Farooq F (eds) Proceeding of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2018, London, UK, August 19–23, 2018
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Aidan GN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp 5998–6008
Velickovic P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference Track Proceedings. OpenReview.net
Velickovic P, Fedus W, Hamilton WL, Liò P, Bengio Y, Hjelm DR (2019) Deep graph infomax. In: 7th international conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. OpenReview.net
Vinyals O, Bengio S, Kudlur M (2016) Order matters: sequence to sequence for sets. In: Bengio Y, LeCun Y (eds) 4th international conference on learning representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings
Wang X, Ji H, Shi C, Wang B, Ye Y, Cui P, Philip YS (2019) Heterogeneous graph attention network. In: Liu L, White RW, Mantrach A, Silvestri F, McAuley JJ, Baeza-Yates R, Zia L (eds) The world wide web conference, WWW 2019, San Francisco, CA, USA, May 13–17, 2019. ACM, pp 2022–2032
Wu J, He J, Xu J (2019) Demo-net: segree-specific graph neural networks for node and graph classification. In: Teredesai A, Kumar V, Li Y, Rosales R, Terzi E, Karypis G (eds) Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2019, Anchorage, AK, USA, August 4–8, 2019. ACM, pp 406–415
Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2021) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24
Xie Y, Xu Z, Wang Z, Ji S (2021) Self-supervised learning of graph neural networks: a unified review. CoRR, arXiv:2102.10757
Xu K, Li C, Tian Y, Sonobe T, Kawarabayashi K-I, Jegelka S (2018) Representation learning on graphs with jumping knowledge networks. In: Dy JC, Krause A (eds) Proceedings of the 35th international conference on machine learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, volume 80 of Proceedings of Machine Learning Research. PMLR, pp 5449–5458
You J, Ying R, Leskovec J (2019) Position-aware graph neural networks. In: Chaudhuri K, Salakhutdinov R(eds) Proceedings of the 36th international conference on machine learning, ICML 2019, 9–15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research. PMLR, pp 7134–7143
You Y, Chen T, Sui Y, Chen T, Wang Z, Shen Y (2020) Graph contrastive learning with augmentations. In: Larochelle H, Ranzato M, Hadsell R, Balcan M-F, Lin H-T (eds) Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, NeurIPS 2020, December 6–12, 2020, virtual
Zhang C, Song D, Huang C, Swami A, Nitesh CV (2019) Heterogeneous graph neural network. In: Teredesai A, Kumar V, Li Y, Rosales R, Terzi E, Karypis G (eds) Proceedings of the 25th ACM SIGKDD International conference on knowledge discovery & data mining, KDD 2019, Anchorage, AK, USA, August 4–8, 2019. ACM. pp 793–803
Zhang D, Yin J, Zhu X, Zhang C (2018) Metagraph2vec: Complex semantic path augmented heterogeneous network embedding. In: Phung DQ, Tseng VS, Webb GI, Ho B, Ganji M, Rashidi L (eds) Advances in knowledge discovery and data mining—22nd pacific-asia conference, PAKDD 2018, Melbourne, VIC, Australia, June 3–6, 2018, Proceedings, Part II, volume 10938 of Lecture Notes in Computer Science. Springer, pp 196–208
Zhang Y, Xiong Y, Kong X, Li S, Mi J, Zhu Y (2018) Deep collective classification in heterogeneous information networks. In: Champin P-A, Gandon F, Lalmas M, Ipeirotis PG (eds) Proceedings of the 2018 world wide web conference on world wide web, WWW 2018, Lyon, France, April 23–27, 2018. ACM, pp 399–408
Zhou D, Bousquet O, Navin TL, Weston J, Schölkopf B (2003) Learning with local and global consistency. In: Thrun S, Saul LK, Schölkopf B (eds) Advances in neural information processing systems 16 [neural information processing systems, NIPS 2003, December 8–13, 2003, Vancouver and Whistler, British Columbia, Canada]. MIT Press, pp 321–328
Zhou X, Belkin M (2014) Semi-supervised learning. In: Academic press library in signal processing. Elsevier, vol 1, pp 1239–1269
Zhuang C, Ma Q (2018) Dual graph convolutional networks for graph-based semi-supervised classification. In: Champin P-A, Gandon F, Lalmas M, Ipeirotis PG (eds) Proceedings of the 2018 world wide web conference on world wide web, WWW 2018, Lyon, France, April 23–27, 2018. ACM, pp 499–508
Acknowledgements
We thank anonymous reviewers for their very useful comments and suggestions. Part of this work was done, while Li Shen and Ling Cheng were doing research in Griffith University. The work was supported by Australian Research Council (ARC) Large Grant A849602031.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Sankar, A., Wang, J., Krishnan, A. et al. Self-supervised role learning for graph neural networks. Knowl Inf Syst 64, 2091–2121 (2022). https://doi.org/10.1007/s10115-022-01694-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-022-01694-5