Yoon-Sik Cho
ABSTRACT
Link prediction is an important learning task for graph-structured data, and has become increasingly popular due to its wide application areas. Graph Neural Network (GNN)-based approaches including Variational Graph Autoencoder (VGAE) have achieved promising performance on link prediction outperforming conventional models which use hand-crafted features. VGAE learns latent node representations and predicts links based on the similarities between nodes. While the inner product based decoder effectively utilizes the node representations for link prediction, it exhibits sub-optimal performance due to the intrinsic limitation of the inner product. We found that the the cosine similarity and norm simultaneously try to explain the link probability, which hinders the gradient flow during training. We also point out the message passing scheme is unexpectedly dominated by the nodes with large norm values. In this paper, we propose a stochastic VGAE-based method that can effectively decouple the norm and angle in the embeddings. Specifically, we relate the cosine similarity and norm to two fundamental principles in graph: homophily and node popularity respectively. Our learning scheme is based on a hard expectation maximization learning method; we infer which of the two has been exerted for link formation, and subsequently optimize based on this guess. Through extensive experiments on real-world datasets, we demonstrate our model outperforms the existing state-of-the-art methods on link prediction and achieves comparable performances on other downstream tasks such as node classification and clustering. Our code is at https://github.com/yoonsikcho/d-vgae.
Supplemental Material
- Seong Jin Ahn and MyoungHo Kim. 2021. Variational Graph Normalized AutoEncoders. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2827--2831.Google Scholar
Digital Library
- Edo M Airoldi, David Blei, Stephen Fienberg, and Eric Xing. 2008. Mixed membership stochastic blockmodels. Advances in neural information processing systems 21 (2008).Google Scholar
- Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 12449--12460. https://proceedings.neurips.cc/paper/2020/file/ 92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdfGoogle Scholar
- Sergey Brin and Lawrence Page. 1998. The Anatomy of a Large-Scale Hypertextual Web Search Engine. In Proceedings of the Seventh International Conference on World Wide Web 7 (Brisbane, Australia) (WWW7). Elsevier Science Publishers B. V., NLD, 107--117.Google ScholarDigital Library
- Lei Cai, Jundong Li, JieWang, and Shuiwang Ji. 2021. Line graph neural networks for link prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).Google ScholarCross Ref
- Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems. 39--46.Google ScholarDigital Library
- Xinhan Di, Pengqian Yu, Rui Bu, and Mingchao Sun. 2020. Mutual information maximization in graph neural networks. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--7.Google ScholarCross Ref
- Yuntao Du, Xinjun Zhu, Lu Chen, Baihua Zheng, and Yunjun Gao. 2022. HAKG: Hierarchy-Aware Knowledge Gated Network for Recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1390--1400.Google ScholarDigital Library
- Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.Google Scholar
- Santo Fortunato. 2010. Community detection in graphs. Physics Reports 486, 3 (2010), 75--174. https://doi.org/10.1016/j.physrep.2009.11.002Google ScholarCross Ref
- C. Lee Giles, Kurt D. Bollacker, and Steve Lawrence. 1998. CiteSeer: An Automatic Citation Indexing System. In Proceedings of the Third ACM Conference on Digital Libraries, Pittsburgh, PA, USA. ACM, New York, NY, USA, 89--98. https://doi.org/ 10.1145/276675.276685Google ScholarDigital Library
- Prem K Gopalan, Chong Wang, and David Blei. 2013. Modeling overlapping communities with node popularities. Advances in neural information processing systems 26 (2013).Google Scholar
- Aditya Grover and Jure Leskovec. 2016. Node2vec: Scalable Feature Learning for Networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD '16). Association for Computing Machinery, New York, NY, USA, 855--864. https: //doi.org/10.1145/2939672.2939754Google ScholarDigital Library
- Aditya Grover, Aaron Zweig, and Stefano Ermon. 2019. Graphite: Iterative generative modeling of graphs. In International conference on machine learning. PMLR, 2434--2444.Google Scholar
- Zhihao Guo, Feng Wang, Kaixuan Yao, Jiye Liang, and Zhiqiang Wang. 2022. Multi-scale variational graph autoencoder for link prediction. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 334--342.Google ScholarDigital Library
- Arman Hasanzadeh, Ehsan Hajiramezanali, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. 2019. Semi-implicit graph variational auto-encoders. Advances in neural information processing systems 32 (2019).Google Scholar
- Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical Reparametrization with Gumbel-Softmax. In Proceedings International Conference on Learning Representations 2017. OpenReviews.net. https://openreview.net/pdf?id=rkE3y85eeGoogle Scholar
- Brian Karrer and Mark EJ Newman. 2011. Stochastic blockmodels and community structure in networks. Physical review E 83, 1 (2011), 016107.Google Scholar
- Leo Katz. 1953. A new status index derived from sociometric analysis. Psychometrika 18, 1 (1953), 39--43.Google ScholarCross Ref
- Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved Variational Inference with Inverse Autoregressive Flow. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2016/file/ ddeebdeefdb7e7e7a697e1c3e3d8ef54-Paper.pdfGoogle Scholar
- Thomas N Kipf and MaxWelling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).Google Scholar
- Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016).Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations (ICLR).Google Scholar
- Pavel N Krivitsky, Mark S Handcock, Adrian E Raftery, and Peter D Hoff. 2009. Representing degree distributions, clustering, and homophily in social networks with latent cluster random effects models. Social networks 31, 3 (2009), 204--213.Google Scholar
- Matt J Kusner and José Miguel Hernández-Lobato. 2016. Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.04051 (2016).Google Scholar
- Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra. 2016. Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc. https://proceedings.neurips.cc/paper_ files/paper/2016/file/20d135f0f28185b84a4cf7aa51f29500-Paper.pdfGoogle Scholar
- Hongchao Li, Jianxun Liu, Buqing Cao, Mingdong Tang, Xiaoqing Liu, and Bing Li. 2017. Integrating tag, topic, co-occurrence, and popularity to recommend web apis for mashup creation. In 2017 IEEE International Conference on Services Computing (SCC). IEEE, 84--91.Google ScholarCross Ref
- Xueqi Li, Guoqing Xiao, Yuedan Chen, Zhuo Tang, Wenjun Jiang, and Kenli Li. 2023. An Explicitly Weighted GCN Aggregator based on Temporal and Popularity Features for Recommendation. ACM Transactions on Recommender Systems (2023).Google Scholar
- David Liben-Nowell and Jon Kleinberg. 2007. The Link-Prediction Problem for Social Networks. J. Am. Soc. Inf. Sci. Technol. 58, 7 (may 2007), 1019--1031.Google ScholarDigital Library
- Kang Liu, Feng Xue, Xiangnan He, Dan Guo, and Richang Hong. 2022. Joint multi-grained popularity-aware graph convolution collaborative filtering for recommendation. IEEE Transactions on Computational Social Systems (2022).Google ScholarCross Ref
- Costas Mavromatis and George Karypis. 2021. Graph InfoClust: Maximizing Coarse-Grain Mutual Information in Graphs. In PAKDD.Google Scholar
- Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 43--52.Google ScholarDigital Library
- Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. 2000. Automating the Construction of Internet Portals with Machine Learning. Information Retrieval 3, 2 (2000), 127--163. https://doi.org/10.1023/A:1009953814988Google ScholarDigital Library
- Miller McPherson, Lynn Smith-Lovin, and JamesMCook. 2001. Birds of a Feather: Homophily in Social Networks. Annual Review of Sociology 27, 1 (2001), 415--444. https://doi.org/10.1146/annurev.soc.27.1.415Google ScholarCross Ref
- Galileo Mark Namata, Ben London, Lise Getoor, and Bert Huang. 2012. Querydriven Active Surveying for Collective Classification. In Workshop on Mining and Learning with Graphs. http://linqs.cs.umd.edu/basilic/web/Publications/2012/ namata:mlg12-wkshp/namata-mlg12.pdfGoogle Scholar
- Liming Pan, Cheng Shi, and Ivan Dokmanic. 2022. Neural Link Prediction with Walk Pooling. In International Conference on Learning Representations. https: //openreview.net/forum?id=CCu6RcUMwK0Google Scholar
- Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. 2018. Adversarially Regularized Graph Autoencoder for Graph Embedding. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (Stockholm, Sweden) (IJCAI'18). AAAI Press, 2609--2615.Google ScholarDigital Library
- George Papamakarios, Theo Pavlakou, and Iain Murray. 2017. Masked Autoregressive Flow for Density Estimation. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/ 6c1da886822c67822bcf3679d04369fa-Paper.pdfGoogle Scholar
- Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 701--710.Google ScholarDigital Library
- Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: Online Learning of Social Representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, New York, USA) (KDD '14). Association for Computing Machinery, New York, NY, USA, 701--710. https://doi.org/10.1145/2623330.2623732Google ScholarDigital Library
- Danilo Rezende and Shakir Mohamed. 2015. Variational Inference with Normalizing Flows. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 37), Francis Bach and David Blei (Eds.). PMLR, Lille, France, 1530--1538. https://proceedings.mlr.press/ v37/rezende15.htmlGoogle Scholar
- Guillaume Salha, Romain Hennequin, and Michalis Vazirgiannis. 2019. Keep It Simple: Graph Autoencoders Without Graph Convolutional Networks. Workshop on Graph Representation Learning, 33rd Conference on Neural Information Processing Systems (NeurIPS).Google Scholar
- Guillaume Salha, Stratis Limnios, Romain Hennequin, Viet-Anh Tran, and Michalis Vazirgiannis. 2019. Gravity-inspired graph autoencoders for directed link prediction. In Proceedings of the 28th ACM international conference on information and knowledge management. 589--598.Google ScholarDigital Library
- Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2018. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 (2018).Google Scholar
- Han Shi, Haozheng Fan, and James T. Kwok. 2020. Effective Decoding in Graph Auto-Encoder using Triadic Closure. In Proceedings of the Thirty-Fourth Conference on Association for the Advancement of Artificial Intelligence (AAAI). 906--913.Google Scholar
- Harald Steck. 2011. Item popularity and recommendation accuracy. In Proceedings of the fifth ACM conference on Recommender systems. 125--132.Google ScholarDigital Library
- Mingyue Tang, Pan Li, and Carl Yang. 2022. Graph Auto-Encoder via Neighborhood Wasserstein Reconstruction. In International Conference on Learning Representations. https://openreview.net/forum?id=ATUh28lnSuWGoogle Scholar
- Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. [n. d.]. Graph Attention Networks. In International Conference on Learning Representations.Google Scholar
- Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2018. Deep Graph Infomax. In International Conference on Learning Representations.Google Scholar
- Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. 2017. NormFace: L2 Hypersphere Embedding for Face Verification. In Proceedings of the 25th ACM International Conference on Multimedia (Mountain View, California, USA) (MM '17). Association for Computing Machinery, New York, NY, USA, 1041--1049. https://doi.org/10.1145/3123266.3123359Google ScholarDigital Library
- Max Welling and Diederik P Kingma. 2014. Auto-encoding variational bayes. ICLR (2014).Google Scholar
- Riting Xia, Yan Zhang, Chunxu Zhang, Xueyan Liu, and Bo Yang. 2023. Multihead Variational Graph Autoencoder Constrained by Sum-product Networks. In Proceedings of the ACM Web Conference 2023. 641--650.Google ScholarDigital Library
- Jaewon Yang and Jure Leskovec. 2014. Overlapping Communities Explain Core--Periphery Organization of Networks. Proc. IEEE 102, 12 (2014), 1892--1902. https://doi.org/10.1109/JPROC.2014.2364018Google ScholarCross Ref
- Mingzhang Yin and Mingyuan Zhou. 2018. Semi-Implicit Variational Inference. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 5660--5669. https://proceedings.mlr.press/v80/yin18b.htmlGoogle Scholar
- Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neural networks. Advances in neural information processing systems 31 (2018).Google Scholar
Index Terms
- Decoupled Variational Graph Autoencoder for Link Prediction
Recommendations
Multi-Scale Variational Graph AutoEncoder for Link Prediction
WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data MiningLink prediction has become a significant research problem in deep learning, and the graph-based autoencoder model is one of the most important methods to solve it. The existing graph-based autoencoder models only learn a single set of distributions, ...
Gravity-Inspired Graph Autoencoders for Directed Link Prediction
CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge ManagementGraph autoencoders (AE) and variational autoencoders (VAE) recently emerged as powerful node embedding methods. In particular, graph AE and VAE were successfully leveraged to tackle the challenging link prediction problem, aiming at figuring out whether ...
Heterogeneous Line Graph Neural Network for Link Prediction
Advanced Data Mining and ApplicationsAbstractHeterogeneous network link prediction is an important network information mining problem. Existing link prediction methods for heterogeneous networks typically require predefined meta-paths with prior knowledge. To address the problem, we propose ...
Comments