Xin Jiang
Xindong Wu
Skip Abstract Section
Abstract
Temporal link prediction (TLP) is among the most important graph learning tasks, capable of predicting dynamic, time-varying links within networks. The key problem of TLP is how to explore potential link-evolving tendency from the increasing number of links over time. There exist three major challenges toward solving this problem: temporal nonlinear sparsity, weak serial correlation, and discontinuous structural dynamics. In this article, we propose a novel transfer learning model, called DNformer, to predict temporal link sequence in dynamic networks. The structural dynamic evolution is sequenced into consecutive links one by one over time to inhibit temporal nonlinear sparsity. The self-attention of the model is used to capture the serial correlation between the input and output link sequences. Moreover, our structural encoding is designed to obtain changing structures from the consecutive links and to learn the mapping between link sequences. This structural encoding consists of two parts: the node clustering encoding of each link and the link similarity encoding between links. These encodings enable the model to perceive the importance and correlation of links. Furthermore, we introduce a measurement of structural similarity in the loss function for the structural differences of link sequences. The experimental results demonstrate that our model outperforms other state-of-the-art TLP methods such as Transformer, TGAT, and EvolveGCN. It achieves the three highest AUC and four highest precision scores in five different representative dynamic networks problems.
- [1] . 2016. Layer normalization. arXiv e-prints, arXiv: stat.ML/1607.06450.Google Scholar
- [2] . 2020. Graph transformer for graph-to-sequence learning. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 7464–7471.Google ScholarCross Ref
- [3] . 2021. E-LSTM-D: A deep learning framework for dynamic network link prediction. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51, 6 (2021), 3699–3712.
DOI: Google ScholarCross Ref - [4] . 2022. Effective deep attributed network representation learning with topology adapted smoothing. IEEE Transactions on Cybernetics 52, 7 (2022), 5935–5946.
DOI: Google ScholarCross Ref - [5] . 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.
DOI: Google ScholarCross Ref - [6] . 2018. Dynamic network embedding: An extended approach for skip-gram based network embedding. In Proceedings of the 27th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 2086–2092.Google ScholarCross Ref
- [7] . 2011. Temporal link prediction using matrix and tensor factorizations. ACM Transactions on Knowledge Discovery from Data 5, 2, Article
10 (Feb. 2011), 27 pages.DOI: Google ScholarDigital Library
- [8] . 2020. A generalization of transformer networks to graphs. arXiv e-prints, arXiv:2012.09699.Google Scholar
- [9] . 2019. Taxonomy and evaluation for microblog popularity prediction. ACM Transactions on Knowledge Discovery from Data 13, 2, Article
15 (March 2019), 40 pages.DOI: Google ScholarDigital Library - [10] . 2020. Conformer: Convolution-augmented transformer for speech recognition. In Proceedings of INTERSPEECH. International Speech Communication Association, Shanghai, China, 5036–5040.Google ScholarCross Ref
- [11] . 2020. GloDyNE: Global topology preserving dynamic network embedding. IEEE Transactions on Knowledge and Data Engineering (2020), 1–1.
DOI: Google ScholarDigital Library - [12] . 2020. Heterogeneous graph transformer. In Proceedings of the Web Conference 2020. Association for Computing Machinery, New York, NY, 2704–2710.
DOI: Google ScholarDigital Library - [13] . 2013. Social trust prediction using heterogeneous networks. ACM Transactions on Knowledge Discovery from Data 7, 4, Article
17 (Dec. 2013), 21 pages.DOI: Google ScholarDigital Library - [14] . 2016. Evaluating link prediction accuracy in dynamic networks with added and removed edges. In Proceedings of 2016 IEEE International Conferences on Big Data and Cloud Computing. IEEE Computer Society, Los Alamitos, CA, 377–384.
DOI: Google ScholarCross Ref - [15] . 2003. Maximizing the spread of influence through a social network. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, 137–146.
DOI: Google ScholarDigital Library - [16] . 2019. Text generation from knowledge graphs with graph transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Minneapolis, Minnesota, 2284–2293.
DOI: Google ScholarCross Ref - [17] . 2014. A deep learning approach to link prediction in dynamic networks. In Proceedings of the 2014 SIAM International Conference on Data Mining, Vol. 1. Society for Industrisl and Applied Mathematics, Philadelphia, Pennsylvania, 289–297.Google ScholarCross Ref
- [18] . 2007. The link-prediction problem for social networks. Journal of the American Society for Information Science and Technology 58, 7 (2007), 1019–1031.
DOI: Google ScholarDigital Library - [19] . 2017. Spatial prediction for multivariate non-Gaussian data. ACM Transactions on Knowledge Discovery from Data 11, 3, Article
36 (March 2017), 27 pages.DOI: Google ScholarDigital Library - [20] . 2020. Learning to encode position for transformer with continuous dynamical model. In Proceedings of the 37th International Conference on Machine Learning. JMLR.org, Vienna, AUSTRIA, Article
587 , 9 pages.Google Scholar - [21] . 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the2021 IEEE/CVF International Conference on Computer Vision. Computer Vision Foundation, Montreal, Canada, 9992–10002.
DOI: Google ScholarCross Ref - [22] . 2020. EvolveGCN: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 5679–5681.Google ScholarCross Ref
- [23] . 2015. Local laplacian filters: Edge-aware image processing with a Laplacian pyramid. Communications of the ACM 58, 3 (
Feb. 2015), 81–91.DOI: Google ScholarDigital Library - [24] . 2020. Dynamic network embedding via incremental skip-gram with negative sampling. Science China Information Sciences 63, 10 (2020), 19.
DOI: Google ScholarCross Ref - [25] . 2022. Lime: Low-cost and incremental learning for dynamic heterogeneous information networks. IEEE Transactions on Computers 71, 3 (2022), 628–642.
DOI: Google ScholarDigital Library - [26] . 2014. DeepWalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, 701–710.Google ScholarDigital Library
- [27] . 2022. The node-similarity distribution of complex networks and its applications in link prediction. IEEE Transactions on Knowledge and Data Engineering 34, 8 (2022), 4011–4023.
DOI: Google ScholarCross Ref - [28] . 2021. Temporal link prediction with motifs for social networks. IEEE Transactions on Knowledge and Data Engineering (2021), 1–1.
DOI: Google ScholarCross Ref - [29] . 2020. Self-supervised graph transformer on large-scale molecular data. In Proceedings of Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., Vancouver, Canada, 12559–12571.Google Scholar
- [30] . 2021. Knowledge graph embedding for link prediction: A comparative analysis. ACM Transactions on Knowledge Discovery from Data 15, 2, Article
14 (Jan. 2021), 49 pages.DOI: Google ScholarDigital Library - [31] . 2015. The network data repository with interactive graph analytics and visualization. In Proceedings of AAAI Conference on Artificial Intelligence.AAAI Press, Palo Alto, CA, 4292–4293. Retrieved from http://networkrepository.com.Google ScholarCross Ref
- [32] . 2008. Temporal-relational classifiers for prediction in evolving domains. In Proceedings of 8th IEEE international conference on data mining. IEEE Computer Society, Los Alamitos, CA, 540–549.Google ScholarDigital Library
- [33] . 2016. Partitioning networks with node attributes by compressing information flow. ACM Transactions on Knowledge Discovery from Data 11, 2 (2016), 1–26.
DOI: Google ScholarDigital Library - [34] . 2018. Substructural regularization with data-sensitive granularity for sequence transfer learning. IEEE Transactions on Neural Networks and Learning Systems 29, 6 (2018), 2545–2557.
DOI: Google ScholarCross Ref - [35] . 2014. Co-evolution of multi-typed objects in dynamic star networks. IEEE Transactions on Knowledge and Data Engineering 26, 12 (2014), 2942–2955.
DOI: Google ScholarCross Ref - [36] . 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, 6000–6010.Google ScholarDigital Library
- [37] . 2017. Nodes’ evolution diversity and link prediction in social networks. IEEE Transactions on Knowledge and Data Engineering 29, 10 (2017), 2263–2274.
DOI: Google ScholarDigital Library - [38] . 1994. Social Network Analysis: Methods and Applications. Cambridge University Press, Cambridge, UK.
DOI: Google ScholarCross Ref - [39] . 2021. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems 32, 1 (2021), 4–24.
DOI: Google ScholarCross Ref - [40] . 2020. Inductive representation learning on temporal graphs. In Proceedings of International Conference on Learning Representations. OpenReview.net, Ithaca, NY, 1.Google Scholar
- [41] . 2020. Exploring evolution of dynamic networks via diachronic node embeddings. IEEE Transactions on Visualization and Computer Graphics 26, 7 (2020), 2387–2402.
DOI: Google ScholarCross Ref - [42] . 2019. How powerful are graph neural networks? In Proceedings of International Conference on Learning Representations. OpenReview.net, Ithaca, NY, 1.Google Scholar
- [43] . 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the 32th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 7444–7452.Google ScholarCross Ref
- [44] . 2020. An advanced deep generative framework for temporal link prediction in dynamic networks. IEEE Transactions on Cybernetics 50, 12 (2020), 4946–4957. Google ScholarCross Ref
- [45] . 2021. Do transformers really perform badly for graph representation? In Proceedings of Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., Vancouver, Canada, 28877–28888.Google Scholar
- [46] . 2017. Temporally factorized network modeling for evolutionary network analysis. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, 455–464.Google ScholarDigital Library
- [47] . 2018. Link prediction based on graph neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, 5171–5181.Google ScholarDigital Library
- [48] . 2018. Dynamic network embedding by modeling triadic closure process. In Proceedings of the 32th AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 571–578.Google ScholarCross Ref
- [49] . 2016. Scalable temporal latent space inference for link prediction in dynamic social networks. IEEE Transactions on Knowledge and Data Engineering 28, 10 (2016), 2765–2777. Google ScholarDigital Library
Index Terms
- DNformer: Temporal Link Prediction with Transfer Learning in Dynamic Networks
Recommendations
Few-shot Link Prediction in Dynamic Networks
WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data MiningDynamic link prediction, which aims at forecasting future edges of a node in a dynamic network, is an important problem in network science and has a wide range of real-world applications. A key property of dynamic networks is that new nodes and links ...
A supervised link prediction method for dynamic networks
Link prediction is an important sub-task in link mining area. This paper discusses link prediction in dynamic networks and proposes a new link prediction method which can learn from the long-term graph evolution of networks. The method first represents ...
Joint multi-label learning and feature extraction for temporal link prediction
Highlights- We show multi-label learning can be applied to temporal link prediction.
- We ...
AbstractNetworks derived from various disciplinary of sociality and nature are dynamic and incomplete, and temporal link prediction has wide applications in recommendation system and data mining system, etc. The current algorithms first obtain ...
Comments