Skip to main content

Modified SkipGram Negative Sampling Model for Faster Convergence of Graph Embedding

  • Conference paper
  • First Online:
Deep Learning Theory and Applications (DeLTA 2022)

Abstract

Graph embedding techniques have been introduced in recent years with the aim of mapping graph data into low-dimensional vector spaces, so that conventional machine learning methods can be exploited. In particular, in the DeepWalk model, truncated random walks are employed in random walk-based approaches to capture structural links-connections between nodes. The SkipGram model is then applied to the truncated random walks to compute the embedded nodes. In this work, the proposed DeepWalk model provides a faster convergence speed than the standard one by introducing a new trainable parameter in the model. Furthermore, experimental results on real-world datasets show that the performance in downstream community detection and link prediction task is improved by using the proposed DeepWalk model.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    \(M_{n \times m}\) denotes the set of matrices \(n \times m\).

References

  1. Banerjee, K., Gupta, R.R., Vyas, K., Mishra, B.: Exploring alternatives to softmax function. In: Proceedings of the 2nd International Conference on Deep Learning Theory and Applications - DeLTA, pp. 81–86. INSTICC, SciTePress (2021). https://doi.org/10.5220/0010502000810086

  2. Bhagat, S., Cormode, G., Muthukrishnan, S.: Node classification in social networks. In: Aggarwal, C. (ed.) Social Network Data Analytics, pp. 115–148. Springer, Boston (2011). https://doi.org/10.1007/978-1-4419-8462-3_5

  3. Cai, H., Zheng, V.W., Chang, K.C.C.: A comprehensive survey of graph embedding: problems, techniques, and applications. IEEE Trans. Knowl. Data Eng. 30(9), 1616–1637 (2018)

    Article  Google Scholar 

  4. Cao, S., Lu, W., Xu, Q.: Grarep: learning graph representations with global structural information. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 891–900 (2015)

    Google Scholar 

  5. Chen, H., Perozzi, B., Hu, Y., Skiena, S.: Harp: hierarchical representation learning for networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  6. Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015)

  7. Fawcett, T.: An introduction to roc analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)

    Article  MathSciNet  Google Scholar 

  8. Goyal, P., Ferrara, E.: Graph embedding techniques, applications, and performance: a survey. Knowl.-Based Syst. 151, 78–94 (2018)

    Article  Google Scholar 

  9. Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864 (2016)

    Google Scholar 

  10. Hartigan, J.A., Wong, M.A.: Algorithm as 136: a k-means clustering algorithm. J. Roy. Stat. Soc. Ser. C (applied statistics) 28(1), 100–108 (1979)

    Google Scholar 

  11. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertainty Fuzziness Knowl.-Based Syst. 6(02), 107–116 (1998)

    Article  MATH  Google Scholar 

  12. Hosmer, Jr, D.W., Lemeshow, S., Sturdivant, R.X.: Applied Logistic Regression, vol. 398. Wiley, Hoboken (2013)

    Google Scholar 

  13. Ketkar, Nikhil: Stochastic gradient descent. In: Deep Learning with Python, pp. 111–130. Apress, Berkeley (2017). https://doi.org/10.1007/978-1-4842-2766-4_8

    Chapter  Google Scholar 

  14. Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. Adv. Neural information processing systems 30 (2017)

    Google Scholar 

  15. Kosmatopoulos, A., Loumponias, K., Chatzakou, D., Tsikrika, T., Vrochidis, S., Kompatsiaris, I.: Random-walk graph embeddings and the influence of edge weighting strategies in community detection tasks. In: Proceedings of the 2021 Workshop on Open Challenges in Online Social Networks. pp. 9–13 (2021)

    Google Scholar 

  16. Leskovec, J., Krevl, A.: SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data (Jun 2014)

  17. Li, J., Zhu, J., Zhang, B.: Discriminative deep random walk for network classification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1004–1013 (2016)

    Google Scholar 

  18. Loumponias, K., Kosmatopoulos, A., Tsikrika, T., Vrochidis, S., Kompatsiaris, I.: A faster converging negative sampling for the graph embedding process in community detection and link prediction tasks. In: Proceedings of the 3rd International Conference on Deep Learning Theory and Applications - Volume 1: DeLTA, pp. 86–93. INSTICC, SciTePress (2022). DOI: 10.5220/0011142000003277

    Google Scholar 

  19. Luo, X., Chang, X., Ban, X.: Regression and classification using extreme learning machine based on l1-norm and l2-norm. Neurocomputing 174, 179–186 (2016)

    Article  Google Scholar 

  20. Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)

    Google Scholar 

  21. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

  22. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26, 3111–3119 (2013)

    Google Scholar 

  23. Pan, S., Hu, R., Long, G., Jiang, J., Yao, L., Zhang, C.: Adversarially regularized graph autoencoder for graph embedding. arXiv preprint arXiv:1802.04407 (2018)

  24. Pan, S., Wu, J., Zhu, X., Zhang, C., Wang, Y.: Tri-party deep network representation. Network 11(9), 12 (2016)

    Google Scholar 

  25. Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: Online learning of social representations. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 701–710 (2014)

    Google Scholar 

  26. Perozzi, B., Kulkarni, V., Chen, H., Skiena, S.: Don’t walk, skip! online learning of multi-scale network embeddings. In: Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. pp. 258–265 (2017)

    Google Scholar 

  27. Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., Eliassi-Rad, T.: Collective classification in network data. AI magazine 29(3), 93–106 (2008)

    Article  Google Scholar 

  28. Vinh, N.X., Epps, J., Bailey, J.: Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. The Journal of Machine Learning Research 11, 2837–2854 (2010)

    MathSciNet  MATH  Google Scholar 

  29. Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., Weinberger, K.: Simplifying graph convolutional networks. In: International conference on machine learning. pp. 6861–6871. PMLR (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kostas Loumponias .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Loumponias, K., Kosmatopoulos, A., Tsikrika, T., Vrochidis, S., Kompatsiaris, I. (2023). Modified SkipGram Negative Sampling Model for Faster Convergence of Graph Embedding. In: Fred, A., Sansone, C., Gusikhin, O., Madani, K. (eds) Deep Learning Theory and Applications. DeLTA 2022. Communications in Computer and Information Science, vol 1858. Springer, Cham. https://doi.org/10.1007/978-3-031-37317-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-37317-6_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-37316-9

  • Online ISBN: 978-3-031-37317-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics