Skip to main content
SpringerLink
Log in

Embedding text-rich graph neural networks with sequence and topical semantic structures

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Graph neural networks (GNNs) have demonstrated great power in tackling various analytical tasks on graph (i.e. network) data. However, graphs in the real world are usually text-rich, implying that valuable semantic structures need to be considered carefully. Existing GNNs for text-rich networks typically treat the text as attribute words alone, which inevitably leads to the loss of important semantic structures, limiting the representation capability of GNNs. To solve this limitation, we propose AS-GNN, an end-to-end adaptive GNN architecture via unified modelling of semantic structure and network propagation on text-rich networks. Specifically, we utilize semantic structure modelling part to capture both the local word-sequence and the global topic semantic structures from text. We then augment the original text-rich network into a tri-typed heterogeneous network (including document nodes, word nodes, and topic nodes) and accordingly design a semantic-aware propagation of information by introducing a discriminative convolutional mechanism. We further train these two parts together by leveraging distribution sharing and joint training strategies, so as to adaptively generate an appropriate network structure aiming at the learning objectives. In addition, we present a simplified semantic architecture S-GNN, which adopts the cascaded “Structure-GNN” pattern, to promote the efficiency of the model and be easily combined with existing GNNs. Extensive experiments on text-rich networks demonstrate the superiority of our new architectures over state of the arts. Meanwhile, such architectures can also be applied to e-commerce search scenes, and experiments on a real e-commerce problem from JD further illustrate the effectiveness of AS-GNN over the baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://zhang18f.myweb.cs.uwindsor.ca/datasets/.

  2. https://www.cs.cornell.edu/projects/kddcup/datasets.html.

References

  1. Long Q, Jin Y, Wu Y, Song G (2021) Theoretically improving graph neural networks via anonymous walk graph kernels. In: Proceedings of WWW, pp 1204–1214

  2. Jin D, Huo C, Liang C, Yang L (2021) Heterogeneous graph neural network via attribute completion. In: Proceedings of WWW, pp 391–400

  3. Rong Y, Huang W, Xu T, Huang J (2020) Dropedge: towards deep graph convolutional networks on node classification. In: Proceedings of ICLR

  4. Wu J, He J, Xu J (2019) Demo-net: degree-specific graph neural networks for node and graph classification. In: Proceedings of SIGKDD, pp 406–415

  5. You J, Ying R, Leskovec J (2019) Position-aware graph neural networks. In: Proceedings of ICML, pp 7134–7143

  6. Liu Z, Wan M, Guo S, Achan K, Yu PS (2020) Basconv: aggregating heterogeneous interactions for basket recommendation with graph convolutional neural network. In: Proceedings of SDM, pp 64–72

  7. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: Proceedings of ICLR

  8. Velickovic P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: Proceedings of ICLR

  9. Bachmann G, Bécigneul G, Ganea O (2020) Constant curvature graph convolutional networks. In: Proceedings of ICML, pp 486–496

  10. Jin D, Huang J, Jiao P, Yang L, He D, Fogelman-Soulié F, Huang Y (2019) A novel generative topic embedding model by introducing network communities. In: Proceedings of WWW, pp 2886–2892

  11. Yang C, Liu Z, Zhao D, Sun M, Chang EY (2015) Network representation learning with rich text information. In: Proceedings of IJCAI, pp 2111–2117

  12. Tang J, Zhang J, Yao L, Li J, Zhang L, Su Z (2008) Arnetminer: extraction and mining of academic social networks. In: Proceedings of SIGKDD, pp 990–998

  13. Errica F, Podda M, Bacciu D, Micheli A (2020) A fair comparison of graph neural networks for graph classification. In: Proceedings of ICLR

  14. Zhu J, Yan Y, Zhao L, Heimann M, Akoglu L, Koutra D (2020) Beyond homophily in graph neural networks: current limitations and effective designs. In: Proceedings of NeurIPS

  15. Zhu J, Rossi RA, Rao AB, Mai T, Lipka N, Ahmed NK, Koutra D (2021) Graph neural networks with heterophily. In: Proceedings of AAAI, pp 11168–11176

  16. Shi Y, Shen J, Li Y, Zhang N, He X, Lou Z, Zhu Q, Walker M, Kim M, Han J (2019) Discovering hypernymy in text-rich heterogeneous information network by exploiting context granularity. In: Proceedings of CIKM, pp 599–608

  17. Shang J, Zhang X, Liu L, Li S, Han J (2020) Nettaxo: Automated topic taxonomy construction from text-rich network. In: Proceedings of WWW, pp 1908–1919

  18. Jin D, Yu Z, Jiao P, Pan S, Yu PS, Zhang W (2021) A survey of community detection approaches: from statistical modeling to deep learning. TKDE. https://doi.org/10.1109/TKDE.2021.3104155

  19. Hu F, Zhu Y, Wu S, Wang L, Tan T (2019) Hierarchical graph convolutional networks for semi-supervised node classification. In: Proceedings of IJCAI, pp 4532–4539

  20. Liu X, You X, Zhang X, Wu J, Lv P (2020) Tensor graph convolutional networks for text classification. In: Proceedings of AAAI, pp 8409–8416

  21. Miao Y, Grefenstette E, Blunsom P (2017) Discovering discrete latent topics with neural variational inference. In: Proceedings of ICML, vol 70, pp 2410–2419

  22. Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: Proceedings of ICLR

  23. Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  24. Devlin J, Chang M, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp 4171–4186

  25. Yang Y, Wang X, Song M, Yuan J, Tao D (2019) SPAGAN: shortest path graph attention network. In: Proceedings of IJCAI, pp 4099–4105

  26. Wang X, Ji H, Shi C, Wang B, Ye Y, Cui P, Yu PS (2019) Heterogeneous graph attention network. In: Proceedings of WWW, pp 2022–2032

  27. Velickovic P, Fedus W, Hamilton WL, Liò P, Bengio Y, Hjelm RD (2019) Deep graph infomax. In: Proceedings of ICLR

  28. Hamilton WL, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs. In: Proceedings of NeurIPS, pp 1024–1034

  29. Wang X, Zhu M, Bo D, Cui P, Shi C, Pei J (2020) AM-GCN: adaptive multi-channel graph convolutional networks. In: Proceedings of SIGKDD, pp 1243–1253

  30. Pei H, Wei B, Chang KC, Lei Y, Yang B (2020) Geom-gcn: Geometric graph convolutional networks. In: Proceedings of ICLR

  31. Jin D, Song X, Yu Z, Liu Z, Zhang H, Cheng Z, Han J (2021) Bite-gcn: a new GCN architecture via bidirectional convolution of topology and features on text-rich networks. In: Proceedings of WSDM, pp 157–165

  32. Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: Proceedings of EMNLP, pp 1532–1543

  33. Cui P, Liu Y, Liu B (2019) A neural topic model based on variational auto-encoder for aspect extraction from opinion texts. In: Proceedings of NLPCC, vol 11838, pp 660–671

  34. Cui P, Hu L, Liu Y (2020) Enhancing extractive text summarization with topic-aware graph neural networks. In: Proceedings of COLING, pp 5360–5371

  35. Laurens VDM, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(2605):2579–2605

    MATH  Google Scholar 

  36. Jin D, Wang K, Zhang G, Jiao P, He D, Fogelman-Soulié F, Huang X (2020) Detecting communities with multiplex semantics by distinguishing background, general, and specialized topics. IEEE Trans Knowl Data Eng 32(11):2144–2158

    Article  Google Scholar 

  37. Wan S, Lan Y, Guo J, Xu J, Pang L, Cheng X (2016) A deep architecture for semantic matching with multiple positional sentence representations. In: Proceedings of AAAI, pp 2835–2841

  38. Hu B, Lu Z, Li H, Chen Q (2014) Convolutional neural network architectures for matching natural language sentences. In: Proceedings of NeurIPS, pp 2042–2050

  39. Xiong C, Dai Z, Callan J, Liu Z, Power R (2017) End-to-end neural ad-hoc ranking with kernel pooling. In: Proceedings of SIGIR, pp 55–64

  40. Pang L, Lan Y, Guo J, Xu J, Wan S, Cheng X (2016) Text matching as image recognition. In: Proceedings of AAAI, vol 16, pp 2793–2799

  41. Mitra B, Diaz F, Craswell N (2017) Learning to match using local and distributed representations of text for web search. In: Proceedings of WWW, pp 1291–1299

  42. Zhang X, Zhang C, Dong XL, Shang J, Han J (2021) Minimally-supervised structure-rich text categorization via learning on text-rich networks. In: Proceedings of WWW

  43. Chen W, Liu C, Yin J, Yan H, Zhang Y (2017) Mining e-commercial data: a text-rich heterogeneous network embedding approach. In: Proceedings of IJCNN, pp 1403–1410

  44. Wang L, Yu X, Tao F (2019) A community-enhanced retrieval model for text-rich heterogeneous information networks. In: Proceedings of ICDM workshops, pp 505–513

  45. Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of AAAI, pp 7370–7377

  46. Ragesh R, Sellamanickam S, Iyer A, Bairi R, Lingam V (2021) Hetegcn: heterogeneous graph convolutional networks for text classification. In: Proceedings of WSDM, pp 860–868

  47. Zhang Y, Yu X, Cui Z, Wu S, Wen Z, Wang L (2020) Every document owns its structure: inductive text classification via graph neural networks. In: Proceedings of ACL, pp 334–339

  48. Sun K, Lin Z, Zhu Z (2020) Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes. In: Proceedings of AAAI, pp 5892–5899

  49. Xu K, Li C, Tian Y, Sonobe T, Kawarabayashi K, Jegelka S (2018) Representation learning on graphs with jumping knowledge networks. In: Proceedings of ICML, vol 80, pp 5449–5458

  50. Jiang Y, Shang Y, Liu Z, Shen H, Xiao Y, Xiong W, Xu S, Yan W, Jin D (2020) BERT2DNN: BERT distillation with massive unlabeled data for online e-commerce search. In: Proceedings of ICDM, pp 212–221

  51. Nguyen TV, Rao N, Subbian K (2020) Learning robust models for e-commerce product search. In: Proceedings of ACL, pp 6861–6869

  52. Niu X, Li B, Li C, Xiao R, Sun H, Wang H, Deng H, Chen Z (2020) Gated heterogeneous graph representation learning for shop search in e-commerce. In: Proceedings of CIKM, pp 2165–2168

  53. Yu Z, Jin D, Liu Z, He D, Wang X, Tong H, Han J (2021) AS-GCN: adaptive semantic architecture of graph convolutional networks for text-rich networks. In: Proceedings of ICDM, pp 837–846

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of China under Grants 62272340, 62276187, 61876128, and 62172052.

Author information

Author notes

  1. Work is done during Ziyang Liu work at JD.com.

Authors and Affiliations

  1. College of Intelligence and Computing, Tianjin University, Tianjin, 300350, China

    Zhizhi Yu, Di Jin & Dongxiao He

  2. School of Software, Tsinghua University, Beijing, 100084, China

    Ziyang Liu

  3. School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Xiao Wang

  4. Department of Computer Science, University of Illinois at Urbana-Champaign, Champaign, IL, 61801, USA

    Hanghang Tong & Jiawei Han

Authors
  1. Zhizhi Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Di Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziyang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dongxiao He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hanghang Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jiawei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dongxiao He.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, Z., Jin, D., Liu, Z. et al. Embedding text-rich graph neural networks with sequence and topical semantic structures. Knowl Inf Syst 65, 613–640 (2023). https://doi.org/10.1007/s10115-022-01768-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-022-01768-4

Keywords

Search

Navigation