Skip to main content
SpringerLink
Log in

Deep learning, graph-based text representation and classification: a survey, perspectives and challenges

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Recently, with the rapid developments of the Internet and social networks, there have been tremendous increase in the amount of complex-structured text resources. These information explosions require extensive studies as well as more advanced methods in order to better understand and effectively model/learn these high-dimensional/structure-complicated textual datasets. Moving along with the recent progresses in deep learning and textual representation learning approaches, many researchers in this domain have been attracted by utilizing different deep neural architectures for learning essential features from texts. These novel neural architectures must enable to handle complex textual feature engineering. Moreover, it also has to be able to extract deeper semantic and structural information from textual resources. Recently, there are several integrations between advanced deep learning architectures, such as recurrent neural networks (RNNs), sequence-to-sequence (seq2seq) and transformers in text classification have been proposed. These hybrid deep neural architectures have shed light on how computers can comprehensively process sequential information from texts to fine-tune for leveraging the performance of multiple tasks in natural language processing, including classification. However, most of recent RNN-based techniques still suffer from several limitations. These limitations are mainly related to the capability of capturing the global long-range dependent as well syntactical structures of the given text corpus. There are some recent studies have shown that a combination of graph-based text representation and graph neural network (GNN) approaches can cope with these challenges. In this survey works, we mainly focus on discussing about recent state-of-the-art studies which are mainly dedicated on the text graph representation learning through GNN, named as TG-GNN. In addition, beside the TG-GNN based models’ features and capability discussions, we also mentioned about the pros/cons. Extensive comparative studies of TG-GNN based techniques in benchmark datasets for text classification problem are also provided in this survey. Finally, we highlight existing challenges as well as identify perspectives which might be useful for future improvements in this research direction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Pre-trained Word2Vec model: https://code.google.com/archive/p/word2vec/

  2. Pre-trained GloVe model: https://nlp.stanford.edu/projects/glove/

  3. Pre-trained fastText model: https://fasttext.cc/docs/en/english-vectors.html

  4. Pre-trained BERT model: https://github.com/google-research/bert

  5. TextGCN (python): https://github.com/codeKgu/Text-GCN

  6. TensorGCN (python): https://github.com/THUMLP/TensorGCN

  7. TextING (python): https://github.com/CRIPAC-DIG/TextING

  8. 20-NewsGroups dataset: http://qwone.com/~jason/20Newsgroups/

  9. WebKB dataset: http://www.cs.cmu.edu/~webkb/

  10. Reuters dataset: https://archive.ics.uci.edu/ml/datasets/reuters-21578+text+categorization+collection

  11. Ohsumed dataset: http://disi.unitn.it/moschitti/corpora.htm

  12. Movie Reviews (MR) dataset: https://github.com/mnqu/PTE/tree/master/data/mr

  13. Amazon Reviews (AR) dataset: https://snap.stanford.edu/data/web-Amazon.html

  14. IMDb dataset: https://datasets.imdbws.com/

  15. TextRNN (Python) (Hemmatian and Sohrabi 2019): https://github.com/ShawnyXiao/TextClassification-Keras

  16. CNN-based text classification (Python) (Singh et al. 2021): https://github.com/yoonkim/CNN_sentence

Abbreviations

AE:

Auto-encoding/auto-encoder

CNN:

Convolutional neural network

DL:

Deep learning

GAT:

Graph attention network

GCN:

Graph convolutional network

GNN:

Graph neural network

LSTM:

Long-short term memory

MLP:

Multi-layer perception

NLP:

Natural language processing

TG-GNN:

Text graph representation learning through graph neural network

\(\mathcal{d}\) and \(\mathcal{D}\) :

A document and a text corpus, respectively

\(\mathcal{w}\) and \(\mathcal{W}\) :

A word and a vocabulary set of the given text corpus, respectively

\(\mathcal{G}=(\mathcal{V},\mathcal{E})\) :

A graph-based structure with a set of nodes (\(\mathcal{V},\mathrm{v}\in \mathcal{V}\)) and edges (\(\mathcal{E},\mathrm{e}\in \mathcal{E}\))

\(\mathcal{A}\) :

The adjacency matrix of a given graph

\(\widehat{\mathcal{A}}\) :

Normalized version of adjacency matrix

\(\mathcal{H}\) :

The hidden state of a given neural network architecture

\(\upsigma (.)\) :

The sigmoid activation function

\(\mathrm{ReLU}(.)\) :

The rectified linear unit activation function

\(\mathrm{softmax}(.)\) :

The softmax function

\(\mathcal{Y}\) and \(\widehat{\mathcal{Y}}\) :

The sets of classification ground-truth and prediction labels, respectively

References

  • Altınel B, Ganiz MC (2018) Semantic text classification: a survey of past and recent advances. Inf Process Manage 54(6):1129–1153

    Article  Google Scholar 

  • Chen F, Wang YC, Wang B, Kuo CCJ (2020) Graph representation learning: a survey. APSIPA Trans Signal Inf Process 9:e15

    Article  Google Scholar 

  • Dhar A, Mukherjee H, Dash NS, Roy K (2021) Text categorization: past and present. Artif Intell Rev 54(4):3007–3054

    Article  Google Scholar 

  • Hemmatian F, Sohrabi MK (2019) A survey on classification techniques for opinion mining and sentiment analysis. Artif Intell Rev 52(3):1495–1545

    Article  Google Scholar 

  • Hu L, Li C, Shi C, Yang C, Shao C (2020a) Graph neural news recommendation with long-term and short-term interest modeling. Inf Process Manage 57(2):102142

    Article  Google Scholar 

  • Kadhim AI (2019) Survey on supervised machine learning techniques for automatic text classification. Artif Intell Rev 52(1):273–292

    Article  MathSciNet  Google Scholar 

  • Kowsari K, Jafari Meimandi K, Heidarysafa M, Mendu S, Barnes L, Brown D (2019) Text classification algorithms: a survey. Inf 10(4):150

    Google Scholar 

  • Ma G, Ahmed NK, Willke TL, Philip SY (2021) Deep graph similarity learning: a survey. Data Min Knowl Discov 35:688

    Article  MathSciNet  MATH  Google Scholar 

  • Minaee S, Kalchbrenner N, Cambria E, Nikzad N, Chenaghlu M, Gao J (2021) Deep learning–based text classification: a comprehensive review. ACM Comput Surv (CSUR) 54(3):1–40

    Article  Google Scholar 

  • Murari A, Gelfusa M, Lungaroni M, Gaudio P, Peluso E (2021) A systemic approach to classification for knowledge discovery with applications to the identification of boundary equations in complex systems. Artif Intell Rev 55:255

    Article  Google Scholar 

  • Nguyen G, Dlugolinsky S, Bobák M, Tran V, García ÁL, Heredia I et al (2019) Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey. Artif Intell Rev 52(1):77–124

    Article  Google Scholar 

  • Pham P, Nguyen LT, Vo B, Yun U (2021) Bot2Vec: a general approach of intra-community oriented representation learning for bot detection in different types of social networks. Inf Syst 103:101771

    Article  Google Scholar 

  • Pham P, Nguyen LT, Nguyen NT, Pedrycz W, Yun U, Vo B (2022a) ComGCN: community-driven graph convolutional network for link prediction in dynamic networks. IEEE Trans Syst Man Cybern Syst. https://doi.org/10.1109/TSMC.2021.3130149

    Article  Google Scholar 

  • Pham P, Pedrycz W, Vo B (2022b) Dual attention-based sequential auto-encoder for Covid-19 outbreak forecasting: a case study in Vietnam. Expert Syst Appl 203:117514

    Article  Google Scholar 

  • Pintas JT, Fernandes LA, Garcia ACB (2021) Feature selection methods for text classification: a systematic literature review. Artif Intell Rev 54:6149

    Article  Google Scholar 

  • Qiu X, Sun T, Xu Y, Shao Y, Dai N, Huang X (2020) Pre-trained models for natural language processing: a survey. Sci China Technol Sci 63:1872

    Article  Google Scholar 

  • Shivakumara P, Alaei A, Pal U (2021) Mining text from natural scene and video images: a survey. Wiley Interdiscip Rev Data Min Knowl Discov 11:e1428

    Article  Google Scholar 

  • Singh RK, Sachan MK, Patel RB (2021) 360 degree view of cross-domain opinion classification: a survey. Artif Intell Rev 54(2):1385–1506

    Article  Google Scholar 

  • Subhashini LDCS, Li Y, Zhang J, Atukorale AS, Wu Y (2021) Mining and classifying customer reviews: a survey. Artif Intell Rev 54:6343

    Article  Google Scholar 

  • Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2020) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24

    Article  MathSciNet  Google Scholar 

  • Wu JMT, Li Z, Herencsar N, Vo B, Lin JCW (2021) A graph-based CNN-LSTM stock price prediction algorithm with leading indicators. Multimedia Syst. https://doi.org/10.1007/s00530-021-00758-w

    Article  Google Scholar 

  • Xu S, Xiang Y (2021) Frog-GNN: multi-perspective aggregation based graph neural network for few-shot text classification. Expert Syst Appl 176:114795

    Article  Google Scholar 

  • Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z et al (2020) Graph neural networks: a review of methods and applications. AI Open 1:57–81

    Article  Google Scholar 

  • Zucco C, Calabrese B, Agapito G, Guzzi PH, Cannataro M (2020) Sentiment analysis for mining texts and social networks data: methods and tools. Wiley Interdiscip Rev Data Min Knowl Discov 10(1):e1333

    Article  Google Scholar 

  • Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: ICLR

  • Blunsom P, Grefenstette E, Kalchbrenner N (2014) A convolutional neural network for modelling sentences. In: ACL

  • Conneau A, Schwenk H, Cun YL, Barrault L (2017) Very deep convolutional networks for text classification. In: ACL

  • Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

  • Ding K, Wang J, Li J, Li D, Liu H (2020) Be more with less: hypergraph attention networks for inductive text classification. In: EMNLP

  • Fei H, Tan S, Li P (2019) Hierarchical multi-task word embedding learning for synonym prediction. In: SIGKDD

  • Hamilton WL, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. In: NIPS

  • Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. https://arXiv.org/1801.06146

  • Hu L, Xu S, Li C, Yang C, Shi C, Duan N et al (2020b) Graph neural news recommendation with unsupervised preference disentanglement. In: ACL

  • Huang L, Ma D, Li S, Zhang X, Wang H (2019) Text Level Graph Neural Network for Text Classification. In: EMNLP-IJCNLP

  • Kim Y (2014) Convolutional neural networks for sentence classification. In: EMNLP

  • Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. In: ICLR

  • Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International Conference on Machine Learning, PMLR

  • Li Q, Peng H, Li J, Xia C, Yang R, Sun L et al (2020) A survey on text classification: from shallow to deep learning. https://arXiv.org/2008.00364

  • Liu P, Qiu X, Chen X, Wu S, Huang XJ (2015) Multi-timescale long short-term memory neural network for modelling sentences and documents. In: EMNLP

  • Liu P, Qiu X, Huang X (2016) Recurrent neural network for text classification with multi-task learning. In: IJCAI

  • Liu X, You X, Zhang X, Wu J, Lv P (2020) Tensor graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence

  • Liu Y, Guan R, Giunchiglia F, Liang Y, Feng X (2021) Deep attention diffusion graph neural networks for text classification. In: EMNLP

  • Luong MT, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: EMNLP

  • Manning CD, Surdeanu M, Bauer J, Finkel JR, Bethard S, McClosky D (2014) The Stanford CoreNLP natural language processing toolkit. In: ACL

  • Mao R, Lin C, Guerin F (2018) Word embedding and wordnet based metaphor identification and interpretation. In: ACL

  • Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. In: ICRL

  • Mikolov T, Grave É, Bojanowski P, Puhrsch C, Joulin A (2018) Advances in pre-training distributed word representations. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation

  • Nguyen D, Luo W, Nguyen TD, Venkatesh S, Phung D (2018a) Learning graph representation via frequent subgraphs. In: ICDM

  • Nguyen D, Luo W, Nguyen TD, Venkatesh S, Phung D (2018b) Sqn2vec: Learning sequence representation via sequential patterns with a gap constraint. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases

  • Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: EMNLP

  • Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. https://arXiv.org/1802.05365

  • Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training. In: OpenAI

  • Rousseau F, Kiagias E, Vazirgiannis M (2015) Text categorization as a graph classification problem. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing

  • Sonawane SS, Kulkarni PA (2014) Graph based representation and analysis of text document: a survey of techniques. Int J Comput Appl 96(19)

  • Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: NIPS

  • Tai KS, Socher R, Manning CD (2015) Improved semantic representations from tree-structured long short-term memory networks. In: ACL

  • Vaswani A et al (2017) Attention is all you need. In: NIPS

  • Veličković P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y (2018) Graph attention networks. In: ICLR

  • Vu T, Wang T, Munkhdalai T, Sordoni A, Trischler A, Mattarella-Micke A et al (2020) Exploring and predicting transferability across NLP tasks. In: EMNLP

  • Wang Z, Wang C, Zhang H, Duan Z, Zhou M, Chen B (2020) Learning dynamic hierarchical topic graph with graph convolutional network for document classification. In: International Conference on Artificial Intelligence and Statistics (PMLR)

  • Xie Q, Huang J, Du P, Peng M, Nie JY (2021) Inductive topic variational graph auto-encoder for text classification. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

  • Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence

  • Zhang H, Zhang J (2020) Text graph transformer for document classification. In: EMNLP

  • Zhang Y, Yu X, Cui Z, Wu S, Wen Z, Wang L (2020) Every document owns its structure: inductive text classification via graph neural networks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

  • Zhu X, Sobihani P, Guo H (2015) Long short-term memory over recursive structures. In: PMLR

Download references

Author information

Authors and Affiliations

  1. Faculty of Information Technology, HUTECH University, Ho Chi Minh City, Vietnam

    Phu Pham & Bay Vo

  2. School of Computer Science and Engineering, International University, Ho Chi Minh City, Vietnam

    Loan T. T. Nguyen

  3. Vietnam National University, Ho Chi Minh City, Vietnam

    Loan T. T. Nguyen

  4. Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, T6R 2V4, Canada

    Witold Pedrycz

  5. Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

    Witold Pedrycz

  6. Faculty of Engineering, King Abdulaziz University, Jeddah, 21589, Saudi Arabia

    Witold Pedrycz

Authors
  1. Phu Pham
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Loan T. T. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Witold Pedrycz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bay Vo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bay Vo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pham, P., Nguyen, L.T.T., Pedrycz, W. et al. Deep learning, graph-based text representation and classification: a survey, perspectives and challenges. Artif Intell Rev 56, 4893–4927 (2023). https://doi.org/10.1007/s10462-022-10265-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-022-10265-7

Keywords

Search

Navigation