research-article

The Study on the Text Classification Based on Graph Convolutional Network and BiLSTM

Authors:

Wenjun ZhuAuthors Info & Claims

ICCAI '22: Proceedings of the 8th International Conference on Computing and Artificial Intelligence

Pages 323 - 331

https://doi.org/10.1145/3532213.3532261

Published: 13 July 2022 Publication History

Abstract

Recently, Graph Convolutional Neural Network (GCN) is widely used in text classification tasks. And it has been effectively used to accomplish tasks that are thought to have a rich relational structure. However, due to the sparse adjacency matrix constructed by GCN, GCN cannot make full use of context-dependent information in text classification, and it is not good at capturing local information. The Bidirectional Encoder Representation from Transformers (BERT) has the ability to capture contextual information in sentences or documents, but it is limited in capturing global information about vocabulary in a language, which is the advantage of GCN. Therefore, this paper proposes an improved model named Improved Mutual Graph Convolution Networks (IMGCN) to solve the above problems. The original GCN uses word co-occurrence relationships to build text graphs. Word connections are not rich enough and cannot capture context dependencies well, so we introduce semantic dictionary (WordNet) and dependencies. While the model enhances the ability to capture contextual dependencies, it lacks the ability to capture sequences. Therefore, we introduced BERT and Bi-directional Long Short-Term Memory (BiLSTM) Network to perform deeper learning on the features of text, thereby improving the classification effect of the model. The experimental results show that our model is more effective than previous research reports on four text classification datasets.

References

[1]

Joachims, T. (1998). Text Categorization with Support Vector Machines: Learning with Many Relevant Features. ECML.

Digital Library

[2]

Alhajj, R., Gao, H., Li, X., Li, J., & Zaiane, O.R. (2007). Advanced Data Mining and Applications, Third International Conference, ADMA 2007, Harbin, China, August 6-8, 2007, Proceedings. ADMA.

[3]

Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. EMNLP.

[4]

Zhang, H., Xiao, L., Wang, Y., & Jin, Y. (2017). A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning. IJCAI.

[5]

Zhao, W., Peng, H., Eger, S., Cambria, E., & Yang, M. (2019). Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications. ACL.

[6]

Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9, 1735-1780.

Digital Library

[7]

Cho, K., Merrienboer, B.V., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP.

[8]

Wang, R., Li, Z., Cao, J., Chen, T., & Wang, L. (2019). Convolutional Recurrent Neural Networks for Text Classification. 2019 International Joint Conference on Neural Networks (IJCNN), 1-6.

[9]

Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017). Attention is All you Need. ArXiv, abs/1706.03762.

[10]

Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL.

[11]

Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V.F., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gülçehre, Ç., Song, H.F., Ballard, A.J., Gilmer, J., Dahl, G.E., Vaswani, A., Allen, K.R., Nash, C., Langston, V., Dyer, C., Heess, N.M., Wierstra, D., Kohli, P., Botvinick, M.M., Vinyals, O., Li, Y., & Pascanu, R. (2018). Relational inductive biases, deep learning, and graph networks. ArXiv, abs/1806.01261.

[12]

Kipf, T., & Welling, M. (2017). Semi-Supervised Classification with Graph Convolutional Networks. ArXiv, abs/1609.02907.

[13]

Yao, L., Mao, C., & Luo, Y. (2019). Graph Convolutional Networks for Text Classification. ArXiv, abs/1809.05679.

[14]

Zhenbo, B., Shiyou, Z., Hongjun, P., Yuanhong, W., & Hua, Y. (2021). A Survey of Preprocessing Methods for Marine Ship Target Detection Based on Video Surveillance. 2021 7th International Conference on Computing and Artificial Intelligence.

Digital Library

[15]

Lu, Z., Du, P., & Nie, J. (2020). VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification. Advances in Information Retrieval, 12035, 369 - 382.

Digital Library

[16]

Xue, B., Zhu, C., Wang, X., & Zhu, W. (2021). An Integration Model for Text Classification using Graph Convolutional Network and BERT. Journal of Physics: Conference Series, 2137.

[17]

Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E.H. (2016). Hierarchical Attention Networks for Document Classification. NAACL.

[18]

Harris, Z.S. (1981). Distributional Structure.

[19]

Rousseau, F., Kiagias, E., & Vazirgiannis, M. (2015). Text Categorization as a Graph Classification Problem. ACL.

[20]

Luo, Y., Uzuner, Ö., & Szolovits, P. (2017). Bridging semantics and syntax with graph algorithms - state-of-the-art of extracting biomedical relations. Briefings in bioinformatics, 18 1, 160-178 .

[21]

Tang, D., Qin, B., & Liu, T. (2015). Document Modeling with Gated Recurrent Neural Network for Sentiment Classification. EMNLP.

[22]

Wang, Y., Huang, M., Zhu, X., & Zhao, L. (2016). Attention-based LSTM for Aspect-level Sentiment Classification. EMNLP.

[23]

Dong, Y., Liu, P., Zhu, Z., Wang, Q., & Zhang, Q. (2020). A Fusion Model-Based Label Embedding and Self-Interaction Attention for Text Classification. IEEE Access, 8, 30548-30559.

[24]

Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio’, P., & Bengio, Y. (2018). Graph Attention Networks. ArXiv, abs/1710.10903.

[25]

Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., & Bengio, Y. (2015). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. ICML.

[26]

Cavallari, S., Cambria, E., Cai, H., Chang, K.C., & Zheng, V.W. (2019). Embedding Both Finite and Infinite Communities on Graphs [Application Notes]. IEEE Comput. Intell. Mag., 14, 39-50.

[27]

Bouma, G. (2009). Normalized (pointwise) mutual information in collocation extraction.

[28]

Wu, Z., & Palmer, M. (1994). Verb Semantics and Lexical Selection. ArXiv, abs/cmp-lg/9406033.

[29]

Zhu, X., Xu, Q., Chen, Y., Chen, H., & Wu, T. (2020). A Novel Class-Center Vector Model for Text Classification Using Dependencies and a Semantic Dictionary. IEEE Access, 8, 24990-25000.

[30]

Tesnière, L. (1959). Éléments de syntaxe structurale.

[31]

Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., & Potts, C. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP.

[32]

Pang, B., & Lee, L. (2005). Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales. ACL.

[33]

Tang, J., Qu, M., & Mei, Q. (2015). PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

Digital Library

[34]

Warstadt, A., Singh, A., & Bowman, S.R. (2019). Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics, 7, 625-641.

[35]

Graves, A., Mohamed, A., & Hinton, G.E. (2013). Speech recognition with deep recurrent neural networks. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 6645-6649.

Cited By

Zengeya TVincent Fonou-Dombeu J(2024)A Review of State of the Art Deep Learning Models for Ontology ConstructionIEEE Access10.1109/ACCESS.2024.340642612(82354-82383)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3406426
Yelmen IGunes AZontul M(2023)Multi-Class Document Classification Using Lexical Ontology-Based Deep LearningApplied Sciences10.3390/app1310613913:10(6139)Online publication date: 17-May-2023
https://doi.org/10.3390/app13106139
Chen W(2023)Deep adversarial neural network model based on information fusion for music sentiment analysisComputer Science and Information Systems10.2298/CSIS221212031C20:4(1797-1817)Online publication date: 2023
https://doi.org/10.2298/CSIS221212031C
Show More Cited By

The Study on the Text Classification Based on Graph Convolutional Network and BiLSTM
1. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
    1. Machine learning approaches

Recommendations

Simplified-Boosting Ensemble Convolutional Network for Text Classification
Abstract
Graph convolutional network (GCN) has a strong ability to extract the global feature but neglects the order of the words, thus leading to its weak effect on short text classification. In contrast, convolutional neural network (CNN) can capture the ...
Local discriminative graph convolutional networks for text classification
Abstract
Recently, graph convolutional networks (GCNs) has demonstrated great success in the text classification. However, the GCN only focuses on the fitness between the ground-truth labels and the predicted ones. Indeed, it ignores the local intra-class ...
A Joint Model for Text Classification with BERT-BiLSTM and GCN
AIPR '22: Proceedings of the 2022 5th International Conference on Artificial Intelligence and Pattern Recognition

With the development of Graph Neural Network (GNN), a lot of GNN-based methods have been proposed in text classification. However, GNNs are difficult to capture the word relationship of the context in the document sequence. In addition, these models are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICCAI '22: Proceedings of the 8th International Conference on Computing and Artificial Intelligence

March 2022

809 pages

ISBN:9781450396110

DOI:10.1145/3532213

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Beijing University of Technology

Conference

ICCAI '22

ICCAI '22: 2022 8th International Conference on Computing and Artificial Intelligence

March 18 - 21, 2022

Tianjin, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
151
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zengeya TVincent Fonou-Dombeu J(2024)A Review of State of the Art Deep Learning Models for Ontology ConstructionIEEE Access10.1109/ACCESS.2024.340642612(82354-82383)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3406426
Yelmen IGunes AZontul M(2023)Multi-Class Document Classification Using Lexical Ontology-Based Deep LearningApplied Sciences10.3390/app1310613913:10(6139)Online publication date: 17-May-2023
https://doi.org/10.3390/app13106139
Chen W(2023)Deep adversarial neural network model based on information fusion for music sentiment analysisComputer Science and Information Systems10.2298/CSIS221212031C20:4(1797-1817)Online publication date: 2023
https://doi.org/10.2298/CSIS221212031C
Ali N(2023)Early Rheumatoid Arthritis Detection by miRNA Data Analysis Using a Hybrid CNN-LSTM Deep Learning Model2023 Intelligent Methods, Systems, and Applications (IMSA)10.1109/IMSA58542.2023.10217733(458-463)Online publication date: 15-Jul-2023
https://doi.org/10.1109/IMSA58542.2023.10217733
Guo QChen XZhou PLiao Y(2023)Cross-Domain Data Extraction and Knowledge Graph Construction for Dispute Analysis2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS57875.2023.00109(959-960)Online publication date: Jul-2023
https://doi.org/10.1109/ICDCS57875.2023.00109
Liu BGuan WYang CFang ZLu Z(2023)Transformer and Graph Convolutional Network for Text ClassificationInternational Journal of Computational Intelligence Systems10.1007/s44196-023-00337-z16:1Online publication date: 4-Oct-2023
https://doi.org/10.1007/s44196-023-00337-z
Ali NShaheen MMabrouk MRizka M(2022)Multiple Sclerosis Biomarkers Detection by a BiLSTM Deep Learning Model for miRNA Data Analysis2022 International Arab Conference on Information Technology (ACIT)10.1109/ACIT57182.2022.9994197(1-6)Online publication date: 22-Nov-2022
https://doi.org/10.1109/ACIT57182.2022.9994197

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten