Skip to main content
SpringerLink
Log in

Two-channel hierarchical attention mechanism model for short text classification

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Text classification plays an important role in information science. In order to address the issues of low classification efficiency, low accuracy, and incomplete text feature extraction in existing classification methods, this work offers a two-channel hierarchical attention mechanism short text classification model (TCHAM). First, a layered word vector attention mechanism is developed to improve the capture of keywords and phrases. Second, the TextBERT model is applied to train the word vector representation to solve the problem of multiple meanings of a word. Third, a two-channel neural network is utilized to achieve parallel acceleration. Finally, the output information of the two-channel neural network is fused to raise the accuracy of news text classification. The experimental results show that under the same environment and dataset, TCHAM increases the accuracy of text classification, reaching 98.03\(\%\) for the THUCNews dataset and 95.65\(\%\) for the SogouNews dataset, and its classification performance outperforms the comparison model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Notes

  1. https://thuctc.thunlp.org/.

  2. https://drive.google.com/uc?export=download &id=0Bz8a_Dbh9QhbUkVqNEszd0pHaFE.

References

  1. Aggarwal CC, Zhai C (2012) A survey of text classification algorithms. In: Mining text data, Springer, Boston, pp 163–222

  2. Joulin A, Grave E, Bojanowski P, Mikolov T (2016) Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759

  3. Kowsari K, Jafari Meimandi K, Heidarysafa M, Mendu S, Barnes L, Brown D (2019) Text classification algorithms: a survey. Information 10(4):150

    Article  Google Scholar 

  4. Church KW (2017) Word2Vec. Nat Lang Eng 23(1):155–162

    Article  Google Scholar 

  5. Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 1532–1543

  6. Joulin A, Grave E, Bojanowski P, Douze M, Jégou H, Mikolov T (2016) Fasttext. zip: compressing text classification models. arXiv preprint arXiv:1612.03651

  7. Minaee S, Kalchbrenner N, Cambria E, Nikzad N, Chenaghlu M, Gao J (2021) Deep learning-based text classification: a comprehensive review. ACM Comput Surv (CSUR) 54(3):1–40

    Article  Google Scholar 

  8. Liu J, Zheng S, Xu G, Lin M (2021) Cross-domain sentiment aware word embeddings for review sentiment analysis. Int J Mach Learn Cybern 12(2):343–354

    Article  Google Scholar 

  9. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  10. Vaswani A, Shazeer N, Parmar Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  11. Huiping C, Lidan W, Shukai D (2016) Sentiment classification model based on word embedding and CNN. Appl Res Comput 33(10):2902–2905

    Google Scholar 

  12. Liu C, Li X, Liu R, Fan X, Du L (2016) Chinese word segment based on character representation learning. J Comput Appl 36(10):2794

    Google Scholar 

  13. Moirangthem DS, Lee M (2021) Hierarchical and lateral multiple timescales gated recurrent units with pre-trained encoder for long text classification. Expert Syst Appl 165:113898

    Article  Google Scholar 

  14. Deng J, Cheng L, Wang Z (2021) Attention-based BiLSTM fused CNN with gating mechanism model for Chinese long text classification. Comput Speech Lang 68:101182

    Article  Google Scholar 

  15. Yu S, Liu D, Zhu W, Zhang Y, Zhao S (2020) Attention-based LSTM, GRU and CNN for short text classification. J Intell Fuzzy Syst 39(1):333–340

    Article  Google Scholar 

  16. Liu Jun, Li Wei, Chen Shuyu, Xu Guangxia (2022) PCA feature extraction algorithm based on anisotropic Gaussian kernel penalty, J Softw pp 1–16

  17. Pappas N, Popescu-Belis A (2017) Multilingual hierarchical attention networks for document classification. arXiv preprint arXiv:1707.00896

  18. Amin MZ, Nadeem N (2018) Convolutional neural network: text classification model for open domain question answering system. arXiv preprint arXiv:1809.02479

  19. Liu G, Guo J (2019) Bidirectional LSTM with attention mechanism and convolutional layer for text classification. Neurocomputing 337:325–338

    Article  Google Scholar 

  20. Wang H, Tian K, Wu Z, Wang L (2021) A short text classification method based on convolutional neural network and semantic extension. Int J Comput Intell Syst 14(1):367–375

    Article  Google Scholar 

  21. Xu J, Cai Y, Wu X, Lei X, Huang Q, Leung HF, Li Q (2020) Incorporating context-relevant concepts into convolutional neural networks for short text classification. Neurocomputing 386:42–53

    Article  Google Scholar 

  22. Liang Y, Li H, Guo B, Yu Z, Zheng X, Samtani S, Zeng DD (2021) Fusion of heterogeneous attention mechanisms in multi-view convolutional neural network for text classification. Inf Sci 548:295–312

    Article  Google Scholar 

  23. Gao W, Huang H (2021) A gating context-aware text classification model with BERT and graph convolutional networks. J Intell Fuzzy Syst 40(3):4331–4343

    Article  Google Scholar 

  24. Lin R, Fu C, Mao C, Wei J, Li J (2018) Academic news text classification model based on attention mechanism and RCNN. In: CCF Conference on Computer Supported Cooperative Work and Social Computing, Springer, Singapore, pp 507–516

  25. Tang Q, Chen J, Lu H, Du Y, Yang K (2019) Full attention-based bi-GRU neural network for news text classification. In: 2019 IEEE 5th International Conference on Computer and Communications (ICCC), IEEE pp 1970–1974

  26. Duan J, Zhao H, Qin W, Qiu M, Liu M (2020) News text classification based on MLCNN and BiGRU hybrid neural network. In: 2020 3rd International Conference on Smart BlockChain (SmartBlock), IEEE pp 1–6

  27. Ruan J, Caballero JM, Juanatas RA (2022) Chinese news text classification method based on attention mechanism. In: 2022 7th International Conference on Business and Industrial Research (ICBIR), IEEE pp 330–334

  28. Huang T, Zhang Q, Tang X, Zhao S, Lu X (2022) A novel fault diagnosis method based on CNN and LSTM and its application in fault diagnosis for complex systems. Artif Intell Rev 55(2):1289–1315

    Article  Google Scholar 

  29. Lai S, Xu L, Liu K, Zhao J (2015) Recurrent convolutional neural networks for text classification. In: Twenty-ninth AAAI Conference on Artificial Intelligence

  30. Zhang Y, Zheng J, Jiang Y, Huang G, Chen R (2019) A text sentiment classification modeling method based on coordinated CNN-LSTM-attention model. Chin J Electron 28(1):120–126

    Article  Google Scholar 

  31. Zheng S, Yang M (2019) A new method of improving BERT for text classification. In: International Conference on Intelligent Science and Big Data Engineering, Springer, Cham, pp 442–452

  32. Khandve SI, Wagh VK, Wani AD, Joshi IM, Joshi RB (2022) Hierarchical neural network approaches for long document classification. In: 2022 14th International Conference on Machine Learning and Computing (ICMLC) pp 115–119

Download references

Author information

Author notes

  1. Shiyang Hu and Haihui Huang have contributed equally to this study.

Authors and Affiliations

  1. School of Software Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Guanghui Chang, Shiyang Hu & Haihui Huang

Authors
  1. Guanghui Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiyang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haihui Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiyang Hu.

Ethics declarations

Conflict of interest

The author declares that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chang, G., Hu, S. & Huang, H. Two-channel hierarchical attention mechanism model for short text classification. J Supercomput 79, 6991–7013 (2023). https://doi.org/10.1007/s11227-022-04950-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04950-1

Keywords

Search

Navigation