skip to main content
10.1145/3507548.3507574acmotherconferencesArticle/Chapter ViewAbstractPublication PagescsaiConference Proceedingsconference-collections
research-article

Short Text Classification Model Based on BERT and Fusion Network

Published: 09 March 2022 Publication History

Abstract

Abstract: Aiming at short texts lacking contextual information, large amount of text data, sparse features, and traditional text feature representations that cannot dynamically obtain the key classification information of a word polysemous and contextual semantics. this paper proposes a pre-trained language model based on BERT. The network model B-BAtt-MPC (BERT-BiLSTM-Attention-Max-Pooling-Concat) that integrates BiLSTM, Attention mechanism and Max-Pooling mechanism. Firstly, obtain multi-dimensional and rich feature information such as text context semantics, grammar, and context through the BERT model; Secondly, use the BERT output vector to obtain the most important feature information worth noting through the BiLSTM, Attention layer and Max-Pooling layer;  In order to optimize the classification model, the BERT and BiLSTM output vectors are fused and input into Max-Pooling; Finally, the classification results are obtained by fusing two feature vectors with Max-Pooling. The experimental results of two data sets show that the model proposed in this paper can obtain the importance and key rich semantic features of short text classification, and can improve the text classification effect.

References

[1]
Shah K, Patel H, Sanghvi D, A comparative analysis of logistic regression, random forest and KNN models for the text classification[J]. Augmented Human Research, 2020, 5(1): 1-16.
[2]
Ruan S, Li H, Li C, Class-specific deep feature weighting for Naïve Bayes text classifiers[J]. IEEE Access, 2020, 8: 20151-20159.
[3]
Y. Bengio,R. Ducharme,P. Vincent. A Neural Probabilistic Language Model[J] Journal of Machine Learning Research,2003,3:1137- 1155.
[4]
Mikolov T, Chen K, Corrado G, .Efficient estimation of word representations in vector space[EB/OL]. (2013- 09-07) [2017-05-08]. https: //arxiv.org /abs/1301.3781.
[5]
Mikolov T, Sutskever I, Chen K, . Distributed representations of words and phrases and their compositionality [C ]/ /MIT. Advances in Neural Information Processing Systems. Massachusetts: MIT Press, 2013: 3111-3119.
[6]
PENNINGTON J, SOCHER R, MANNING C D. Glove: global vectors for word representation[C]//Proceedings of the 2014 conference on empirical methods in natural language processing, Stroudsburg, PA: Association for Computational Linguistics, 2014,
[7]
VASWANI A, SHAZEER N, PARMAR N, Attention is all you need[C]//Proceedings of the 2017 conference on neural information processing systems, Stroudsburg, PA: Association for Computational Linguistics,2017, 5998-6008.
[8]
DEVLIN J, CHANG M W, LEE K, BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA: Association for Computational Linguistics,2019:
[9]
KIM Y.Convolutional neural networks for sentence classification[J]. arXiv preprint arXiv: 1408.5882, 2014.
[10]
Jordan M I. Serial order: A parallel distributed processing approach[M]//Advances in psychology. North-Holland, 1997, 121: 471-495.
[11]
Tai K S, Socher R, Manning C D. Improved semantic representations from tree-structured long short-term memory networks[J]. arXiv preprint arXiv:1503.00075, 2015.
[12]
Mnih V, Heess N, Graves A. Recurrent models of visual attention[C]//Advances in neural information processing systems. 2014: 2204-2212.
[13]
Cai L, Song Y, Liu T, A hybrid BERT model that incorporates label semantics via adjustive attention for multi-label text classification[J]. Ieee Access, 2020, 8: 152183-152192.
[14]
Joulin A, Grave E, Bojanowski P, Bag of tricks for efficient text classification[C] // Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain, April, 2016: 427–431.
[15]
Lai S, Xu L, Liu K, Recurrent convolutional neural networks for text classification[C]//Twenty-ninth AAAI conference on artificial intelligence. 2015.
[16]
Aurangzeb Khan, Baharum Baharudin, Lam Hong Lee, and Khairullah khan, "A Review of Machine Learning Algorithms for Text-Documents Classification," Journal of Advances in Information Technology, Vol. 1, No. 1,pp. 4-20, February, 2010.
[17]
Ismaïl Biskri, Abdelghani Achouri, Louis Rompré, Steve Descoteaux and Boucif Amar Bensaber, "Computer-Assisted Reading: Getting Help from Text Classification and Maximal Association Rules," Journal of Advances in Information Technology, Vol. 4, No. 4, pp. 157-165, November, 2013.

Cited By

View all
  • (2022)A Generative Adversarial Net Assisted Method for User Intention Recognition on Imbalanced Dataset2022 IEEE International Conference on Knowledge Graph (ICKG)10.1109/ICKG55886.2022.00027(157-163)Online publication date: Nov-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
CSAI '21: Proceedings of the 2021 5th International Conference on Computer Science and Artificial Intelligence
December 2021
437 pages
ISBN:9781450384155
DOI:10.1145/3507548
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 March 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Attention
  2. BERT
  3. BiLSTM
  4. Max-Pooling
  5. Vector Fusion

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

CSAI 2021

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)29
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)A Generative Adversarial Net Assisted Method for User Intention Recognition on Imbalanced Dataset2022 IEEE International Conference on Knowledge Graph (ICKG)10.1109/ICKG55886.2022.00027(157-163)Online publication date: Nov-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media