Dongxue Bao
ABSTRACT
Aiming at the large amount of news and review text data, sparse features, and the inability of traditional text feature representation to dynamically obtain grammatical structure, semantic information, and multi-dimensional rich feature representation of entity phrases. This paper proposes to obtain more generalized knowledge semantic feature information such as rich context phrases, entity words and so on by integrating knowledge enhanced semantic representation (Enhanced Representation Through Knowledge Integration, ERNIE). The pre-trained language model ERNIE hides words and entities by random Semantic unit prediction context realizes word vector language representation, and the output vector representation of ERNIE is input to BiLSTM, Attention mechanism and DPCNN network model to generate high-order text feature vectors, and each channel vector is processed by BatchNormalization and ReLU activation functions respectively.Thus, the semantic description information of the multi-channel word vector is fused. The model proposed in this paper can not only improve the training speed and prevent overfitting, but also enhance the feature information such as semantics and grammatical structure, thereby improving the text classification effect. By comparing the two datasets with other improved ERNIE models in terms of accuracy, precision, recall, and F1 value, the experimental results show that the model proposed in this paper can obtain multi-dimensional rich semantic grammatical structure features for text classification, and then improve the Text classification effect.
- Li M, Wang L. A survey on personalized news recommendation technology[J]. IEEE Access, 2019, 7: 145861-145879.Google Scholar
Cross Ref
- Mitra B, Craswell N. Neural models for information retrieval[J]. arXiv preprint arXiv:1705.01509, 2017.Google Scholar
- Shrawan Kumar Trivedi,Prabin Kumar Panigrahi. Spam classification: a comparative analysis of different boosted decision tree approaches[J]. Journal of Systems and Information Technology,2018,20(3):Google Scholar
- Mukherjee S. Sentiment analysis[M]//ML. NET Revealed. Apress, Berkeley, CA, 2021: 113-127.Google Scholar
- Shah K, Patel H, Sanghvi D, A comparative analysis of logistic regression, random forest and KNN models for the text classification[J]. Augmented Human Research, 2020, 5(1): 1-16.Google ScholarCross Ref
- Ruan S, Li H, Li C, Class-specific deep feature weighting for Naïve Bayes text classifiers[J]. IEEE Access, 2020, 8: 20151-20159.Google ScholarCross Ref
- Y. Bengio,R. Ducharme,P. Vincent. A Neural Probabilistic Language Model[J] Journal of Machine Learning Research,2003,3:1137- 1155.Google Scholar
- Mikolov T, Chen K, Corrado G, et al.Efficient estimation of word representations in vector space [EB/OL] ( 2013-09-07) [2017-05-08] https: / /arxiv.org /abs/1301.3781.Google Scholar
- Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words and phrases and their compositionality [C ]/ /MIT. Advances in Neural Information Processing Systems. Massachusetts: MIT Press, 2013: 3111-3119.Google Scholar
- PENNINGTON J, SOCHER R, MANNING C D. Glove: global vectors for word representation[C]//Proceedings of the 2014 conference on empirical methods in natural language processing, Stroudsburg, PA: Association for Computational Linguistics, 2014.Google Scholar
- PETERS M, NEUMANN M, IYYER M, Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA: Association for Computational Linguistics, 2018.Google Scholar
- VASWANI A, SHAZEER N, PARMAR N, Attention is all you need[C]//Proceedings of the 2017 conference on neural information processing systems, Stroudsburg, PA: Association for Computational Linguistics,2017, 5998-6008.Google Scholar
- DEVLIN J, CHANG M W, LEE K, BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA: Association for Computational Linguistics,2019.Google Scholar
- Sun Y, Wang S, Li Y, Ernie: Enhanced representation through knowledge integration[J]. arXiv preprint arXiv:1904.09223, 2019.Google Scholar
- KIM Y.Convolutional neural networks for sentence classification [J]. arXiv preprint arXiv: 1408.5882, 2014.Google Scholar
- Jordan M I. Serial order: A parallel distributed processing approach[M]//Advances in psychology. North-Holland, 1997, 121: 471-495.Google Scholar
- Liu P, Qiu X, Huang X. Recurrent neural network for text classification with multi-task learning[J]. arXiv preprint arXiv:1605.05101, 2016.Google Scholar
- Chen Jie, Ma Jing, Li Xiaofeng. A short text classification method incorporating text features of pre-training models [J]. Data Analysis and Knowledge Discovery, 2021,5(09):21-30.Google Scholar
- Shuwei X, Xuyang G, Ying W. A Study on the EDA-based Classification of News Text[C]//Journal of Physics: Conference Series. IOP Publishing, 2021, 1792(1): 012080.Google ScholarCross Ref
- Guo H, Chi C, Zhan X. ERNIE-BiLSTM Based Chinese Text Sentiment Classification Method[C]//2021 International Conference on Computer Engineering and Application (ICCEA). IEEE, 2021: 84-88.Google Scholar
- Li J, Cao H. Research on Dual Channel News Headline Classification Based on ERNIE Pre-training Model[J]. arXiv preprint arXiv:2202.06600, 2022.Google Scholar
- Lai S, Xu L, Liu K, Recurrent convolutional neural networks for text classification[C]//Twenty-ninth AAAI conference on artificial intelligence. 2015.Google Scholar
- Johnson R, Zhang T. Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017: 562-570.Google Scholar
Recommendations
Multi-prototype Morpheme Embedding for Text Classification
SMA 2020: The 9th International Conference on Smart Media and ApplicationsRepresenting a word into a continuous space, also known as a word vector, has been successful in various NLP tasks. The word-based embedding has two problems; one is the out-of-vocabulary problem and the other is does not take into account the context ...
Urdu text classification
FIT '09: Proceedings of the 7th International Conference on Frontiers of Information TechnologyThis paper compares statistical techniques for text classification using Naïve Bayes and Support Vector Machines, in context of Urdu language. A large corpus is used for training and testing purpose of the classifiers. However, those classifiers cannot ...
Persian text classification based on K-NN using wordnet
IEA/AIE'12: Proceedings of the 25th international conference on Industrial Engineering and Other Applications of Applied Intelligent Systems: advanced research in applied artificial intelligenceK-NN is widely used for text classification purpose. Basic K-NN has poor accuracy; other methods should be applied to basic K-NN to improve accuracy and efficiency. In this paper we propose a method that uses wordnet to increase similarity of documents ...
Comments