Abstract
This paper presents a lightweight model based on the self-attention mechanism for text classification tasks. In our model, we incorporate auxiliary information of the label through the label embedding method, enabling the model to capture the contextual language variations of the same word. Furthermore, we address the issue of misclassification of similar texts by introducing the contrastive loss function, in conjunction with the traditional cross-entropy loss function. Experimental evaluations are conducted on multiple datasets, comparing our model against others with similar parameter scales, thus demonstrating the effectiveness of the proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for image classification. IEEE Trans. Pattern Anal. Mach. Intell. 38(7), 1425–1438 (2016). https://doi.org/10.1109/TPAMI.2015.2487986
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling (2018)
Johnson, R., Zhang, T.: Deep pyramid convolutional neural networks for text categorization. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 562–570. Association for Computational Linguistics, Vancouver, Canada (2017). https://doi.org/10.18653/v1/P17-1052
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746–1751. Association for Computational Linguistics, Doha, Qatar (2014). https://doi.org/10.3115/v1/D14-1181
Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29. no. 1 (2015). https://doi.org/10.1609/aaai.v29i1.9513
Liu, C.Z., Sheng, Y.x., Wei, Z.Q., Yang, Y.Q.: Research of text classification based on improved TF-IDF algorithm. In: 2018 IEEE International Conference of Intelligent Robotic and Control Engineering (IRCE), pp. 218–222 (2018). https://doi.org/10.1109/IRCE.2018.8492945
Liu, M., Liu, L., Cao, J., Du, Q.: Co-attention network with label embedding for text classification. Neurocomputing 471, 61–69 (2022). https://doi.org/10.1016/j.neucom.2021.10.099
Musgrave, K., Belongie, S., Lim, S.-N.: A metric learning reality check. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 681–699. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_41
Peng, S., et al.: A survey on deep learning for textual emotion analysis in social networks. Digital Commun. Netw. 8(5), 745–762 (2022). https://doi.org/10.1016/j.dcan.2021.10.003
Schick, T., Schütze, H.: Exploiting cloze-questions for few-shot text classification and natural language inference. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255–269. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.eacl-main.20. https://aclanthology.org/2021.eacl-main.20
Song, P., Geng, C., Li, Z.: Research on text classification based on convolutional neural network. In: 2019 International Conference on Computer Network, Electronic and Automation (ICCNEA), pp. 229–232. IEEE, Xi’an, China (2019). https://doi.org/10.1109/ICCNEA.2019.00052
Tan, C., Ren, Y., Wang, C.: An adaptive convolution with label embedding for text classification. Appl. Intell. 53(1), 804–812 (2023). https://doi.org/10.1007/s10489-021-02702-x
Tian, H., Wu, L.: Microblog emotional analysis based on TF-IWF weighted word2vec model. In: 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), pp. 893–896. IEEE, Beijing, China (2018). https://doi.org/10.1109/ICSESS.2018.8663837
Tu Zhenchao, M.J.: Item categorization algorithm based on improved text representation. Data Anal. Knowl. Discovery 6, 34–43 (2022). https://doi.org/10.11925/infotech.2096-3467.2021.0958
Vaswani, A., et al.: Attention is all you need (2017)
Wang, G., et al.: Joint embedding of words and labels for text classification (2018)
Wang, Y., Huang, M., Zhu, X., Zhao, L.: Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 606–615. Association for Computational Linguistics, Austin, Texas (2016). https://doi.org/10.18653/v1/D16-1058
Xu Yuemei, Fan Zuwei, C.H.: A multi-task text classification model based on label embedding of attention mechanism. Data Anal. Knowl. Discovery, 6(2/3) pp. 105–116 (2022). https://doi.org/10.11925/infotech.2096-3467.2021.0912
XUAN, W.: Logistics service quality sentiment analysis with deeper attention lstm model with aspect embedding. Tehnicki vjesnik - Technical Gazette 30(2), 634–641 (2023). https://doi.org/10.17559/TV-20221018031450
Yan, C., Liu, J., Liu, W., Liu, X.: Research on public opinion sentiment classification based on attention parallel dual-channel deep learning hybrid model. Eng. Appl. Artif. Intell. 116, 105448 (2022). https://doi.org/10.1016/j.engappai.2022.105448
Yao, L., Mao, C., Luo, Y.: Graph convolutional networks for text classification. CoRR abs/1809.05679 (2018). http://arxiv.org/abs/1809.05679
Zhao, W., Zhu, L., Wang, M., Zhang, X., Zhang, J.: WTL-CNN: a news text classification method of convolutional neural network based on weighted word embedding. Connect. Sci. 34(1), 2291–2312 (2022). https://doi.org/10.1080/09540091.2022.2117274
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, F., Chen, G., Yi, J.W., Luo, G. (2024). A Lightweight Text Classification Model Based on Label Embedding Attentive Mechanism. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1966. Springer, Singapore. https://doi.org/10.1007/978-981-99-8148-9_46
Download citation
DOI: https://doi.org/10.1007/978-981-99-8148-9_46
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8147-2
Online ISBN: 978-981-99-8148-9
eBook Packages: Computer ScienceComputer Science (R0)