Abstract
In recent years, machine learning technology has made great success in fields of computer vision, natural language processing, speech recognition and so on. And machine learning model has been widely used in face recognition, automatic driving, malware detection, intelligent medical analysis and other practical tasks. In this paper, attention mechanism is proposed to combine with LSTM model to extract features in text classification. The results show that, on the one hand, LSTM + Attention can improve classification performance; on the other hand, by sorting by word weights generated by the attention layer, we find some meaningful word features, however, its recognition performance is not good. Some possible reasons were analyzed and it was found that attention mechanism sometimes misjudges wrong word features, resulting from these wrong words often appearing at the same time with meaningful word features.
This work is supported by National Defense Science and Technology Innovation Special Zone Project (No. 18-163-11-ZT-002-045-04).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Ibrahim, M., et al.: Global explanations of neural networks. In: AAAI/ACM Conference ACM (2019)
Li, J., Ji, S., Du, T., et al.: Text bugger: generating adversarial text against real-world applications. In: Proceedings of the 26th Annual Network and Distributed Systems Security Symposium. ISOC, Reston (2019)
Doshi-Velez, F., Kim, B.: Towards A Rigorous Science of Interpretable Machine Learning (2017)
Riccardo, G., et al.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5), 1–42 (2018)
Sebastian, B., et al.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), e0130140 (2015)
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why Should I Trust You?”: Explaining the Predictions of Any Classifier (2016)
Vaswani, A., Shazer, N., Parmar, N., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc, Red Hook (2017)
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: Computer EnCE, pp. 2048–2057 (2015)
Arras, L., et al.: “What is relevant in a text document?”: an interpretable machine learning approach. PLoS ONE 12(8), e0181142 (2017)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Yang, Z., et al.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016)
Poerner, N., Roth, B., Schütze, H.: Evaluating neural network explanation methods using hybrid documents and morphological agreement (2018)
Malhotra, P., et al.: Long short term memory networks for anomaly detection in time series. In: European Symposium on Artificial Neural Networks (2015)
Deng, X., et al.: Feature selection for text classification: a review. Multimedia Tools Appl. 78(3), 3797–3816 (2019)
Kowsari, et al.: Text classification algorithms: a survey. Information 10(4) (2019)
Xie, J., et al.: Chinese text classification based on attention mechanism and feature-enhanced fusion neural network. Computing 102(3), 683–700 (2019). https://doi.org/10.1007/s00607-019-00766-9
Liu, G., Guo, J.: Bidirectional LSTM with attention mechanism and convolutional layer for text classification. Neurocomputing 337, 325–338 (2019)
Tong, X., et al.: Semantic correlations loss: improving model interpretability for multi-class classification. In: 2019 IEEE International Conference on Big Data (Big Data). IEEE (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, Y., Wang, Y. (2020). Feature Extraction by Using Attention Mechanism in Text Classification. In: Qin, P., Wang, H., Sun, G., Lu, Z. (eds) Data Science. ICPCSEE 2020. Communications in Computer and Information Science, vol 1258. Springer, Singapore. https://doi.org/10.1007/978-981-15-7984-4_6
Download citation
DOI: https://doi.org/10.1007/978-981-15-7984-4_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-7983-7
Online ISBN: 978-981-15-7984-4
eBook Packages: Computer ScienceComputer Science (R0)