Abstract
There are tens of thousands of unstructured textual data generated from social media platforms every day, as well as vast sensitive words that may spread harmful information and even insult with each other. Therefore, effectively identifying sensitive words on the network, can not only filter out these inappropriate remarks, but also build a healthy and clean network environment. Recently, there are many researchers devote themselves into the study of emotional vocabulary, just fewer studies draw attention to the identification of sensitive vocabulary on the network. Furthermore, traditional methods of vocabulary recognition need a lot of manpower to make rules and extract features. In this paper, we firstly construct the dataset of uncivilized language (ULN dataset), which is acquired through web crawler on the website. Secondly, we propose a method to identify Chinese sensitive words on the network, which combine the Bidirectional Encoder Representations from Transformers with Bidirectional Long Short Term Memory Network and Conditional Random Field (BERT-BiLSTM-CRF). We use three models to identify sensitive words in ULN dataset. The experimental results show that, the model proposed in this paper has excellent performance, compared with the classical Bidirectional Long Short Term Memory Network (BiLSTM-CRF) and Convolutional Neural Network (CNN).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aggarwal, P., Horsmann, T., Wojatzki, M., Zesch, T.: LTL-UDE at SemEval-2019 Task 6: BERT and two-vote classification for categorizing offensiveness. In: 2019 Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 678–682 (2019)
Altin, L.S.M., Serrano, À.B., Saggion, H.: LaSTUS/TALN at SemEval-2019 Task 6: identification and categorization of offensive language in social media with attention-based Bi-LSTM model. In: 2019 Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 672–677 (2019)
LI-Lin, F., Xiao-Dong, W.: An algorithm of Chinese keys matching used in filtering junk mail. J. HenNan Univ. Sci. Technol. (Nat. Sci.) 27(5), 35–37 (2006)
Dong, Z., Dong, Q.: HowNet-a hybrid language and knowledge resource. In: 2003 International Conference on Natural Language Processing and Knowledge Engineering, pp. 820–824. IEEE (2003)
Linhong, X., Hongfei, L., Yu, P., Hui, R., Jianmei, C.: Constructing the affective lexicon ontology. J. China Soc. Sci. Tech. Inf. 27(2), 180–185 (2008)
Liming: new words discovery research for specific areas. Nanjing University of Aeronautics and Astronautics (2012)
Sasano, R., Kurohashi, S., Okumura, M.: A simple approach to unknown word processing in Japanese morphological analysis. In: 2013 Proceedings of the Sixth International Joint Conference on Natural Language Processing, pp. 162–170 (2013)
Wenkun, L., Yangsen, Z., Ruoyu, C.: New word detection based on inner combination degree and boundary freedom degree of word. Appl. Res. Comput. 32(8), 2302–2304 (2015)
Su, Q., Liu, B.: Chinese new word extraction from MicroBlog data. In: 2013 International Conference on Machine Learning and Cybernetics, 2013. IEEE, pp. 1874–1879 (2013)
Xiaowen, X., Xiaodong, S., Hualin, Z.: Chinese named entity recognition system using statistics-based and rules-based method. Comput. Appl. 25(10), 2404–2406 (2005)
Habibi, M., Weber, L., Neves, M., Wiegandt, D.L., Leser, U.: Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 33(14), i37–i48 (2017)
Dong, C., Zhang, J., Zong, C., Hattori, M., Di, H.: Character-based LSTM-CRF with radical-level features for Chinese named entity recognition. In: Lin, C.-Y., Xue, N., Zhao, D., Huang, X., Feng, Y. (eds.) ICCPOL/NLPCC -2016. LNCS (LNAI), vol. 10102, pp. 239–250. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-50496-4_20
Goldberg, Y., Levy, O.: word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722 (2014)
Peters, M.E., et al.: Deep contextualized word representations (2018)
Vaswani, A., et al.: Attention is all you need. In: 2017 Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/researchcovers/languageunsupervised/languageunderstandingpaper.pdf (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2018)
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3(Feb), 1137–1155 (2003)
Lipton, Z.C., Berkowitz, J., Elkan, C.: A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019 (2015)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)
Dyer, C., Ballesteros, M., Ling, W., Matthews, A., Smith, N. A.: Transition-based dependency parsing with stack long short-term memory. arXiv preprint arXiv:1505.08075 (2015)
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
Prechelt, L.: Automatic early stopping using cross validation: quantifying the criteria. Neural Netw. 11(4), 761–767 (1998)
Chalapathy, R., Borzeshi, E.Z., Piccardi, M.: Bidirectional LSTM-CRF for clinical concept extraction (2016)
Zeng, D., Sun, C., Lin, L., Liu, B.: LSTM-CRF for drug-named entity recognition. Entropy-Switz. Entropy-Switz 19(6), 283 (2017)
Chen, T., Xu, R., He, Y., Wang, X.: Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Syst. Appl. 72, 221–230 (2017)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. P IEEE 86(11), 2278–2324 (1998)
Zha, S., Luisier, F., Andrews, W., Srivastava, N., Salakhutdinov, R.: Exploiting image-trained CNN architectures for unconstrained video classification. arXiv preprint arXiv:1503.04144 (2015)
Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
Acknowledgements
This research is supported by the National Language Commission Key Research Project (ZDI135-61), the National Natural Science Foundation of China (No.61532008 and 61872157), and the National Science Foundation of China (61572223).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yang, Y., Shen, X., Wang, Y. (2020). BERT-BiLSTM-CRF for Chinese Sensitive Vocabulary Recognition. In: Li, K., Li, W., Wang, H., Liu, Y. (eds) Artificial Intelligence Algorithms and Applications. ISICA 2019. Communications in Computer and Information Science, vol 1205. Springer, Singapore. https://doi.org/10.1007/978-981-15-5577-0_19
Download citation
DOI: https://doi.org/10.1007/978-981-15-5577-0_19
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-5576-3
Online ISBN: 978-981-15-5577-0
eBook Packages: Computer ScienceComputer Science (R0)