Abstract
Chinese short text classification is one of the increasingly significant tasks in Natural Language Processing (NLP). Different from documents and paragraphs, short text faces the problems of shortness, sparseness, non-standardization, etc., which brings enormous challenges for traditional classification methods. In this paper, we propose a novel model named BERT-KG, which can classify Chinese short text promptly and accurately and overcome the difficulty of short text classification. BERT-KG enriches short text features by obtaining background knowledge from the knowledge graph and further embeds the three-tuple information of the target entity into a BERT-based model. Then we fuse the dynamic word vector with the knowledge of the short text to form a feature vector for short text. And finally, the learned feature vector is input into the Softmax classifier to obtain a target label for short text. Extensive experiments conducted on two real-world datasets demonstrate that BERT-KG significantly improves the classification performance compared with state-of-the-art baselines.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Zhao, H., Liu, H.: Classification algorithm of Chinese short texts based on wikipedia. Libr. Inf. Serv. 57(11), 120–124 (2013)
Kilimci, Z.H., Omurca, S.I.: Extended feature spaces based classifier ensembles for sentiment analysis of short texts. Inf. Technol. Control 47(3), 457–470 (2018)
Bollegala, D., Mastsuo, Y., Lshizuka, M.: Measuring semantic similarity between words using web search engines. In: Proceedings of the 2nd ACM International Conference on World Wide Web, pp. 757–766. ACM (2007)
Peters, M.E., Neumann, M., Iyyer, M., et al.: Deep contextualized word representations. In: Proceedings of NAACL-HLT, pp. 2227–2237 (2018)
Radford, A., Narasimhan, K., Salimans, T., et al.: Improving language understanding by generative pre-training[EB/OL] (2018). https://s3-us-west2.amazonaws.com/openai-assets/researchcovers/languageunsupervised/languageunderstanding paper.pdf
Cui, Y., Che, W., Liu, T., et al.: Pre-training with whole word masking for chinese bert (2019). https://arxiv.org/abs/1906.08101
Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP, pp. 1746–1751 (2014)
Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances in Neural Information Processing Systems, pp. 649–657 (2015)
Lai, S., Xu, L., Liu, K., et al.: Recurrent convolutional neural networks for text classification. In: Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)
Lee, J.Y., Dernoncourt, F.: Sequential short-text classification with recurrent and convolutional neural networks (2016). https://arxiv.org/abs/1603.03827
Wu, W., Li, H., Wang, H., et al.: Probase: a probabilistic taxonomy for text understanding. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp. 481–492 (2012)
Devlin, J., Chang, M.W., Lee, K., et al.: Bert: pre-training of deep bidirectional transformers for language understanding (2018). https://arxiv.org/abs/1810.04805
Zhou, Y., Jiaming, X., Cao, J., Bo, X., Li, C., Bo, X.: Hybrid attention networks for chinese short text classification. Comput. Sist. 21(4), 759–769 (2018)
Amit, S.: Introducing the Knowledge Graph. Official Blog of Google, America (2012)
Bollacker, K., Cook, R., Tufts, P.: Freebase: a shared database of structured general human knowledge. In: Procedings of the 22nd AAAI Conf on Artificial Intelligence, Menlo Park, CA, pp. 1962–1963. AAAI (2007)
Bizer, C., Lehmann, J., Kobilarov, G., et al.: DBpedia-a crystallization point for the web of data. J. Web Semant. 7(3), 154–165 (2009)
Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a large ontology from wikipedia and wordnet. J. Web Semant. 6(3), 203–217 (2008)
Liu, W., Zhou, P., Zhao, Z., et al.: K-bert: enabling language representation with knowledge graph (2019). https://arxiv.org/abs/1909.07606
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhong, Y., Zhang, Z., Zhang, W., Zhu, J. (2021). BERT-KG: A Short Text Classification Model Based on Knowledge Graph and Deep Semantics. In: Wang, L., Feng, Y., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2021. Lecture Notes in Computer Science(), vol 13028. Springer, Cham. https://doi.org/10.1007/978-3-030-88480-2_58
Download citation
DOI: https://doi.org/10.1007/978-3-030-88480-2_58
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88479-6
Online ISBN: 978-3-030-88480-2
eBook Packages: Computer ScienceComputer Science (R0)
