Abstract
Commonsense can be vital in some applications like Natural Language Understanding, where it is often required to resolve ambiguity arising from implicit knowledge and under-specification. In spite of the remarkable success of neural network approaches on a variety of Natural Language Processing tasks, many of them struggle to react effectively in cases that require commonsense knowledge.
In the present research paper, we take advantage of the availability of the open multilingual knowledge graph ConceptNet, by using it as an additional external resource in a Named Entity Recognition system (NER). Our proposed architecture involves BiLSTM layers combined with a CRF layer that was augmented with some features such as pre-trained word embedding layers and dropout layers. Moreover, apart from using word representations, we used also character-based representation to capture the morphological and the orthographic information. Our experiments and evaluations showed an improvement in the overall performance with +2.86 in the F1-measure.
To the best of our knowledge, there is no study relating the integration of a commonsense knowledge base in NER.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amplayo, R.K., Lim, S., Hwang, S.: Entity commonsense representation for neural abstractive summarization. In: CoRR (2018)
Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: DBpedia: a nucleus for a web of open data. In: Aberer, K., et al. (eds.) ASWC/ISWC-2007. LNCS, vol. 4825, pp. 722–735. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-76298-0_52
Balahur, A., Hermida, J.-M., Montoyo, A.: Detecting implicit expressions of sentiment in text based on commonsense knowledge. In: Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, pp. 53–60. Association for Computational Linguistics (2011)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)
Chiu, J.-P.-C., Nichols, E.: Named entity recognition with bidirectional LSTM-CNNs. Trans. Assoc. Comput. Linguist. 4, 357–370 (2016)
Dyer, C., Ballesteros, M., Ling, W., Matthews, A., Smith, N.-A.: Transition-based dependency parsing with stack long short-term memory. In: Proceedings of ACL-2015 (vol. 1: Long Papers), pp. 334–343, China (2015)
Elkan, C., Greiner R.: Building large knowledge-based systems: representation and inference in the cyc project: D.B. Lenat and R.V. Guhae. In: Artificial Intelligence (1993)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Introducing the knowledge graph: things, not strings. https://googleblog.blogspot.com/2012/05/introducing-knowledge-graph-things-not.html
Lafferty, J., McCallum, A., Pereira, F.-C.N.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: ICML (2001)
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: CoRR (2016)
Ling, W., et al.: Finding function in form: compositional character models for open vocabulary word representation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2015)
Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 1064–1074, Berlin (2016)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: CoRR (2013)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: CoRR (2013)
Pennington, J., Socher, R., Manning, C.-D.: GloVe: Global vectors for word representation. In: 12th Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), pp. 1532–1543 (2014)
Speer, R., Chin, J., Havasi, C.: ConceptNet 5.5: an open multilingual graph of general knowledge. In: CoRR (2016)
Tjong Kim Sang, E.-F., De Meulder, F.: Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In: CoNLL (2003)
Tjong Kim Sang, E.-F., Veenstra, J.: Representing text chunks. In: Proceedings on EACL 1999, pp. 173–179, Norway (1999)
Trichelair, P., Emami, A., Cheung, J.-C.-K., Trischler, A., Suleman, K., Diaz, F.: On the evaluation of common-sense reasoning in natural language understanding. In: CoRR (2018)
Trinh, T.-H., Le, Q.V.: A simple method for commonsense reasoning. In: CoRR (2018)
Wang, L., Sun, M., Zhao, W., Shen, K., Liu, J.: Yuanfudao at SemEval-2018 Task 11: three-way attention and relational knowledge for commonsense machine comprehension. In: CoRR (2018)
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., Van Durme, B.: ReCoRD: bridging the gap between human and machine commonsense reading comprehension. In: CoRR (2018)
Zhong, W., Tang, D., Duan, N., Zhou, M., Wang, J., Yin, J.: Improving question answering by commonsense-based pre-training. In: CoRR (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Dekhili, G., Le, N.T., Sadat, F. (2019). Improving Named Entity Recognition with Commonsense Knowledge Pre-training. In: Ohara, K., Bai, Q. (eds) Knowledge Management and Acquisition for Intelligent Systems. PKAW 2019. Lecture Notes in Computer Science(), vol 11669. Springer, Cham. https://doi.org/10.1007/978-3-030-30639-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-30639-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30638-0
Online ISBN: 978-3-030-30639-7
eBook Packages: Computer ScienceComputer Science (R0)