Abstract
Named entity recognition plays an important role in extracting valuable information from digital libraries, which can help stakeholders to take full advantage of large quantities of documents to boost the development of scholarly knowledge discovery. Nevertheless, there aren’t many annotated NER datasets aiming at scientific literature except medical domain, restricting to utilize abundant of advanced deep learning models. As an alternative solution, distant supervision provides a feasible way to eliminate the need of human annotations by automatically generating annotated datasets based on external resources such as knowledge base, while introducing noise inevitably. In this work, we study the noisy-labeled named entity recognition under distant supervision setting. Considering that most NER systems based on confidence estimation deal with noisy labels ignoring the fact that model has different levels of confidence towards different categories, we propose a Category-oriented confidence calibration (Coca) strategy with an automatically confidence threshold calculation module. We integrate our method into a teacher-student self-training framework to improve the model performance. Our proposed approach achieves promising performance among advanced baseline models and can be easily integrated into other confidence based model frameworks (Our code is publicly available at: https://github.com/possible1402/BOND_Coca).
The work is supported by the Project ‘Research on The Semantic Evaluation System of Scientific and Technological Literature Driven by Big Data’ (Grant No.21 &ZD329).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aguilar, G., López-Monroy, A.P., González, F.A., Solorio, T.: Modeling noisiness to recognize named entities using multitask neural networks on social media. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 1401–1412 (2018)
Al-Moslmi, T., Ocaña, M.G., Opdahl, A.L., Veres, C.: Named entity extraction for knowledge graphs: A literature overview. IEEE Access 8, 32862–32881 (2020)
Aramaki, E., Miura, Y., Tonoike, M., Ohkuma, T., Mashuichi, H., Ohe, K.: TEXT2TABLE: Medical text summarization system based on named entity recognition and modality identification. In: Proceedings of the BioNLP 2009 Workshop. pp. 185–192. Association for Computational Linguistics, Boulder, Colorado (Jun 2009), https://aclanthology.org/W09-1324
Balasuriya, D., Ringland, N., Nothman, J., Murphy, T., Curran, J.R.: Named entity recognition in wikipedia. In: Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources (People’s Web). pp. 10–18 (2009)
Brandsen, A., Verberne, S., Lambers, K., Wansleeben, M.: Can bert dig it?-named entity recognition for information retrieval in the archaeology domain. Journal on Computing and Cultural Heritage (JOCCH) (2022)
Cao, Y., Hu, Z., Chua, T.S., Liu, Z., Ji, H.: Low-resource name tagging learned with weakly labeled data. arXiv preprint arXiv:1908.09659 (2019)
Cao, Y., Hu, Z., Chua, T.s., Liu, Z., Ji, H.: Low-resource name tagging learned with weakly labeled data. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 261–270. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1025,https://aclanthology.org/D19-1025
Godin, F., Vandersmissen, B., De Neve, W., Van de Walle, R.: Multimedia lab@ acl wnut ner shared task: Named entity recognition for twitter microposts using distributed word representations. In: Proceedings of the workshop on noisy user-generated text. pp. 146–153 (2015)
Jie, Z., Xie, P., Lu, W., Ding, R., Li, L.: Better modeling of incomplete annotations for named entity recognition. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 729–734. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1079,https://aclanthology.org/N19-1079
Lamurias, A., Couto, F.M.: Lasigebiotm at mediqa 2019: biomedical question answering using bidirectional transformers and named entity recognition. In: Proceedings of the 18th BioNLP workshop and shared task. pp. 523–527 (2019)
Li, J., Sun, Y., Johnson, R.J., Sciaky, D., Wei, C.H., Leaman, R., Davis, A.P., Mattingly, C.J., Wiegers, T.C., Lu, Z.: Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database 2016 (2016)
Liang, C., Yu, Y., Jiang, H., Er, S., Wang, R., Zhao, T., Zhang, C.: Bond: Bert-assisted open-domain named entity recognition with distant supervision. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1054–1064 (2020)
Liu, K., Fu, Y., Tan, C., Chen, M., Zhang, N., Huang, S., Gao, S.: Noisy-labeled NER with confidence estimation. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 3437–3445. Association for Computational Linguistics, Online (Jun 2021). 10.18653/v1/2021.naacl-main.269, https://aclanthology.org/2021.naacl-main.269
Mayhew, S., Chaturvedi, S., Tsai, C.T., Roth, D.: Named entity recognition with partially annotated training data. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). pp. 645–655. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/K19-1060,https://aclanthology.org/K19-1060
Min, B., Grishman, R., Wan, L., Wang, C., Gondek, D.: Distant supervision for relation extraction with an incomplete knowledge base. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 777–782 (2013)
Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for relation extraction without labeled data. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. pp. 1003–1011 (2009)
Peng, M., Xing, X., Zhang, Q., Fu, J., Huang, X.: Distantly supervised named entity recognition using positive-unlabeled learning. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 2409–2419. Association for Computational Linguistics, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1231,https://aclanthology.org/P19-1231
Ratinov, L., Roth, D.: Design challenges and misconceptions in named entity recognition. In: Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009). pp. 147–155 (2009)
Ritter, A., Zettlemoyer, L., Mausam, M., Etzioni, O.: Modeling missing data in distant supervision for information extraction. Transactions of the Association for Computational Linguistics 1, 367–378 (2013)
Sang, E.F., De Meulder, F.: Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050 (2003)
Shang, J., Liu, L., Gu, X., Ren, X., Ren, T., Han, J.: Learning named entity tagger using domain-specific dictionary. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2054–2064. Association for Computational Linguistics, Brussels, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1230,https://aclanthology.org/D18-1230
Yang, Y., Chen, W., Li, Z., He, Z., Zhang, M.: Distantly supervised NER with partial annotation learning and reinforcement learning. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 2159–2169. Association for Computational Linguistics, Santa Fe, New Mexico, USA (Aug 2018), https://aclanthology.org/C18-1183
Zhang, W., Lin, H., Han, X., Sun, L.: De-biasing distantly supervised named entity recognition via causal intervention. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4803–4813. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.acl-long.371,https://aclanthology.org/2021.acl-long.371
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ding, L., Huang, TY., Liu, H., Wang, Y., Zhang, Z. (2022). Distantly Supervised Named Entity Recognition with Category-Oriented Confidence Calibration. In: Tseng, YH., Katsurai, M., Nguyen, H.N. (eds) From Born-Physical to Born-Virtual: Augmenting Intelligence in Digital Libraries. ICADL 2022. Lecture Notes in Computer Science, vol 13636. Springer, Cham. https://doi.org/10.1007/978-3-031-21756-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-21756-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21755-5
Online ISBN: 978-3-031-21756-2
eBook Packages: Computer ScienceComputer Science (R0)