Abstract
Contextualized sense embedding has been shown to carry useful semantic information to improve the final results of various Natural Language Processing tasks. However, it is still challenging to integrate them with the information of the knowledge base, which is one lack in current state-of-the-art representations. This integration is helpful in NLP tasks, specifically in the lexical ambiguity problem. In this paper, we present C-KASE (Contextualized-Knowledge base Aware Sense Embedding), a novel approach to producing sense embeddings for the lexical meanings within a lexical knowledge base. The novel difference of our representation is the integration of the knowledge base information and the input text. This representation lies in a space that is comparable to that of contextualized word vectors. C-KASE representations enable a simple 1-Nearest-Neighbour algorithm to perform as well as state-of-the-art models in the English Word Sense Disambiguation task. Since this embedding is specified for each individual knowledge base, it also outperforms in other similar tasks, i.e., Wikification and Named Entity Recognition. The results of comparing our method with recent state-of-the-art methods show the efficiency of our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A mention can be one or more tokens.
- 2.
References
Aghaebrahimian, A., Cieliebak, M.: Named entity disambiguation at scale. In: Schilling, F.-P., Stadelmann, T. (eds.) ANNPR 2020. LNCS (LNAI), vol. 12294, pp. 102–110. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58309-5_8, https://digitalcollection.zhaw.ch/bitstream/11475/21530/3/2020_Aghaebrahimian-Cieliebak_Named-entity-disambiguation-at-scale.pdf
Agirre, E., de Lacalle, O.L., Soroa, A.: Random walks for knowledge-based word sense disambiguation. Comput. Linguist. 40(1), 57–84 (2014). https://direct.mit.edu/coli/article/40/1/57/145
Aleksandrova, D., Drouin, P., Lareau, F.C.C.O., Venant, A.: The multilingual automatic detection of ’e nonc é s bias ’e s in wikip é dia. ACL (2020). https://www.aclweb.org/anthology/R19-1006.pdf
Azad, H.K., Deepak, A.: A new approach for query expansion using wikipedia and wordnet. Inf. Sci. 492, 147–163 (2019). https://www.sciencedirect.com/science/article/pii/S0020025519303263
Calvo, H., Rocha-Ramírez, A.P., Moreno-Armendáriz, M.A., Duchanoy, C.A.: Toward universal word sense disambiguation using deep neural networks. IEEE Access 7, 60264–60275 (2019). https://ieeexplore.ieee.org/abstract/document/8706934
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: North American Association for Computational Linguistics (NAACL) (2018). https://www.aclweb.org/anthology/N19-1423/
Dixit, V., Dutta, K., Singh, P.: Word sense disambiguation and its approaches. CPUH-Res. J. 1(2), 54–58 (2015). http://www.cpuh.in/academics/pdf
Edmonds, P., Cotton, S.: SENSEVAL-2: overview. In: Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pp. 1–5. Association for Computational Linguistics, Toulouse, France, July 2001. https://www.aclweb.org/anthology/S01-1001.pdf
Ferreira, R.S., Pimentel, M.D.G., Cristo, M.: A wikification prediction model based on the combination of latent, dyadic, and monadic features. IST 69(3), 380–394 (2018). https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.23922
Fogarolli, A.: Word sense disambiguation based on wikipedia link structure. In: 2009 IEEE International Conference on Semantic Computing, pp. 77–82. IEEE (2009). https://ieeexplore.ieee.org/stamp/stamp.jsp
Kwon, S., Oh, D., Ko, Y.: Word sense disambiguation based on context selection using knowledge-based word similarity. Inf. Process. Manag. 58(4), 102551 (2021). https://www.sciencedirect.com/science/article/pii/S0306457321000558
Li, B.: Named entity recognition in the style of object detection. arXiv preprint arXiv:2101.11122 (2021). https://arxiv.org/pdf/2101.11122.pdf
Logeswaran, L., Chang, M.W., Lee, K., Toutanova, K., Devlin, J., Lee, H.: Zero-shot entity linking by reading entity descriptions. arXiv preprint arXiv:1906.07348 (2019). https://arxiv.org/pdf/1906.07348.pdf
Loureiro, D., Jorge, A.: Language modelling makes sense: propagating representations through wordnet for full-coverage word sense disambiguation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5682–5691 (2019). https://www.aclweb.org/anthology/P19-1569
Martinez-Rodriguez, J.L., Hogan, A., Lopez-Arevalo, I.: Information extraction meets the semantic web: a survey. Semantic Web Preprint, pp. 1–81 (2020). http://repositorio.uchile.cl/bitstream/handle/2250/174484/Information-extraction-meets-the-Semantic-Web.pdf?sequence=1
Mikolov, T., Chen, K., Corrado, G.S., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of ICLR, vol. 4, pp. 321–329 (2013). https://arxiv.org/pdf/1301.3781.pdf
Miller, G.A., Beckwith, R., Fellbaum, C., Gross, D., Miller, K.J.: Introduction to wordnet: An on-line lexical database. Int. J. Lexicography 3(4), 235–244 (1990). https://watermark.silverchair.com/235.pdf
Miller, G.A., Leacock, C., Tengi, R., Bunker, R.T.: A semantic concordance. In: Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, 21–24 March, 1993 (1993). https://www.aclweb.org/anthology/H93-1061/
Moro, A., Navigli, R.: SemEval-2015 task 13: multilingual all-words sense disambiguation and entity linking. In: SEM, pp. 288–297. Association for Computational Linguistics, Denver, Colorado, June 2015. https://www.aclweb.org/anthology/S15-2049.pdf
Moro, A., Raganato, A., Navigli, R.: Entity linking meets word sense disambiguation: a unified approach. Trans. Assoc. Comput. Linguist. 2, 231–244 (2014). https://watermark.silverchair.com/tacl_a_00179.pdf
Navigli, R.: Word sense disambiguation: a survey. ACM Comput. Surv. (CSUR) 41(2), 1–69 (2009). https://dl.acm.org/doi/abs/10.1145/1459352.1459355
Navigli, R., Jurgens, D., Vannella, D.: SemEval-2013 task 12: multilingual word sense disambiguation. In: SEM. Association for Computational Linguistics, Atlanta, Georgia, USA, June 2013. https://www.aclweb.org/anthology/S13-2040.pdf
Navigli, R., Ponzetto, S.P.: Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif. Intell. 193, 217–250 (2012). https://www.sciencedirect.com/science/article/pii/S0004370212000793
Nguyen, D.B., Hoffart, J., Theobald, M., Weikum, G.: Aida-light: high-throughput named-entity disambiguation. LDOW 14, 22–32 (2014). http://ceur-ws.org/Vol-1184/ldow2014_paper_03.pdf
Pasini, T., Elia, F.M., Navigli, R.: Huge automatically extracted training sets for multilingual word sense disambiguation. arXiv preprint arXiv:1805.04685 (2018). https://arxiv.org/abs/1805.04685
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1532–1543. EMNLP, Qatar (2014). https://www.aclweb.org/anthology/D14-1162.pdf
Peters, M., et al.: Deep contextualized word representations. Association for Computational Linguistics, pp. 2227–2237 (2018). https://www.aclweb.org/anthology/N18-1202
Peters, M.E., Logan IV, R.L., Schwartz, R., Joshi, V., Singh, S., Smith, N.A.: Knowledge enhanced contextual word representations. arXiv preprint arXiv:1909.04164 (2019). https://arxiv.org/pdf/1909.04164.pdf
Peters, M.E., Neumann, M., Zettlemoyer, L., Yih, W.T.: Dissecting contextual word embeddings: architecture and representation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1499–1509 (2018). https://www.aclweb.org/anthology/D18-1179/
Pilehvar, M.T., Jurgens, D., Navigli, R.: Align, disambiguate and walk: a unified approach for measuring semantic similarity. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1341–1351 (2013). https://www.aclweb.org/anthology/P13-1132.pdf
Pradhan, S., Loper, E., Dligach, D., Palmer, M.: SemEval-2007 task-17: English lexical sample, SRL and all words. In: Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pp. 87–92. Association for Computational Linguistics, Prague, Czech Republic, June 2007. https://www.aclweb.org/anthology/S07-1016
Raganato, A., Bovi, C.D., Navigli, R.: Automatic construction and evaluation of a large semantically enriched wikipedia. In: IJCAI, pp. 2894–2900 (2016). http://wwwusers.di.uniroma1.it/~navigli/pubs/IJCAI_2016_Raganatoetal.pdf
Raganato, A., Camacho-Collados, J., Navigli, R.: Word sense disambiguation: a unified evaluation framework and empirical comparison. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 99–110 (2017). https://www.aclweb.org/anthology/E17-1010/
Reisinger, J., Mooney, R.: Multi-prototype vector-space models of word meaning. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 109–117 (2010). https://www.aclweb.org/anthology/N10-1013.pdf
Saeidi, M., da Sousa, S.B.S., Milios, E., Zeh, N., Berton, L.: Categorizing online harassment on twitter. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1168, pp. 283–297. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43887-6_22, https://link.springer.com/chapter/10.1007/978-3-030-43887-6_22
Scarlini, B., Pasini, T., Navigli, R.: Sensembert: context-enhanced sense embeddings for multilingual word sense disambiguation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8758–8765 (2020). https://ojs.aaai.org//index.php/AAAI/article/view/6402
Scarlini, B., Pasini, T., Navigli, R.: With more contexts comes better performance: contextualized sense embeddings for all-round word sense disambiguation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3528–3539 (2020). https://www.aclweb.org/anthology/2020.emnlp-main.285/
Snyder, B., Palmer, M.: The English all-words task. In: Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pp. 41–43. Association for Computational Linguistics, Barcelona, Spain, July 2004. https://www.aclweb.org/anthology/W04-0811
Wang, A., et al.: Superglue: a stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537 (2019)
Wang, Y., Wang, M., Fujita, H.: Word sense disambiguation: a comprehensive knowledge exploitation framework. Knowl. Based Syst. 43, 105–117 (2019). https://www.sciencedirect.com/science/article/pii/S0950705119304344
Weikum, G., Dong, L., Razniewski, S., Suchanek, F.: Machine knowledge: creation and curation of comprehensive knowledge bases. arXiv preprint arXiv:2009.11564 (2020). https://arxiv.org/pdf/2009.11564.pdf
West, R., Paranjape, A., Leskovec, J.: Mining missing hyperlinks from human navigation traces: a case study of wikipedia. In: Proceedings of the 24th International Conference on World Wide Web, pp. 1242–1252 (2015). https://dl.acm.org/doi/pdf/10.1145/2736277.2741666
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: generalized autoregressive pretraining for language understanding. Curran Associates, Inc., vol. 32, pp. 221–229 (2019). https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf
Zhao, G., Wu, J., Wang, D., Li, T.: Entity disambiguation to wikipedia using collective ranking. Inf. Process. Manag. 52(6), 1247–1257 (2016). https://www.sciencedirect.com/science/article/pii/S0306457316301893
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Saeidi, M., Milios, E., Zeh, N. (2021). Contextualized Knowledge Base Sense Embeddings in Word Sense Disambiguation. In: Barney Smith, E.H., Pal, U. (eds) Document Analysis and Recognition – ICDAR 2021 Workshops. ICDAR 2021. Lecture Notes in Computer Science(), vol 12917. Springer, Cham. https://doi.org/10.1007/978-3-030-86159-9_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-86159-9_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86158-2
Online ISBN: 978-3-030-86159-9
eBook Packages: Computer ScienceComputer Science (R0)
