Abstract
Domain-Specific Senses (DSS) acquisition has been one of the major topics in Natural Language Processing (NLP). However, most results from unsupervised learning methods are not effective. This paper addresses the problem and proposes an approach for improving performance based on deep learning. To obtain DSS, we utilize Approximate Personalized Propagation of Neural Predictions (APPNP) consisting of Graph Convolutional Networks (GCN) and PageRank. GCN is a neural network that performs on graphs to learning sense features from neighbors’ senses and using Personalized PageRank for propagation. For constructing sense features, we collect glosses from WordNet and obtained sense embedding by using Bidirectional Encoder Representations from Transformers (BERT). Our experimental results show that the approach works well and attain at 0.614 Macro F1-score. In addition, to demonstrate the efficacy that DSS can work well in the NLP task, we apply the results on DSS to text categorization and gain a macro F1-score at 0.920, while the CNN baseline method is 0.776.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Koeling, R., McCarthy, D., Carroll, J.: Domain-specific sense distributions and predominant sense acquisition. In: Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 419–426 (2005)
Miller, G.A.: WordNet: a lexical database for English. J. Commun. ACM 38, 39–41 (1995)
Rose, T., Stevenson, M., Whitehead, M.: The Reuters corpus volume 1 - from yesterday’s news to tomorrow’s language resources. In: Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC 2002), pp. 29–31 (2002)
Devlin, J., Chang, M-W, Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. Journal of CoRR, arXiv:1810.04805, pp. 4171–4186 (2018)
Klicpera, J., Bojchevski, A., Günnemann, S.: Personalized embedding propagation: combining neural networks on graphs with personalized page rank. Journal of CoRR, arXiv:1810.05997 (2018)
Magnini, B., Cavaglià, G.: Integrating subject field codes into WordNet. In: Proceedings of the Second International Conference on Language and Evaluation (LREC 2000), pp. 1413–1418 (2000)
Gale, W.A., Church, K.W., Yarowsky, D.: One sense per discourse. In: Proceedings of the workshop and Speech and Natural Language, pp. 233–237 (1992)
Magnini, B., Strapparava, C., Pezzulo, G., Gliozzo, A.: Using domain information for word sense disambiguation. In: Proceedings of the the Second International Workshop on Evaluating Word Sense Disambiguation Systems, pp. 111–114 (2001)
Fukumoto, F., Suzuki, Y.: Identifying domain-specific senses and its application to text classification. In: Proceedings of the International Conference on Knowledge Engineering and Ontology Development, pp. 263–268 (2010)
Pasini, T., Navigli, R.: Train-o-Matic: supervised Word Sense Disambiguation with no (manual) effort. J. Artif. Intell. 279, 103215 (2019)
Mikolov, T., Chen K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of 1st International Conference on Learning Representations, ICLR 2013 (2013)
Perozzi, B., Al-Rfou, R., Skiena, S.: DeepWalk: Online learning of social representations. Journal of CoRR, arXiv:1403.6652 (2014)
Agirre, E., Lacalle, O.L., Soroa, A.: Random walks for knowledge-based word sense disambiguation. J. Comput. Linguist. 40, 57–84 (2014)
Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. J. Comput. Netw. 30, 107–117 (1998)
Kipf, T. N., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks. Journal of CoRR, arXiv:1609.02907 (2016)
Li, Q., Han, Z., Wu, X-M: Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning. Journal of CoRR, arXiv:1801.07606 (2018)
Kim, Y.: Convolutional Neural Networks for Sentence Classification. Journal of CoRR, arXiv:1408.5882 (2014)
Liu, J., Chang, W.C., Wu, Y., Yang, Y.: Deep learning for extreme multi-label text classification. In: Proceedings of 40th International ACM SIGIR conference on Research and Development in Information Retrieval (2017)
Wang, J., Wang, Z., Zhang, D., Yan, J.: Combining knowledge with deep convolutional neural networks for short text classification. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence IJCAI-17 (2017)
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60 (2014)
Kusner, M., Sun, Y., Kolkin, N., Weinberger, K.: From word embeddings to document distances. In: Proceedings of Machine Learning Research, pp. 957–966 (2015)
Akiba, T., Sano, S, Yanase, T., Ohta, T., Koyama, M.: Optuna: a next-generation Hyperparameter Optimization Framework. Jounal of CoRR, arXiv:1907.10902 (2019)
Bevilacqua, M., Navigli, R.: Breaking through the 80% glass ceiling: raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2854–2864 (2020)
Yuan, D., Richardson, J., Doherty R. Evans, C., Altendorf, E.: Semi-supervised Word Sense Disambiguation with Neural Models. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 1374–1385 (2016)
Zhou, D., Bousquet, O., Lal, T.N., Weston, J., Schol̈kopf, B.: Learning with local and global consistency. In: Proceedings of Advances in Neural Information Processing Systems, vol. 16, pp. 321–328 (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wangpoonsarp, A., Fukumoto, F. (2021). Predominant Sense Acquisition with a Neural Random Walk Model. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Lecture Notes in Computer Science(), vol 13110. Springer, Cham. https://doi.org/10.1007/978-3-030-92238-2_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-92238-2_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92237-5
Online ISBN: 978-3-030-92238-2
eBook Packages: Computer ScienceComputer Science (R0)