Abstract
This paper is focused on discovering diachronic semantic shifts in Russian news and social media using different embedding methods. Namely, in our work, we explore the effectiveness of static, dynamic, and contextualized approaches. Using these methods, we reveal social, political, and cultural changes through semantic shifts in the News and Social media corpora; the latter was collected and released as a part of this work. In addition, we compare the performance of these three approaches and highlight their strengths and weaknesses for this task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
The period is chosen based on the available data and is bounded by 2019 before the Covid period.
- 5.
- 6.
The period is chosen based on the available data and is bounded by 2019 before the Covid period.
- 7.
In other words, changes in associations with a word driven by social, cultural an political events and processes.
- 8.
- 9.
Popularity depends on the user that is making the request, that is why the dataset may be biased.
- 10.
- 11.
- 12.
- 13.
The change of the president in some country is a vivid example of such change.
- 14.
For the Social media dataset.
- 15.
We chose 20 for the analysis as the bigger number includes more “noise” words.
- 16.
The analysis was performed by experts with a degree in social sciences. The final decision was made on a binary scale based on the closest semantic neighbors, however, broader textual contexts were also available.
- 17.
To include as many words from the classification dataset as possible, we had to retrain the Word2vec and Dynamic word embeddings models since the latter is initialized with static word2vec embeddings. For retraining, we lowered the frequency threshold from 50 to 5 and increased the window size to 10.
- 18.
We train Random Forest Classifier with the following parameters, which were selected on cross-validation: 1) for Word2vec: number of trees - 120, maximum depth of the tree - 4, minimum number of samples for a split - 4; 2) for Dynamic word embeddings: number of trees - 100, minimum number of samples for a split - 2; 3) for BERT: number of trees - 105, maximum depth of the tree - 5, minimum number of samples for a split - 5; 4) for combined features: number of trees - 120, maximum depth of the tree - 4, minimum number of samples for a split - 12, class weight - balanced.
- 19.
We use a Random Forest Classifier since it has proved efficient for classification tasks and is not as sensitive to the hyperparameters’ choice as gradient boosting algorithms.
- 20.
We use 5-fold stratified cross-validation for evaluation.
- 21.
- 22.
References
Fomin, V., Bakshandaeva, D., Ju, R., Kutuzov, A.: Tracing cultural diachronic semantic shifts in Russian using word embeddings: test sets and baselines. In: Komp’juternaja Lingvistika i Intellektual’nye Tehnologii, pp. 213–227 (2019)
Giulianelli, M.: Lexical semantic change analysis with contextualised word representations. Unpublished master’s thesis, University of Amsterdam, Amsterdam (2019)
Giulianelli, M., Del Tredici, M., Fernández, R.: Analysing lexical semantic change with contextualised word representations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3960–3973 (2020)
Hamilton, W.L., Leskovec, J., Jurafsky, D.: Diachronic word embeddings reveal statistical laws of semantic change. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1489–1501 (2016)
Hu, R., Li, S., Liang, S.: Diachronic sense modeling with deep contextualized word embeddings: an ecological view. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3899–3908 (2019)
Kim, Y., Chiu, Y.I., Hanaki, K., Hegde, D., Petrov, S.: Temporal analysis of language through neural language models. In: Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pp. 61–65 (2014)
Kulkarni, V., Al-Rfou, R., Perozzi, B., Skiena, S.: Statistically significant detection of linguistic change. In: Proceedings of the 24th International Conference on World Wide Web, pp. 625–635 (2015)
Kutuzov, A., Pivovarova, L.: RuShiftEval: a shared task on semantic shift detection for Russian. Comput. Linguist. Intellect. Technol. (2021)
Kenton, J.D., Chang, M.W., Toutanova, L.K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)
Martinc, M., Montariol, S., Zosa, E., Pivovarova, L.: Discovery team at SemEval-2020 task 1: context-sensitive embeddings not always better than static for semantic change detection. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation. International Committee for Computational Linguistics (2020)
Martinc, M., Novak, P.K., Pollak, S.: Leveraging contextual embeddings for detecting diachronic semantic shift. In: Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4811–4819 (2020)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol. 26 (2013)
Montanelli, S., Periti, F.: A survey on contextualised semantic shift detection. arXiv preprint arXiv:2304.01666 (2023)
Montariol, S.: Models of diachronic semantic change using word embeddings. Doctoral dissertation, Université Paris-Saclay (2021)
Peters, M.E., et al.: Deep contextualized word representations. NAACL-HLT. arXiv (2018)
Rodina, J., Trofimova, Y., Kutuzov, A., Artemova, E.: ELMo and BERT in semantic change detection for Russian. In: van der Aalst, W.M.P., et al. (eds.) AIST 2020. LNCS, vol. 12602, pp. 175–186. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72610-2_13
Rosenfeld, A., Erk, K.: Deep neural models of semantic shift. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 474–484 (2018)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wu, Y., et al.:. Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
Yao, Z., Sun, Y., Ding, W., Rao, N., Xiong, H.: Dynamic word embeddings for evolving semantic discovery. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 673–681 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nikonova, V., Tikhonova, M. (2024). Static, Dynamic, or Contextualized: What is the Best Approach for Discovering Semantic Shifts in Russian Media?. In: Ignatov, D.I., et al. Analysis of Images, Social Networks and Texts. AIST 2023. Lecture Notes in Computer Science, vol 14486. Springer, Cham. https://doi.org/10.1007/978-3-031-54534-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-54534-4_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54533-7
Online ISBN: 978-3-031-54534-4
eBook Packages: Computer ScienceComputer Science (R0)