Abstract
Offensive language can lead to uncomfortable situations, psychological harm, and, in particular cases, even violence. Social networks and websites struggle to reduce the prevalence of these messages by using an automated detector. One goal of Human-computer interaction (HCI) sciences is to provide respectful, safe, and user-friendly systems. This extends to any form of computer-mediated social interaction. This chapter contributes to this objective by proposing a Romanian language dataset for offensive message detection. We manually annotated 4,052 comments on a Romanian local news website into one of the following classes: non-offensive, targeted insults, racist, homophobic, and sexist. In addition, we establish a baseline of five automated classifiers, out of which the model based on RoBERT and two layers of CNN achieves the highest performance with a weighted F1-score of 74.74%.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
https://github.com/readerbench/news-ro-offense (Accessed on 27th November 2023).
- 2.
https://www.statista.com/topics/7134/social-media-usage-in-romania (Accessed on 27th November 2023).
- 3.
https://www.trafic.ro/vizitatori/top-siteuri-stiri-massmedia/luna-martie-2022-pg1 (Accessed on 17th October 2023).
- 4.
https://www.ranks.nl/stopwords/romanian (Accessed on 15th November 2023).
- 5.
https://spacy.io/models/ro Accessed on 27th November 2023.
- 6.
https://github.com/readerbench/ReaderBench (Accessed on 27th November 2023).
- 7.
Slur for Roma people.
References
T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: a next-generation hyperparameter optimization framework, in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
R. Akula, I. Garibay, Interpretable multi-head self-attention architecture for sarcasm detection in social media. Entropy 23(4), 394 (2021)
I. Alfina, R. Mulia, M.I. Fanany, Y. Ekanata, Hate speech detection in the indonesian language: A dataset and preliminary study, in 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS) (IEEE, 2017), pp. 233–238
F. Alkomah, X. Ma, A literature review of textual hate speech detection methods and datasets. Information 13(6), 273 (2022)
R. Alshalan, H. Al-Khalifa, D. Alsaeed, H. Al-Baity, S. Alshalan et al., Detection of hate speech in covid-19-related tweets in the arab region: deep learning and topic modeling approach. J. Med. Internet Res. 22(12), e22609 (2020)
P. Bojanowski, É. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
T. Caselli, A. Schelhaas, M. Weultjes, F. Leistra, H. van der Veen, G. Timmerman, M. Nissim, Dalc: the dutch abusive language corpus, in Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021) (2021), pp. 54–66
F. Celli, M. Lai, A. Duzha, C. Bosco, V. Patti, Policycorpus xl: An italian corpus for the detection of hate speech against politics, in Proceedings of the Eighth Italian Conference on Computational Linguistics, CLiC-it 2021, Milan, Italy, January 26–28, 2022. Volume 3033 (2021), pp. 56–62
B.R. Chakravarthi, R. Priyadharshini, V. Muralidaran, N. Jose, S. Suryawanshi, E. Sherly, J.P. McCrae, Dravidiancodemix: sentiment analysis and offensive language identification dataset for dravidian languages in code-mixed text. Lang. Resour. Eval. 765–806 (2022)
S.A. Chowdhury, H. Mubarak, A. Abdelali, S.g. Jung, B.J. Jansen, J. Salminen, A multi-platform arabic news comment dataset for offensive language detection, in Proceedings of the 12th Language Resources and Evaluation (2020), pp. 6203–6212
M. Cinelli, A. Pelicon, I. Mozetic, W. Quattrociocchi, P.K. Novak, F. Zollo, Online hate: behavioural dynamics and relationship with misinformation. CoRR arXiv:abs/2105.14005 (2021)
A. Cojocaru, A. Paraschiv, M. Dascalu, News-ro-offense—a romanian offensive language dataset and baseline models centered on news article comments, in 19th International Conference on Human-Computer Interaction, RoCHI 2022, Craiova, Romania/Hybrid, October 6–7, 2022, Matrix Rom, ed. by P. Popescu, C. Kolski (2022), pp. 65–72
Ç. Çøltekin, A corpus of turkish offensive language on social media, in Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association (2020), pp. 6174–6184
Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q.V. Le, R. Salakhutdinov, Transformer-xl: attentive language models beyond a fixed-length context, in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, Association for Computational Linguistics (2019), pp. 2978–2988
T. Davidson, D. Warmsley, M. Macy, I. Weber, Automated hate speech detection and the problem of offensive language, in Proceedings of the International AAAI Conference on Web and Social Media, vol. 11 (2017), pp. 512–515
O. De Gibert, N. Perez, A. García-Pablos, M. Cuadros, Hate speech dataset from a white supremacy forum. arXiv:1809.04444 (2018)
R.P. de Pelle, V.P. Moreira, Offensive comments in the brazilian web: a dataset and baseline results, in Anais do VI Brazilian Workshop on Social Network Analysis and Mining, SBC (2017), pp. 510–519
F. Del Vigna12, A. Cimino23, F. Dell’Orletta, M. Petrocchi, M. Tesconi, Hate me, hate me not: hate speech detection on facebook, in Proceedings of the First Italian Conference on Cybersecurity (ITASEC17) (2017), pp. 86–95
J. Devlin, M.W. Chang, K. Lee, K. Toutanova, Bert: pre-training of deep bidirectional transformers for language understanding, in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NACL, Volume 1 (Long and Short Papers). ACL 2018, 4171–4186 (2018)
J.L. Fleiss, Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378 (1971)
P. Fortuna, J. Soler-Company, L. Wanner, How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets? Inf. Process. Manag. 58(3), 102524 (2021)
B. Gambäck, U.K. Sikdar, Using convolutional neural networks to classify hate-speech, in Proceedings of the First Workshop on Abusive Language Online (2017), pp. 85–90
S. Gupta, Z. Waseem, A comparative study of embeddings methods for hate speech detection from tweets, in EMNLP 2018 Workshop on Noisy User Generated Text (Poster presentation, Association for Computational Linguistics, 2017)
V. Gvozdovaitė, A. Naujalytė, J. Mandravickaitė, T. Krilavičius, An overview of the lithuanian hate speech corpus. Int. J. Des. Anal. Tools Integr. Circuits Syst. 54–57 (2020)
A. Hande, R. Priyadharshini, B.R. Chakravarthi, Kancmd: Kannada codemixed dataset for sentiment analysis and offensive language detection, in Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media (2020), pp. 54–63
B. Kennedy, M. Atari, A.M. Davani, L. Yeh, A. Omrani, Y. Kim, K. Coombs, S. Havaldar, G. Portillo-Wightman, E. Gonzalez, et al., The gab hate corpus: a collection of 27k posts annotated for hate speech (2018)
T.N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings, OpenReview.net (2017)
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, ALBERT: a lite BERT for self-supervised learning of language representations, in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020, OpenReview.net (2020)
S. Li, W. Li, C. Cook, C. Zhu, Y. Gao, Independently recurrent neural network (indrnn): building a longer and deeper RNN, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 5457–5466
Y. Lin, Y. Meng, X. Sun, Q. Han, K. Kuang, J. Li, F. Wu, Bertgcn: transductive text classification by combining GCN and BERT. CoRR arXiv:abs/2105.05727 (2021)
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: a robustly optimized BERT pretraining approach. CoRR arXiv:abs/1907.11692 (2019)
N. Ljubešić, D. Fišer, T. Erjavec, The FRENK datasets of socially unacceptable discourse in slovene and english, in Text, Speech, and Dialogue. ed. by K. Ekštein (Springer International Publishing, Cham, 2019), pp.103–114
Z. Lu, P. Du, J.Y. Nie: Vgcn-bert: augmenting bert with graph embedding for text classification, in European Conference on Information Retrieval, Springer (2020), pp. 369–382
M. Manolescu, Ç. Çöltekin, Roff-a romanian twitter dataset for offensive language, in Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) (2021), pp. 895–900
M. Masala, S. Ruseti, M. Dascalu, Robert–a romanian bert model, in Proceedings of the 28th International Conference on Computational Linguistics (2020), pp. 6626–6637
T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, in 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2–4, 2013, Workshop Track Proceedings (2013)
M. Mohiyaddeen, S. Siddiqui, Automatic hate speech detection: a literature review. Int. J. Eng. Manag. Res. 11, 116–121 (2021)
I. Mollas, Z. Chrysopoulou, S. Karlos, G. Tsoumakas, Ethos: a multi-label hate speech detection dataset. Complex Intell. Syst. 1–16 (2022)
J.P. Montani, P. Schüller, Tuwienkbs19 at germeval task 2, 2019: ensemble learning for german offensive language detection, in Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019, Erlangen, Germany, October 9–11, 2019 (2019)
J. Moon, W.I. Cho, J. Lee, Beep! Korean corpus of online news comments for toxic speech detection, in Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media (2020), pp. 25–31
N. Ousidhoum, Z. Lin, H. Zhang, Y. Song, D.Y. Yeung, Multilingual and multi-aspect hate speech analysis, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019), pp. 4675–4684
A. Paraschiv, I. Sandu, D.C. Cercel, M. Dascalu, Fighting romanian offensive language with ro-offense: a dataset and classification models for online comments. (under review)
A. Paraschiv, D.C. Cercel, Upb at germeval-2019 task 2: Bert-based offensive language classification of german tweets, in Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019, Erlangen, Germany, October 9–11, 2019 (2019)
J.H. Park, P. Fung, One-step and two-step classification for abusive language detection on Twitter, in Proceedings of the First Workshop on Abusive Language Online, Vancouver, BC, Canada, Association for Computational Linguistics (August 2017), pp. 41–45
R. Pascanu, T. Mikolov, Y. Bengio, On the difficulty of training recurrent neural networks, in Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013. Volume 28 of JMLR Workshop and Conference Proceedings., JMLR.org (2013), pp. 1310–1318
J. Pennington, R. Socher, C.D. Manning: Glove: global vectors for word representation, in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014), pp. 1532–1543
H.T. Phan, N.T. Nguyen, D. Hwang, Aspect-level sentiment analysis using CNN over BERT-GCN. IEEE Access 10, 110402–110409 (2022)
T. Pires, E. Schlinger, D. Garrette, How multilingual is multilingual bert, in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), pp. 4996–5001
Z. Pitenis, M. Zampieri, T. Ranasinghe, Offensive language identification in greek, in Proceedings of the 12th Language Resources and Evaluation Conference (2020), pp. 5113–5119
G.K. Pitsilis, H. Ramampiaro, H. Langseth, Effective hate-speech detection in twitter data using recurrent neural networks. Appl. Intell. 48(12), 4730–4742 (2018)
F.M. Plaza-del Arco, A. Montejo-Ráez, L.A.U. Lopez, M.T. Martín-Valdivia, Offendes: a new corpus in spanish for offensive language research, in Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) (2021), pp. 1096–1108
M. Ptaszynski, A. Pieciukiewicz, P. Dybała, Results of the poleval 2019 shared task 6: first dataset and open shared task for automatic cyberbullying detection in polish twitter, in Proceedings of the PolEval 2019 Workshop, vol. 89 (2019)
J.D. Rennie, L. Shih, J. Teevan, D.R. Karger, Tackling the poor assumptions of naive bayes text classifiers, in Proceedings of the 20th International Conference on Machine Learning (ICML-03) (2003), pp. 616–623
D. Robinson, Z. Zhang, J. Tepper, Hate speech detection on twitter: feature engineering vs feature selection, in European Semantic Web Conference (Springer, 2018), pp. 46–49
S. Rosenthal, P. Atanasova, G. Karadzhov, M. Zampieri, P. Nakov, SOLID: a large-scale semi-supervised dataset for offensive language identification, in Findings of the Association for Computational Linguistics: ACL-IJCNLP, Online, Association for. Computational Linguistics 2021, 915–928 (2021)
A. Safaya, M. Abdullatif, D. Yuret, Kuisail at semeval-2020 task 12: BERT-CNN for offensive speech identification in social media, in Proceedings of the Fourteenth Workshop on Semantic Evaluation (2020), pp. 2054–2059
M. Sanguinetti, F. Poletto, C. Bosco, V. Patti, M. Stranisci, An italian twitter corpus of hate speech against immigrants, in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)
V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. CoRR arXiv:abs/1910.01108 (2019)
A. Sharma, A. Kabra, M. Jain, Ceasing hate with moh: hate speech detection in hindi-english code-switched language. Inf. Process. Manag. 59(1), 102760 (2022)
G.I. Sigurbergsson, L. Derczynski, Offensive language and hate speech detection for danish, in Proceedings of the 12th Language Resources and Evaluation Conference (2020), pp. 3498–3508
K. Singh, How to improve class imbalance using class weights in machine learning (2020). https://www.analyticsvidhya.com/blog/2020/10/improve-class-imbalance-class-weights/. Accessed 14 June 2022
E. Spertus, Smokey: automatic recognition of hostile messages, in Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Innovative Applications of Artificial Intelligence Conference, AAAI 97, IAAI 97, July 27–31, 1997, Providence, Rhode Island, USA, AAAI Press/The MIT Press, ed. by B. Kuipers, B.L. Webber (1997), pp. 1058–1065
J.M. Struß, M. Siegel, J. Ruppenhofer, M. Wiegand, M. Klenner, Overview of germeval task 2, 2019 shared task on the identification of offensive language, in Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019, Erlangen, Germany, October 9–11, 2019 (2019)
A. Sureka, S. Agarwal, Learning to classify hate and extremism promoting tweets. IEEE joint intelligence and security informatics conference. IEEE 2014, 320–320 (2014)
X. Tang, X. Shen, Y. Wang, Y. Yang, Categorizing offensive language in social networks: a chinese corpus, systems and an explanation tool, in China National Conference on Chinese Computational Linguistics (Springer, 2020), pp. 300–315
Twitter-API. https://developer.twitter.com/en/docs (n.d.). Accessed 1 Sept 2022
P. Törnberg, How digital media drive affective polarization through partisan sorting. Proc. Natl. Acad. Sci. 119(42), e2207159119 (2022)
F. Vargas, I. Carvalho, F. Rodrigues de Góes, T. Pardo, F. Benevenuto, HateBR: a large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection, in Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France, European Language Resources Association (2022), pp. 7174–7183
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł Kaiser, I. Polosukhin, Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 5998–6008 (2017)
Z. Waseem, T. Davidson, D. Warmsley, I. Weber, Understanding abuse: a typology of abusive language detection subtasks. arXiv:1705.09899 (2017)
J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L.A. Hendricks, K. Anderson, P. Kohli, B. Coppin, P. Huang, Challenges in detoxifying language model, in Findings of the Association for Computational Linguistics: EMNLP, Virtual Event/Punta Cana, Dominican Republic, 16–20 November, 2021, Association for Computational Linguistics, ed. by M. Moens, X. Huang, L. Specia, S.W. Yih 2021, 2447–2469 (2021)
G. Xiang, B. Fan, L. Wang, J. Hong, C. Rose, Detecting offensive tweets via topical feature discovery over a large scale twitter corpus, in Proceedings of the 21st ACM International Conference on Information and Knowledge Management (2012), pp. 1980–1984
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R.R. Salakhutdinov, Q.V. Le: Xlnet: generalized autoregressive pretraining for language understanding. Adv. Neural Inf. Process. Syst. 32 (2019)
L. Yao, C. Mao, Y. Luo, Graph convolutional networks for text classification, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33 (2019), pp. 7370–7377
M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, R. Kumar, Predicting the type and target of offensive posts in social media, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, Association for Computational Linguistics (2019), pp. 1415–1420
Y. Zhang, B. Wallace, A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. arXiv:1510.03820 (2015)
Z. Zhang, D. Robinson, J. Tepper, Hate speech detection using a convolution-lstm based deep neural network. Indones. J. Comput. Cybern. Syst. (IJCCS) 11816, 2546–2553 (2019)
J. Zhao, F. Huang, J. Lv, Y. Duan, Z. Qin, G. Li, G. Tian, Do RNN and LSTM have long memory? in International Conference on Machine Learning, PMLR (2020), pp. 11365–11375
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Additional information
Disclaimer Please be advised that this paper includes real comments that may be regarded as profane, offensive, or abusive. This inclusion is unavoidable due to the nature of the research topic.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Paraschiv, A., Cojocaru, A., Dascalu, M. (2024). Automated Offensive Comment Detection for the Romanian Language. In: Kolski, C., Mihăescu, M.C., Rebedea, T. (eds) AI Approaches for Designing and Evaluating Interactive Intelligent Systems. ROCHI 2022. Learning and Analytics in Intelligent Systems, vol 36. Springer, Cham. https://doi.org/10.1007/978-3-031-53957-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-53957-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53956-5
Online ISBN: 978-3-031-53957-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)