Skip to main content
SpringerLink
Log in

A Deep Dive into Multilingual Hate Speech Classification

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track (ECML PKDD 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12461))

Abstract

Hate speech is a serious issue that is currently plaguing the society and has been responsible for severe incidents such as the genocide of the Rohingya community in Myanmar. Social media has allowed people to spread such hateful content even faster. This is especially concerning for countries which lack hate speech detection systems. In this paper, using hate speech dataset in 9 languages from 16 different sources, we perform the first extensive evaluation of multilingual hate speech detection. We analyze the performance of different deep learning models in various scenarios. We observe that in low resource scenario LASER embedding with Logistic regression perform the best, whereas in high resource scenario, BERT based models perform much better. We also observe that simple techniques such as translating to English and using BERT, achieves competitive results in several languages. For cross-lingual classification, we observe that data from other languages seem to improve the performance, especially in the low resource settings. Further, in case of zero-shot classification, evaluation on Italian and Portuguese dataset achieve good results. Our proposed framework could be used as an efficient solution for low-resource languages. These models could also act as good baselines for future multilingual hate speech detection tasks. Our code (Code: https://github.com/punyajoy/DE-LIMIT) and models (Models: https://huggingface.co/Hate-speech-CNERG) are available online.

Warning: contains material that many will find offensive or hateful.

S. S. Aluru and B. Mathew—Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Note that although Table 2 contains 19 entries, there are three occurrences of Ousidhoum et al. [27] and two occurrences of Basile et al. [3] for different languages.

  2. 2.

    We relied on http://hatespeechdata.com for most of the datasets.

  3. 3.

    https://github.com/Hala-Mulki/L-HSAB-First-Arabic-Levantine-HateSpeech-Dataset.

  4. 4.

    https://github.com/HKUST-KnowComp/MLMA_hate_speech.

  5. 5.

    https://github.com/t-davidson/hate-speech-and-offensive-language.

  6. 6.

    https://github.com/aitor-garcia-p/hate-speech-dataset.

  7. 7.

    www.stormfront.org.

  8. 8.

    https://github.com/zeerakw/hatespeech.

  9. 9.

    https://github.com/msang/hateval.

  10. 10.

    https://github.com/ENCASEH2020/hatespeech-twitter.

  11. 11.

    https://github.com/UCSM-DUE/IWG_hatespeech_public.

  12. 12.

    http://www.ub-web.de/research/.

  13. 13.

    https://github.com/okkyibrohim/id-multi-label-hate-speech-and-abusive-language-detection.

  14. 14.

    https://github.com/ialfina/id-hatespeech-detection.

  15. 15.

    https://github.com/msang/hate-speech-corpus.

  16. 16.

    https://github.com/msang/haspeede2018.

  17. 17.

    http://poleval.pl/tasks/task6.

  18. 18.

    https://github.com/paulafortuna/Portuguese-Hate-Speech-Dataset.

  19. 19.

    https://zenodo.org/record/2592149.

  20. 20.

    https://github.com/facebookresearch/LASER.

  21. 21.

    https://github.com/facebookresearch/MUSE.

  22. 22.

    In the total data 0.17% datapoints have more than 128 tokens when tokenized, thus justifying our choice.

  23. 23.

    https://tinyurl.com/yxh57v3a.

  24. 24.

    https://github.com/sergei4e/gtrans.

  25. 25.

    https://en.wikipedia.org/wiki/Gab_(social_network).

  26. 26.

    Note that we rely on translation for interpretations of the errors and the translation itself might also have some error.

References

  1. Alfina, I., Mulia, R., Fanany, M.I., Ekanata, Y.: Hate speech detection in the Indonesian language: a dataset and preliminary study. In: 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pp. 233–238. IEEE (2017)

    Google Scholar 

  2. Artetxe, M., Schwenk, H.: Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguist. 7, 597–610 (2019)

    Article  Google Scholar 

  3. Basile, V., et al.: Semeval-2019 task 5: multilingual detection of hate speech against immigrants and women in twitter. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 54–63 (2019)

    Google Scholar 

  4. Bosco, C., Felice, D., Poletto, F., Sanguinetti, M., Maurizio, T.: Overview of the evalita 2018 hate speech detection task. In: EVALITA 2018-Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. vol. 2263, pp. 1–9. CEUR (2018)

    Google Scholar 

  5. Bretschneider, U., Peters, R.: Detecting offensive statements towards foreigners in social media. In: Proceedings of the 50th Hawaii International Conference on System Sciences (2017)

    Google Scholar 

  6. Burnap, P., Williams, M.L.: Us and them: identifying cyber hate on twitter across multiple protected characteristics. EPJ Data Sci. 5(1), 11 (2016)

    Article  Google Scholar 

  7. Conneau, A., Lample, G., Ranzato, M., Denoyer, L., Jégou, H.: Word translation without parallel data. arXiv preprint arXiv:1710.04087 (2017)

  8. Corazza, M., Menini, S., Cabrio, E., Tonelli, S., Villata, S.: A multilingual evaluation for online hate speech detection. ACM Trans. Internet Technol. (TOIT) 20(2), 1–22 (2020)

    Article  Google Scholar 

  9. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. In: Eleventh International AAAI Conference on Web and Social Media (2017)

    Google Scholar 

  10. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding (2018)

    Google Scholar 

  11. ElSherief, M., Kulkarni, V., Nguyen, D., Wang, W.Y., Belding, E.: Hate lingo: a target-based linguistic analysis of hate speech in social media. In: Twelfth International AAAI Conference on Web and Social Media (2018)

    Google Scholar 

  12. Fasoli, F., Maass, A., Carnaghi, A.: Labelling and discrimination: do homophobic epithets undermine fair distribution of resources? Br. J. Soc. Psychol. 54(2), 383–393 (2015)

    Article  Google Scholar 

  13. Fortuna, P., Nunes, S.: A survey on automatic detection of hate speech in text. ACM Comput. Surv. (CSUR) 51(4), 85 (2018)

    Google Scholar 

  14. Fortuna, P., da Silva, J.R., Wanner, L., Nunes, S., et al.: A hierarchically-labeled Portuguese hate speech dataset. In: Proceedings of the Third Workshop on Abusive Language Online, pp. 94–104 (2019)

    Google Scholar 

  15. Founta, A.M., et al.: Large scale crowdsourcing and characterization of twitter abusive behavior. In: Twelfth International AAAI Conference on Web and Social Media (2018)

    Google Scholar 

  16. Gagliardone, I., Gal, D., Alves, T., Martinez, G.: Countering Online Hate Speech. Unesco Publishing (2015)

    Google Scholar 

  17. de Gibert, O., Perez, N., Pablos, A.G., Cuadros, M.: Hate speech dataset from a white supremacy forum. In: Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pp. 11–20 (2018)

    Google Scholar 

  18. Greenberg, J., Pyszczynski, T.: The effect of an overheard ethnic slur on evaluations of the target: how to spread a social disease. J. Exper. Soc. Psychol. 21(1), 61–72 (1985)

    Article  Google Scholar 

  19. Guermazi, R., Hammami, M., Hamadou, A.B.: Using a semi-automatic keyword dictionary for improving violent web site filtering. In: 2007 Third International IEEE Conference on Signal-Image Technologies and Internet-Based System, pp. 337–344. IEEE (2007)

    Google Scholar 

  20. Huang, X., Xing, L., Dernoncourt, F., Paul, M.J.: Multilingual twitter corpus and baselines for evaluating demographic bias in hate speech recognition. arXiv preprint arXiv:2002.10361 (2020)

  21. Ibrohim, M.O., Budi, I.: Multi-label hate speech and abusive language detection in indonesian twitter. In: Proceedings of the Third Workshop on Abusive Language Online, pp. 46–57 (2019)

    Google Scholar 

  22. Mathew, B., Dutt, R., Goyal, P., Mukherjee, A.: Spread of hate speech in online social media. In: Proceedings of the 10th ACM Conference on Web Science, pp. 173–182 (2019)

    Google Scholar 

  23. Mathew, B., Illendula, A., Saha, P., Sarkar, S., Goyal, P., Mukherjee, A.: Temporal effects of unmoderated hate speech in gab. arXiv preprint arXiv:1909.10966 (2019)

  24. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

  25. Mulki, H., Haddad, H., Ali, C.B., Alshabani, H.: L-hsab: A levantine twitter dataset for hate speech and abusive language. In: Proceedings of the Third Workshop on Abusive Language Online, pp. 111–118 (2019)

    Google Scholar 

  26. Mullen, B., Rice, D.R.: Ethnophaulisms and exclusion: the behavioral consequences of cognitive representation of ethnic immigrant groups. Personal. Soc. Psychol. Bull. 29(8), 1056–1067 (2003)

    Article  Google Scholar 

  27. Ousidhoum, N., Lin, Z., Zhang, H., Song, Y., Yeung, D.Y.: Multilingual and multi-aspect hate speech analysis. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4667–4676 (2019)

    Google Scholar 

  28. Pereira-Kohatsu, J.C., Quijano-Sánchez, L., Liberatore, F., Camacho-Collados, M.: Detecting and monitoring hate speech in twitter. Sensors (Basel, Switzerland), 19(21) (2019)

    Google Scholar 

  29. Ptaszynski, M., Pieciukiewicz, A., Dybała, P.: Results of the poleval 2019 shared task 6: first dataset and open shared task for automatic cyberbullying detection in polish twitter. In: Proceedings of the PolEval2019Workshop, p. 89 (2019)

    Google Scholar 

  30. Ribeiro, M.H., Calais, P.H., Santos, Y.A., Almeida, V.A., Meira Jr, W.: Characterizing and detecting hateful users on twitter. In: Twelfth International AAAI Conference on Web and Social Media (2018)

    Google Scholar 

  31. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you? explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)

    Google Scholar 

  32. Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., Wojatzki, M.: Measuring the reliability of hate speech annotations: The case of the European refugee crisis. arXiv preprint arXiv:1701.08118 (2017)

  33. Sanguinetti, M., Poletto, F., Bosco, C., Patti, V., Stranisci, M.: An Italian twitter corpus of hate speech against immigrants. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)

    Google Scholar 

  34. Singhal, P., Bhattacharyya, P.: Borrow a little from your rich cousin: using embeddings and polarities of English words for multilingual sentiment classification. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 3053–3062 (2016)

    Google Scholar 

  35. Soral, W., Bilewicz, M., Winiewski, M.: Exposure to hate speech increases prejudice through desensitization. Aggressive Behav. 44(2), 136–146 (2018)

    Article  Google Scholar 

  36. Waseem, Z., Hovy, D.: Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In: Proceedings of the NAACL Student Research Workshop, pp. 88–93 (2016)

    Google Scholar 

  37. Zhang, Z., Robinson, D., Tepper, J.: Detecting hate speech on twitter using a convolution-GRU based deep neural network. In: Gangemi, A., et al. (eds.) ESWC 2018. LNCS, vol. 10843, pp. 745–760. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93417-4_48

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology Kharagpur, Kharagpur, India

    Sai Saketh Aluru, Binny Mathew, Punyajoy Saha & Animesh Mukherjee

Authors
  1. Sai Saketh Aluru
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Binny Mathew
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Punyajoy Saha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Animesh Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Binny Mathew .

Editor information

Editors and Affiliations

  1. Microsoft Research, Redmond, WA, USA

    Yuxiao Dong

  2. University College Dublin, Dublin, Ireland

    Georgiana Ifrim

  3. Jožef Stefan Institute, Ljubljana, Slovenia

    Dunja Mladenić

  4. Amazon Alexa Knowledge, Cambridge, UK

    Craig Saunders

  5. Ghent University, Kotrijk, Belgium

    Sofie Van Hoecke

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aluru, S.S., Mathew, B., Saha, P., Mukherjee, A. (2021). A Deep Dive into Multilingual Hate Speech Classification. In: Dong, Y., Ifrim, G., Mladenić, D., Saunders, C., Van Hoecke, S. (eds) Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track. ECML PKDD 2020. Lecture Notes in Computer Science(), vol 12461. Springer, Cham. https://doi.org/10.1007/978-3-030-67670-4_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67670-4_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67669-8

  • Online ISBN: 978-3-030-67670-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation