skip to main content
10.1145/3451471.3451498acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicsimConference Proceedingsconference-collections
research-article

All-in-One Hate Speech Detectors May not be what You Want

Published: 13 July 2021 Publication History

Abstract

The detection of Hate speech has been an increasingly active research topic. The results reported by the state-of-the-art systems to automatically detect hateful contents achieved almost perfect performance on common data sets. However, “hate speech” is a very subjective term, and people with different backgrounds have different levels of tolerance to what constitutes hate. In this paper, we show the limitations of having a single classifier handling the problem of hate speech detection. We then propose to build classifiers customized for different people, instead of a single classifier. The main obstacle towards achieving such a goal is the scarcity of data. Therefore, we use transfer learning to overcome this issue and use very limited amount of annotated data to build these customized classifiers. In a first stage, we build a classifier on a large data set which classifies tweets into 3 classes: hate, offensive, clean, and which we refer to as the general classifier. In the second stage, we asked 3 annotators with different backgrounds to re-annotate a small sub-set of tweets (600 tweets) from the original one. We refer to this newly created data set as “the customized data set.” We then fine-tune the general classifier on the customized data set and build the customized classifier for each annotator. The accuracy of classification of corresponding customized data set got 0.08, 0.06 and 0.11 higher than the general classifier. The result shows that it is possible to start with a general classifier, and adjusted it to each individual despite the very limited amount of the training data for him/her.

References

[1]
Sweta Agrawal and Amit Awekar. 2018. Deep Learning for Detecting Cyberbullying Across Multiple Social Media Platforms. In Advances in Information Retrieval, Gabriella Pasi, Benjamin Piwowarski, Leif Azzopardi, and Allan Hanbury (Eds.). Springer International Publishing, Cham, 141–153.
[2]
Marvin Ammori. 2014. The ’New’ New York Times: Free Speech Lawyering in the Age of Google and Twitter. Harvard Law Review 127, Aug. (June 2014), 2259–2295.
[3]
Aymé Arango, Jorge Pérez, and Barbara Poblete. 2019. Hate Speech Detection is Not as Easy as You May Think: A Closer Look at Model Validation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Paris, France) (SIGIR’19). Association for Computing Machinery, New York, NY, USA, 45–54. https://doi.org/10.1145/3331184.3331262
[4]
Twitter Help Center. 2020. Hateful conduct policy. Retrieved February 1, 2020 from https://help.twitter.com/en/rules-and-policies/hateful-conduct-policy
[5]
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15665
[6]
YouTube help. 2020. Hate speech policy. Retrieved February 1, 2020 from https://support.google.com/youtube/answer/2801939
[7]
Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. 328–339. https://doi.org/10.18653/v1/P18-1031
[8]
Irene Kwok and Yuzhou Wang. 2013. Locate the Hate: Detecting Tweets against Blacks. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence (Bellevue, Washington) (AAAI’13). AAAI Press, 1621–1622.
[9]
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182(2017).
[10]
David van Mill. 2018. Freedom of Speech. In The Stanford Encyclopedia of Philosophy (summer 2018 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
[11]
Zeerak Waseem and Dirk Hovy. 2016. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In Proceedings of the NAACL Student Research Workshop. Association for Computational Linguistics, San Diego, California, 88–93. https://doi.org/10.18653/v1/N16-2013
[12]
H. Watanabe, M. Bouazizi, and T. Ohtsuki. 2018. Hate Speech on Twitter: A Pragmatic Approach to Collect Hateful and Offensive Expressions and Perform Hate Speech Detection. IEEE Access 6(2018), 13825–13835. https://doi.org/10.1109/ACCESS.2018.2806394
[13]
Ginette Law William H. Dutton, Anna Dopatkaand Victoria Nash. 2011. Freedom of connection, freedom of expression: the changing legal and regulatory ecology shaping the Internet. United Nations Educational, Scientific and Cultural Organization.
[14]
Ziqi Zhang, David Robinson, and Jonathan Tepper. 2018. Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network. In The Semantic Web, Aldo Gangemi, Roberto Navigli, Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam (Eds.). Springer International Publishing, Cham, 745–760.

Cited By

View all
  • (2023)Exploring Automatic Hate Speech Detection on Social Media: A Focus on Content-Based AnalysisSage Open10.1177/2158244023118131113:2Online publication date: 17-Jun-2023

Index Terms

  1. All-in-One Hate Speech Detectors May not be what You Want
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Other conferences
          ICSIM '21: Proceedings of the 2021 4th International Conference on Software Engineering and Information Management
          January 2021
          251 pages
          ISBN:9781450388955
          DOI:10.1145/3451471
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 13 July 2021

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. Deep Learning
          2. Hate Speech Detection
          3. Transfer Learning
          4. Twitter

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Conference

          ICSIM 2021

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)6
          • Downloads (Last 6 weeks)2
          Reflects downloads up to 16 Feb 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2023)Exploring Automatic Hate Speech Detection on Social Media: A Focus on Content-Based AnalysisSage Open10.1177/2158244023118131113:2Online publication date: 17-Jun-2023

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media