Skip to main content
SpringerLink
Log in

Unraveling Disagreement Constituents in Hateful Speech

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14611))

Included in the following conference series:

  • 250 Accesses

Abstract

This paper presents a probabilistic semantic approach to identifying disagreement-related textual constituents in hateful content. Several methodologies to exploit the selected constituents to determine if a message could lead to disagreement have been defined. The proposed approach is evaluated on 4 datasets made available for the SemEval 2023 Task 11 shared task, highlighting that a few constituents can be used as a proxy to identify if a sentence could be perceived differently by multiple readers. The source code of our approaches is publicly available (https://github.com/MIND-Lab/Unrevealing-Disagreement-Constituents-in-Hateful-Speech).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The selected datasets contain information about the disagreement among annotators in the form of soft labels. Such agreement values have been transformed into boolean values to represent complete agreement and disagreement among the annotators.

References

  1. Akhtar, S., Basile, V., Patti, V.: Whose opinions matter? Perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896 (2021)

  2. Almanea, D., Poesio, M.: ArMIS - the Arabic misogyny and sexism corpus with annotator subjective disagreements. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2282–2291. European Language Resources Association, Marseille, France (2022). https://aclanthology.org/2022.lrec-1.244

  3. Astorino, A., Rizzi, G., Fersini, E.: Integrated gradients as proxy of disagreement in hateful content. In: CEUR WORKSHOP PROCEEDINGS, vol. 3596. CEUR-WS.org (2023)

    Google Scholar 

  4. Beigman Klebanov, B., Beigman, E.: From annotator agreement to noise models. Comput. Linguist. 35(4), 495–503 (2009)

    Article  Google Scholar 

  5. Cercas Curry, A., Abercrombie, G., Rieser, V.: ConvAbuse: data, analysis, and benchmarks for nuanced abuse detection in conversational AI. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7388–7403. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (2021). https://doi.org/10.18653/v1/2021.emnlp-main.587, https://aclanthology.org/2021.emnlp-main.587

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)

    Google Scholar 

  7. Dumitrache, A., Mediagroep, F., Aroyo, L., Welty, C.: A crowdsourced frame disambiguation corpus with ambiguity. In: Proceedings of NAACL-HLT, pp. 2164–2170 (2019)

    Google Scholar 

  8. Fornaciari, T., Uma, A., Paun, S., Plank, B., Hovy, D., Poesio, M., et al.: Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics (2021)

    Google Scholar 

  9. Han, L., et al.: Crowd worker strategies in relevance judgment tasks. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 241–249 (2020)

    Google Scholar 

  10. Leonardelli, E., Menini, S., Palmero Aprosio, A., Guerini, M., Tonelli, S.: Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10528–10539. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (2021). https://doi.org/10.18653/v1/2021.emnlp-main.822, https://aclanthology.org/2021.emnlp-main.822

  11. Leonardelli, E., et al.: SemEval-2023 task 11: Learning with disagreements (LeWiDi) (2023)

    Google Scholar 

  12. Rizzi, G., Astorino, A., Scalena, D., Rosso, P., Fersini, E.: Mind at SemEval-2023 task 11: From uncertain predictions to subjective disagreement. In: Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pp. 556–564 (2023)

    Google Scholar 

  13. Sandri, M., Leonardelli, E., Tonelli, S., Jezek, E.: Why don’t you do it right? analysing annotators’ disagreement in subjective tasks. In: Vlachos, A., Augenstein, I. (eds.) Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2428–2441. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://doi.org/10.18653/v1/2023.eacl-main.178, https://aclanthology.org/2023.eacl-main.178

  14. Sang, Y., Stanton, J.: The origin and value of disagreement among data labelers: A case study of individual differences in hate speech annotation. In: Smits, M. (eds.) Information for a Better World: Shaping the Global Future: 17th International Conference, iConference 2022, Virtual Event, February 28-March 4, 2022, Proceedings, Part I, pp. 425–444. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-96957-8_36

  15. Uma, A.N., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M.: Learning from disagreement: a survey. J. Artif. Intell. Res. 72, 1385–1470 (2021)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

We acknowledge the support of the PNRR ICSC National Research Centre for High Performance Computing, Big Data and Quantum Computing (CN00000013), under the NRRP MUR program funded by the NextGenerationEU. The work of Paolo Rosso was in the framework of the FairTransNLP-Stereotypes research project (PID2021-124361OB-C31) funded by MCIN/AEI/10.13039/501100011033 and by ERDF, EU A way of making Europe.

Author information

Authors and Affiliations

  1. University of Milano-Bicocca, Milan, Italy

    Giulia Rizzi, Alessandro Astorino & Elisabetta Fersini

  2. Universitat Politècnica de València, Valencia, Spain

    Giulia Rizzi & Paolo Rosso

  3. ValgrAI - Valencian Graduate School and Research Network of Artificial Intelligence, Valencia, Spain

    Paolo Rosso

Authors
  1. Giulia Rizzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Astorino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paolo Rosso
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elisabetta Fersini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giulia Rizzi .

Editor information

Editors and Affiliations

  1. Georgetown University, Washington, WA, USA

    Nazli Goharian

  2. University of Pisa, PISA, Pisa, Italy

    Nicola Tonellotto

  3. King's College London, London, UK

    Yulan He

  4. University College London, London, UK

    Aldo Lipani

  5. University of Glasgow, Glasgow, UK

    Graham McDonald

  6. University of Glasgow, Glasgow, UK

    Craig Macdonald

  7. University of Glasgow, Glasgow, UK

    Iadh Ounis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rizzi, G., Astorino, A., Rosso, P., Fersini, E. (2024). Unraveling Disagreement Constituents in Hateful Speech. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14611. Springer, Cham. https://doi.org/10.1007/978-3-031-56066-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56066-8_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56065-1

  • Online ISBN: 978-3-031-56066-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation