Skip to main content
SpringerLink
Log in

Overview of EXIST 2023 – Learning with Disagreement for Sexism Identification and Characterization

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14163))

Abstract

In recent years, the rapid increase in the dissemination of offensive and discriminatory material aimed at women through social media platforms has emerged as a significant concern. This trend has had adverse effects on women’s well-being and their ability to freely express themselves. The EXIST campaign has been promoting research in online sexism detection and categorization since 2021. The third edition of EXIST, hosted at the CLEF 2023 conference, consists of three tasks, two of which are the continuation of EXIST 2022 (sexism identification and sexism categorization), and a third and novel one is on source intention identification. For this edition, new test and training data are provided and the “learning with disagreement” paradigm is adopted to address disagreements in the labelling process and promote the development of equitable systems that are able to learn from different perspectives on the sexism phenomena. 28 teams participated in the three EXIST 2023 tasks, submitting 232 runs. This lab overview describes the tasks, dataset, evaluation methodology, approaches and results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://nlp.uned.es/exist2023/. Accessed 14 June 2023.

  2. 2.

    https://gab.com/. Accessed 14 June 2023.

  3. 3.

    https://everydaysexism.com/. Accessed 14 June 2023.

  4. 4.

    No personally identifiable information about the crowd-workers was collected. Workers were informed that the tweets could contain offensive information and were allowed to withdraw voluntarily at any time. Full consent was obtained.

  5. 5.

    https://www.prolific.co/. Accessed 14 June 2023.

  6. 6.

    In the case of zero variance, we must consider that the probability for values equals or below the mean is 1 (zero IC) and the probability for values above the mean must be smoothed.

References

  1. Amigó, E., Delgado, A.: Evaluating extreme hierarchical multi-label classification. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. vol. 1: Long Papers, pp. 5809–5819. ACL, Dublin, Ireland (2022)

    Google Scholar 

  2. Angel, J., Aroyehun, S., Gelbukh, A.: Multilingual Sexism Identification using contrastive learning. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  3. Asnani, H., Davis, A., Rajanala, A., Kübler, S.: Tlatlamiztli: fine-tuned RoBERTuito for sexism detection. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  4. Burgos, A., et al.: Violencias de género 2.0. (2014)

    Google Scholar 

  5. Buzzell, M., Dickinson, J., Singh, N., Kübler, S.: IU-NLP-JeDi: investigating sexism detection in English and Spanish. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  6. Böck, J., et al.: AIT_FHSTP at EXIST 2023 benchmark: sexism detection by transfer learning, sentiment and toxicity embeddings and hand-crafted features. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  7. Chaudhary, A., Kumar, R.: Sexism identification in social networks. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  8. Dhrodia, A.: Social media and the silencing effect: why misogyny online is a human rights issue (2017). https://www.newstatesman.com/culture/social-media/2017/11/social-media-and-silencing-effect-why-misogyny-online-human-rights-issue

  9. Díaz, J.A.G., Pan, R., Valencia-Garcia, R.: UMUTeam at EXIST 2023: sexism identification and categorisation fine-tuning multilingual large language models. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  10. Erbani, J., Egyed-Zsigmond, E., Nurbakova, D., Portier, P.E.: When multiple perspectives and an optimization process lead to better performance, an automatic sexism identification on social media with pretrained transformers in a soft label context. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  11. Gabel, E., Redman, H., Swanson, D., Kübler, S.: IU-Percival: linear models for sexism detection. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  12. Gerbner, G., Gross, L., Morgan, M., Signorielli, N.: Living with television: the dynamics of the cultivation process. Perspect. Media Effects 1986, 17–40 (1986)

    Google Scholar 

  13. Hatekar, Y., Abdo, M., Khanna, S., Kübler, S.: IUEXIST: multilingual pre-trained language models for sexism detection on twitter in EXIST2023. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  14. Hovy, D., Prabhumoye, S.: Five sources of bias in natural language processing. Lang. Linguist. Compass 15(8), e12432 (2021)

    Article  Google Scholar 

  15. Jhakal, C., Singal, K., Suri, M., Chaudhary, D., Gorton, I., Kumar, B.: Detection of sexism on social media with multiple simple transformers. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  16. Kelkar, S., Ravi, S., Madasamy, A.K.: LSTM-attention architecture for online bilingual sexism detection. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  17. Kirk, H.R., Yin, W., Vidgen, B., Röttger, P.: SemEval-2023 task 10: explainable detection of online sexism. In: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval) (2023)

    Google Scholar 

  18. Koonireddy, R., Adel, N.: ROH_NEIL@EXIST2023: detecting sexism in tweets using multilingual language models. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  19. Mohammadi, H., Giachanou, A., Bagheri, A.: Towards robust online sexism detection: a multi-model approach with BERT, XLM-RoBERTa, and DistilBERT for EXIST 2023 Tasks. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  20. Muti, A., Mancini, E.: Enriching hate-tuned transformer-based embeddings with emotions for the categorization of sexism. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  21. Cordón, P., Jacinto Mata, V.P., Domínguez, J.L.: I2C at CLEF-2023 EXIST task: leveraging ensembling language models to detect multilingual sexism in social media. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  22. de Paula, A.F.M., Rizzi, G., Fersini, E., Spina, D.: AI-UPV at EXIST 2023 - sexism characterization using large language models under the learning with disagreement regime. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  23. Pedrosa-Marín, J., de Albornoz, J.C., Plaza, L.: Combining large language models with socio-demographic information for improving sexism detection in social media. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  24. Peterson, J., Battleday, R., Griffiths, T., Russakovsky, O.: Human uncertainty makes classification more robust. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9616–9625. IEEE Computer Society, Los Alamitos, CA, USA (2019)

    Google Scholar 

  25. Petrescu, A.: Leveraging MiniLMv2 pipelines for EXIST2023. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  26. Plaza, L., et al.: Overview of EXIST 2023 - learning with disagreement for sexism identification and characterization (extended overview). In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  27. Radler, G., Ersoy, B.I., Carpentieri, S.: CLassifiers at EXIST 2023: detecting sexism in spanish and english tweets with XLM-T. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  28. Regulagedda, R.M., Leech, Z., Kübler, S.: Specialized or generalized? Sexism detection for EXIST 2023 at CLEF. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  29. Rodrigues, F., Pereira, F.C.: Deep learning from crowds. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI 2018/IAAI 2018/EAAI 2018, AAAI Press (2018)

    Google Scholar 

  30. Rodríguez-Sánchez, F., et al.: Overview of EXIST 2021: sexism identification in social networks. Procesamiento del Lenguaje Nat. 67, 195–207 (2021)

    Google Scholar 

  31. Rodríguez-Sánchez, F., et al.: Overview of EXIST 2022: sexism identification in social networks. Procesamiento del Lenguaje Nat. 69, 229–240 (2022)

    Google Scholar 

  32. Roselli, D., Matthews, J., Talagala, N.: Managing bias in AI. In: Companion Proceedings of The 2019 World Wide Web Conference, pp. 539–544. WWW 2019, Association for Computing Machinery, New York, NY, USA (2019)

    Google Scholar 

  33. Sanchez-Urbina, A., Gómez-Adorno, H., Bel-Enguix, G., Rodríguez-Figueroa, V., Monge-Barrera, A.: iimasGIL_NLP@Exist2023: unveiling sexism on twitter with fine-tuned transformers. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  34. Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 614–622. KDD 2008, Association for Computing Machinery, New York, NY, USA (2008)

    Google Scholar 

  35. Tian, L., Huang, N., Zhang, X.: Efficient multilingual sexism detection via large language models cascades. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  36. Uma, A., et al.: SemEval-2021 task 12: learning with disagreements. In: Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pp. 338–347. Association for Computational Linguistics, Online (2021)

    Google Scholar 

  37. Vallecillo-Rodríguez, M.E., del Arco, F.M.P., Ureña-López, L.A., Martín-Valdivia, M.T., Montejo-Ráez, A.: Integrating annotator information in transformer fine-tuning for sexism detection. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

  38. Vetagiri, A., Adhikary, P.K., Pakray, P., Das, A.: Leveraging GPT-2 for automated classification of online sexist content. In: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum (2023)

    Google Scholar 

Download references

Acknowledgments

This work has been financed by the European Union (NextGenerationEU funds) through the “Plan de Recuperación, Transformación y Resiliencia”, by the Ministry of Economic Affairs and Digital Transformation and by the UNED University. It has also been financed by the Spanish Ministry of Science and Innovation (project FairTransNLP (PID2021-124361OB-C31 and PID2021-124361OB-C32)) funded by MCIN/AEI/10.13039/501100011033 and by ERDF, EU A way of making Europe, and by the Australian Research Council (DE200100064 and CE200100005).

Author information

Authors and Affiliations

  1. Universidad Nacional de Educación a Distancia (UNED), 28040, Madrid, Spain

    Laura Plaza, Jorge Carrillo-de-Albornoz, Roser Morante, Enrique Amigó & Julio Gonzalo

  2. RMIT University, 3000, Melbourne, Australia

    Laura Plaza, Jorge Carrillo-de-Albornoz & Damiano Spina

  3. Universitat Politècnica de València (UPV), 46022, Valencia, Spain

    Paolo Rosso

Authors
  1. Laura Plaza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jorge Carrillo-de-Albornoz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roser Morante
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Enrique Amigó
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Julio Gonzalo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Damiano Spina
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Paolo Rosso
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Plaza .

Editor information

Editors and Affiliations

  1. Democritus University of Thrace, Xanthi, Greece

    Avi Arampatzis

  2. University of Amsterdam, Amsterdam, The Netherlands

    Evangelos Kanoulas

  3. CERTH-ITI, Thessaloniki, Greece

    Theodora Tsikrika

  4. CERTH-ITI, Thessaloniki, Greece

    Stefanos Vrochidis

  5. Utrecht University, Utrecht, The Netherlands

    Anastasia Giachanou

  6. Elsevier, Amsterdam, The Netherlands

    Dan Li

  7. University of Amsterdam, Amsterdam, The Netherlands

    Mohammad Aliannejadi

  8. University of Lausanne, Lausanne, Switzerland

    Michalis Vlachos

  9. University of Padua, Padova, Italy

    Guglielmo Faggioli

  10. University of Padua, Padova, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Plaza, L. et al. (2023). Overview of EXIST 2023 – Learning with Disagreement for Sexism Identification and Characterization. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42448-9_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42447-2

  • Online ISBN: 978-3-031-42448-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation