Skip to main content
SpringerLink
Log in

Overview of JOKER – CLEF-2023 Track on Automatic Wordplay Analysis

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14163))

Abstract

The goal of the JOKER track series is to bring together linguists, translators, and computer scientists to foster progress on the automatic interpretation, generation, and translation of wordplay. Being clearly important for various applications, these tasks are still extremely challenging despite significant recent progress in AI in information retrieval and natural language processing. Building on the lessons learned from last year’s edition, JOKER-2023 held three shared tasks aligned with human approaches to the translation of wordplay, or more specifically of puns in English, French, and Spanish: detection, location and interpretation, and finally translation. In this paper, we define these three tasks and describe our approaches to corpus creation and evaluation. We then present an overview of the participating systems, including the summaries of their approaches and a comparison of their performance. As in JOKER-2022, this year’s track also solicited contributions making further use of our data (an “unshared task”), which we also report on.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/DocNow/twarc/.

References

  1. Aliannejadi, M., Faggioli, G., Ferro, N., Vlachos, M. (eds.): Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings, CEUR-WS.org (2023)

    Google Scholar 

  2. Anjum, A., Lieberum, N.: Exploring humor in natural language processing: a comprehensive review of JOKER tasks at CLEF symposium 2023. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  3. Bell, N.D.: Learning about and through humor in the second language classroom. Lang. Teach. Res. 13(3), 241–258 (2009). https://doi.org/10.1177/1362168809104697

    Article  Google Scholar 

  4. Brunelière, O., Germann, C., Salina, K.: CLEF 2023 JOKER Task 2: using Chat GPT for pun location and interpretation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  5. Delabastita, D.: There’s a Double Tongue: an Investigation into the Translation of Shakespeare’s Wordplay, with Special Reference to Hamlet. Rodopi, Amsterdam (1993)

    Google Scholar 

  6. Delabastita, D.: Wordplay as a translation problem: a linguistic perspective. In: Ein internationales Handbuch zur Übersetzungsforschung, vol. 1, pp. 600–606. De Gruyter Mouton, June 2008. https://doi.org/10.1515/9783110137088.1.6.600

  7. Dsilva, R.R.: AKRaNLU @ CLEF JOKER 2023: using sentence embeddings and multilingual models to detect and interpret wordplay. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  8. Dubreuil, Q.: UBO team @ CLEF JOKER 2023 track for task 1, 2 and 3 - applying AI models in regards to pun translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  9. Ermakova, L., Gwenn-Bosser, A., Jatowt, A., Miller, T.: The JOKER corpus: English-French parallel data for multilingual wordplay recognition. In: SIGIR ’23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY (2023). https://doi.org/10.1145/3539618.3591885

  10. Ermakova, L., Miller, T., Bosser, A.G., Preciado, V.M.P., Sidorov, G., Jatowt, A.: Science for fun: the CLEF 2023 JOKER track on automatic wordplay analysis. In: Kamps, J., et al. (eds.) Advances in Information Retrieval. ECIR 2023. LNCS, vol. 13982, pp. 546–556. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-28241-6_63

  11. Ermakova, L., et al.: Overview of JOKER@CLEF 2022: automatic wordplay and humour translation workshop. In: Barron-Cedeno, A., et al. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2022. LNCS, vol. 13390, pp. 447–469. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-13643-6_27

  12. Ermakova, L., et al.: Overview of the CLEF 2022 JOKER task 3: pun translation from English into French. In: Faggioli, G., Ferro, N., Hanbury, A., Potthast, M. (eds.) Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, 5th to 8th September 2022. CEUR Workshop Proceedings, vol. 3180, pp. 1681–1700, August 2022

    Google Scholar 

  13. Glemarec, L., Bosser, A., Boccou, J., Ermakova, L.: Humorous wordplay generation in French. In: Faggioli, G., Ferro, N., Hanbury, A., Potthast, M. (eds.) Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, 5th to 8th September 2022. CEUR Workshop Proceedings, vol. 3180, pp. 1793–1806. CEUR-WS.org (2022)

    Google Scholar 

  14. Glemarec, L., Charles, F.: BU-Pier team @ CLEF JOKER 2023 open task: slip of the tongue generation to improve social interaction with virtual agents. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  15. Jentzsch, S., Kersting, K.: ChatGPT is fun, but it is not funny! Humor is still challenging large language models (2023)

    Google Scholar 

  16. Kolb, W., Miller, T.: Human-computer interaction in pun translation. In: Hadley, J.L., Taivalkoski-Shilov, K., Teixeira, C.S.C., Toral, A. (eds.) Using Technologies for Creative-Text Translation, pp. 66–88. Routledge (2022). https://doi.org/10.4324/9781003094159-4

  17. Komorowska, J., Čatipović, I., Vujica, D.: CLEF2023’ JOKER working notes. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  18. Lieber, O., Sharir, O., Lentz, B., Shoham, Y.: Jurassic-1: technical details and evaluation. White paper, AI21 Labs, August 2021. https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf

  19. Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)

    Book  MATH  Google Scholar 

  20. Miller, T., Hempelmann, C.F., Gurevych, I.: SemEval-2017 Task 7: detection and interpretation of English puns. In: Proceedings of the 11th International Workshop on Semantic Evaluation, pp. 58–68, August 2017. https://doi.org/10.18653/v1/S17-2005

  21. Ohnesorge, F., Gutiérrez, M.Á., Plichta, J.: CLEF 2023 JOKER tasks 2 and 3: using NLP models for pun location, interpretation and translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  22. Palmer, M., Ng, H.T., Dang, H.T.: Evaluation of WSD systems. In: Agirre, E., Edmonds, P. (eds.) Word Sense Disambiguation. Text, Speech and Language Technology, vol. 33, pp. 75–106. Springer, Dordrecht (2007). https://doi.org/10.1007/978-1-4020-4809-8_4

  23. Popova, O., Dadić, P.: Does AI have a sense of humor? CLEF 2023 JOKER tasks 1, 2 and 3: using BLOOM, GPT, SimpleT5, and more for pun detection, location, interpretation and translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  24. Preciado, V.M.P., Preciado, C.P., Sidorov, G.: NLPalma @ CLEF 2023 JOKER: a BLOOMZ and BERT approach for wordplay detection and translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  25. Prnjak, A., Davari, D.R., Schmitt, K.: CLEF 2023 JOKER task 1, 2, 3: pun detection, pun interpretation, and pun translation. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  26. Reicho, S., Jatowt, A.: Innsbruck @ CLEF JOKER 2023 track’s task 1: data augmentation techniques for humor recognition in text. In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

  27. Thomas-Young, T.: Why sentiment analysis is a joke with JOKER data? word-level and interpretation analysis (CLEF 2023 JOKER Task 2). In: Proceedings of the Working Notes of CLEF 2023: Conference and Labs of the Evaluation Forum [1] (2023)

    Google Scholar 

Download references

Acknowledgements

This project has received a government grant managed by the National Research Agency under the program “Investissements d’avenir” integrated into France 2030, with the Reference ANR-19-GURE-0001. JOKER is supported by La Maison des sciences de l’homme en Bretagne. We thank Carolina Palma Preciado, Leopoldo Jesús Gutierrez Galeano, Khatima El Krirh, Nathalie Narváez Bruneau, and Rachel Kinlay for their help and support in the first Spanish pun translation contest. We also thank Quentin Dubreuil, Keith Salina, Constance Germann, Océane Brunelière, Aurianne Damoy, Angelique Robert, and all other colleagues and students who participated in data construction, the translation contests, and the CLEF JOKER track.

Author information

Authors and Affiliations

  1. Université de Bretagne Occidentale, HCTI, Brest, France

    Liana Ermakova & Victor Manuel Palma Preciado

  2. Austrian Research Institute for Artificial Intelligence (OFAI), Vienna, Austria

    Tristan Miller

  3. École Nationale d’Ingénieurs de Brest, Lab-STICC CNRS UMR 6285, Plouzané, France

    Anne-Gwenn Bosser

  4. Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC), Mexico City, Mexico

    Victor Manuel Palma Preciado & Grigori Sidorov

  5. University of Innsbruck, Innsbruck, Austria

    Adam Jatowt

Authors
  1. Liana Ermakova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tristan Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anne-Gwenn Bosser
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Victor Manuel Palma Preciado
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Grigori Sidorov
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Adam Jatowt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tristan Miller .

Editor information

Editors and Affiliations

  1. Democritus University of Thrace, Xanthi, Greece

    Avi Arampatzis

  2. University of Amsterdam, Amsterdam, The Netherlands

    Evangelos Kanoulas

  3. CERTH-ITI, Thessaloniki, Greece

    Theodora Tsikrika

  4. CERTH-ITI, Thessaloniki, Greece

    Stefanos Vrochidis

  5. Utrecht University, Utrecht, The Netherlands

    Anastasia Giachanou

  6. Elsevier, Amsterdam, The Netherlands

    Dan Li

  7. University of Amsterdam, Amsterdam, The Netherlands

    Mohammad Aliannejadi

  8. University of Lausanne, Lausanne, Switzerland

    Michalis Vlachos

  9. University of Padua, Padova, Italy

    Guglielmo Faggioli

  10. University of Padua, Padova, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ermakova, L., Miller, T., Bosser, AG., Palma Preciado, V.M., Sidorov, G., Jatowt, A. (2023). Overview of JOKER – CLEF-2023 Track on Automatic Wordplay Analysis. In: Arampatzis, A., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2023. Lecture Notes in Computer Science, vol 14163. Springer, Cham. https://doi.org/10.1007/978-3-031-42448-9_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42448-9_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42447-2

  • Online ISBN: 978-3-031-42448-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation